Oct 30 02:40:13.965845 ip-10-0-141-200 systemd[1]: Starting Kubernetes Kubelet... Oct 30 02:40:14.422340 ip-10-0-141-200 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 02:40:14.422340 ip-10-0-141-200 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 02:40:14.422340 ip-10-0-141-200 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 02:40:14.422340 ip-10-0-141-200 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 02:40:14.425746 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425665 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:40:14.425746 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425745 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:40:14.425746 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425749 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:40:14.425839 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425753 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:40:14.425839 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425755 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:40:14.425839 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425758 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:40:14.425839 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425761 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:40:14.425839 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425764 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:40:14.425839 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425768 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:40:14.425839 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425770 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:40:14.425839 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425773 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:40:14.425839 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425776 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:40:14.425839 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425779 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:40:14.425839 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425782 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:40:14.425839 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425785 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:40:14.425839 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425788 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:40:14.425839 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425790 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:40:14.425839 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425793 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:40:14.425839 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425796 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:40:14.425839 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425798 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:40:14.425839 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425801 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:40:14.425839 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425804 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:40:14.426274 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425817 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:40:14.426274 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425821 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:40:14.426274 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425824 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:40:14.426274 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425826 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:40:14.426274 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425829 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:40:14.426274 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425832 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:40:14.426274 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425834 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:40:14.426274 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425837 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:40:14.426274 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425839 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:40:14.426274 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425842 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:40:14.426274 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425845 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:40:14.426274 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425849 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:40:14.426274 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425851 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:40:14.426274 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425854 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:40:14.426274 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425857 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:40:14.426274 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425859 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:40:14.426274 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425862 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:40:14.426274 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425864 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:40:14.426274 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425867 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:40:14.426745 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425869 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:40:14.426745 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425872 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:40:14.426745 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425874 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:40:14.426745 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425877 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:40:14.426745 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425879 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:40:14.426745 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425882 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:40:14.426745 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425884 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:40:14.426745 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425887 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:40:14.426745 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425889 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:40:14.426745 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425892 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:40:14.426745 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425895 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:40:14.426745 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425898 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:40:14.426745 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425900 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:40:14.426745 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425903 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:40:14.426745 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425907 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:40:14.426745 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425910 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:40:14.426745 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425912 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:40:14.426745 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425915 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:40:14.426745 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425917 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:40:14.426745 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425920 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:40:14.427215 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425923 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:40:14.427215 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425925 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:40:14.427215 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425929 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:40:14.427215 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425933 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:40:14.427215 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425937 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:40:14.427215 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425940 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:40:14.427215 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425943 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:40:14.427215 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425946 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:40:14.427215 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425952 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:40:14.427215 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425955 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:40:14.427215 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425958 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:40:14.427215 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425960 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:40:14.427215 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425963 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:40:14.427215 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425966 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:40:14.427215 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425968 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:40:14.427215 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425971 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:40:14.427215 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425973 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:40:14.427215 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425976 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:40:14.427215 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425978 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:40:14.427692 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425981 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:40:14.427692 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425984 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:40:14.427692 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425986 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:40:14.427692 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425989 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:40:14.427692 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425991 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:40:14.427692 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425994 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:40:14.427692 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.425996 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:40:14.427692 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426000 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:40:14.427692 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426003 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:40:14.427692 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426006 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:40:14.427692 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426009 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:40:14.427692 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426011 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:40:14.427692 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426015 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:40:14.427692 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426017 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:40:14.427692 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426020 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:40:14.427692 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426022 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:40:14.427692 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426025 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:40:14.427692 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426027 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:40:14.427692 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426030 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:40:14.427692 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426032 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:40:14.428151 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426035 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:40:14.428151 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426037 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:40:14.428151 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426040 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:40:14.428151 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426042 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:40:14.428151 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426045 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:40:14.428151 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426047 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:40:14.428151 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426834 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:40:14.428151 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426841 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:40:14.428151 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426844 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:40:14.428151 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426849 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:40:14.428151 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426853 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:40:14.428151 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426856 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:40:14.428151 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426859 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:40:14.428151 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426862 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:40:14.428151 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426864 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:40:14.428151 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426867 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:40:14.428151 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426870 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:40:14.428151 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426872 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:40:14.428151 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426875 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:40:14.428151 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426878 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:40:14.428726 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426881 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:40:14.428726 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426885 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:40:14.428726 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426888 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:40:14.428726 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426891 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:40:14.428726 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426893 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:40:14.428726 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426896 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:40:14.428726 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426898 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:40:14.428726 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426901 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:40:14.428726 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426904 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:40:14.428726 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426906 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:40:14.428726 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426908 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:40:14.428726 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426911 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:40:14.428726 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426915 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:40:14.428726 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426919 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:40:14.428726 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426922 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:40:14.428726 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426925 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:40:14.428726 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426928 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:40:14.428726 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426931 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:40:14.428726 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426934 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:40:14.429182 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426936 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:40:14.429182 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426939 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:40:14.429182 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426941 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:40:14.429182 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426944 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:40:14.429182 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426947 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:40:14.429182 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426949 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:40:14.429182 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426952 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:40:14.429182 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426954 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:40:14.429182 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426957 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:40:14.429182 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426960 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:40:14.429182 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426962 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:40:14.429182 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426964 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:40:14.429182 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426967 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:40:14.429182 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426970 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:40:14.429182 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426973 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:40:14.429182 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426976 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:40:14.429182 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426979 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:40:14.429182 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426982 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:40:14.429182 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426984 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:40:14.429646 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426987 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:40:14.429646 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426989 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:40:14.429646 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426992 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:40:14.429646 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426994 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:40:14.429646 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426997 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:40:14.429646 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.426999 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:40:14.429646 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427002 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:40:14.429646 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427004 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:40:14.429646 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427007 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:40:14.429646 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427009 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:40:14.429646 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427012 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:40:14.429646 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427014 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:40:14.429646 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427017 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:40:14.429646 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427020 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:40:14.429646 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427022 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:40:14.429646 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427024 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:40:14.429646 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427027 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:40:14.429646 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427030 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:40:14.429646 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427033 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:40:14.429646 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427035 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:40:14.430120 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427037 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:40:14.430120 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427040 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:40:14.430120 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427043 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:40:14.430120 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427045 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:40:14.430120 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427048 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:40:14.430120 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427051 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:40:14.430120 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427054 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:40:14.430120 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427056 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:40:14.430120 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427059 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:40:14.430120 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427062 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:40:14.430120 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427065 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:40:14.430120 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427067 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:40:14.430120 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427070 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:40:14.430120 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427072 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:40:14.430120 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427075 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:40:14.430120 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427078 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:40:14.430120 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427080 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:40:14.430120 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427083 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:40:14.430120 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427085 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:40:14.430120 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427087 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:40:14.430611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427090 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:40:14.430611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427093 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:40:14.430611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427096 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:40:14.430611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427099 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:40:14.430611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427101 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:40:14.430611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427104 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:40:14.430611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427107 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:40:14.430611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427109 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:40:14.430611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427111 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:40:14.430611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427114 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:40:14.430611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427116 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:40:14.430611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427119 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:40:14.430611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427121 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:40:14.430611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427124 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:40:14.430611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427676 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:40:14.430611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427682 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:40:14.430611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427686 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:40:14.430611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427693 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:40:14.430611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427696 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:40:14.430611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427698 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:40:14.431115 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427701 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:40:14.431115 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427704 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:40:14.431115 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427707 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:40:14.431115 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427710 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:40:14.431115 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427713 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:40:14.431115 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427716 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:40:14.431115 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427718 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:40:14.431115 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427721 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:40:14.431115 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427724 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:40:14.431115 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427726 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:40:14.431115 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427729 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:40:14.431115 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427731 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:40:14.431115 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427734 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:40:14.431115 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427737 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:40:14.431115 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427739 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:40:14.431115 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427741 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:40:14.431115 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427744 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:40:14.431115 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427747 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:40:14.431115 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427749 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:40:14.431115 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427752 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:40:14.431596 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427754 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:40:14.431596 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427756 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:40:14.431596 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427759 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:40:14.431596 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427761 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:40:14.431596 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427764 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:40:14.431596 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427766 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:40:14.431596 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427770 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:40:14.431596 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427774 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:40:14.431596 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427777 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:40:14.431596 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427781 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:40:14.431596 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427784 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:40:14.431596 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427787 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:40:14.431596 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427790 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:40:14.431596 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427792 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:40:14.431596 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427795 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:40:14.431596 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427798 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:40:14.431596 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427800 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:40:14.431596 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427803 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:40:14.431596 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427805 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:40:14.432035 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427808 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:40:14.432035 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427811 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:40:14.432035 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427814 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:40:14.432035 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427817 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:40:14.432035 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427819 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:40:14.432035 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427822 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:40:14.432035 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427824 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:40:14.432035 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427827 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:40:14.432035 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427829 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:40:14.432035 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427831 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:40:14.432035 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427834 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:40:14.432035 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427837 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:40:14.432035 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427839 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:40:14.432035 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427843 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:40:14.432035 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427846 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:40:14.432035 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427848 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:40:14.432035 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427851 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:40:14.432035 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427853 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:40:14.432035 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427856 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:40:14.432512 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427858 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:40:14.432512 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427861 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:40:14.432512 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427863 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:40:14.432512 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427867 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:40:14.432512 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427870 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:40:14.432512 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427872 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:40:14.432512 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427875 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:40:14.432512 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427878 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:40:14.432512 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427880 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:40:14.432512 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427883 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:40:14.432512 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427886 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:40:14.432512 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427888 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:40:14.432512 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427891 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:40:14.432512 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427894 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:40:14.432512 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427897 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:40:14.432512 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427899 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:40:14.432512 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427902 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:40:14.432512 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427904 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:40:14.432512 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427907 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:40:14.432512 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427911 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:40:14.433011 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427914 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:40:14.433011 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427917 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:40:14.433011 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427920 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:40:14.433011 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427922 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:40:14.433011 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427925 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:40:14.433011 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427927 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:40:14.433011 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427931 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:40:14.433011 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427934 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:40:14.433011 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427936 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:40:14.433011 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427939 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:40:14.433011 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427941 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:40:14.433011 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427944 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:40:14.433011 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427946 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:40:14.433011 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427949 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:40:14.433011 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427951 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:40:14.433011 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427955 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:40:14.433011 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427958 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:40:14.433011 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427962 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:40:14.433011 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427965 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:40:14.433011 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427968 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427970 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.427973 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429905 2573 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429921 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429927 2573 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429932 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429936 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429940 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429944 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429948 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429952 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429955 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429958 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429961 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429965 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429968 2573 flags.go:64] FLAG: --cgroup-root="" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429971 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429973 2573 flags.go:64] FLAG: --client-ca-file="" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429977 2573 flags.go:64] FLAG: --cloud-provider="external" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429980 2573 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429986 2573 flags.go:64] FLAG: --cluster-domain="" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429988 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429992 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 02:40:14.433516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429995 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.429998 2573 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430002 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430005 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430009 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430013 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430016 2573 flags.go:64] FLAG: --contention-profiling="false" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430019 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430022 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430025 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430027 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430032 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430036 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430039 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430042 2573 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430045 2573 flags.go:64] FLAG: --enable-server="true" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430048 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430052 2573 flags.go:64] FLAG: --event-burst="100" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430056 2573 flags.go:64] FLAG: --event-qps="50" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430058 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430062 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430065 2573 flags.go:64] FLAG: --eviction-hard="" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430069 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430072 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430075 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 02:40:14.434082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430078 2573 flags.go:64] FLAG: --eviction-soft="" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430081 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430084 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430087 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430090 2573 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430094 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430096 2573 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430099 2573 flags.go:64] FLAG: --feature-gates="" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430103 2573 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430106 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430110 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430113 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430117 2573 flags.go:64] FLAG: --healthz-port="10248" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430120 2573 flags.go:64] FLAG: --help="false" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430124 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-141-200.ec2.internal" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430127 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430130 2573 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430133 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430137 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430140 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430144 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430147 2573 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430150 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 02:40:14.434729 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430153 2573 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430156 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430159 2573 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430162 2573 flags.go:64] FLAG: --kube-reserved="" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430165 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430168 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430171 2573 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430174 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430177 2573 flags.go:64] FLAG: --lock-file="" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430179 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430182 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430185 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430191 2573 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430194 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430197 2573 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430200 2573 flags.go:64] FLAG: --logging-format="text" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430203 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430207 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430210 2573 flags.go:64] FLAG: --manifest-url="" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430213 2573 flags.go:64] FLAG: --manifest-url-header="" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430217 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430220 2573 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430224 2573 flags.go:64] FLAG: --max-pods="110" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430227 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430230 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 02:40:14.435292 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430233 2573 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430236 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430239 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430242 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430244 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430252 2573 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430256 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430259 2573 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430262 2573 flags.go:64] FLAG: --pod-cidr="" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430265 2573 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430267 2573 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430270 2573 flags.go:64] FLAG: --pods-per-core="0" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430273 2573 flags.go:64] FLAG: --port="10250" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430276 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430279 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f639581efe4a7dc6" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430282 2573 flags.go:64] FLAG: --qos-reserved="" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430285 2573 flags.go:64] FLAG: --read-only-port="10255" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430288 2573 flags.go:64] FLAG: --register-node="true" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430291 2573 flags.go:64] FLAG: --register-with-taints="" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430295 2573 flags.go:64] FLAG: --registry-burst="10" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430298 2573 flags.go:64] FLAG: --registry-qps="5" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430300 2573 flags.go:64] FLAG: --reserved-cpus="" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430304 2573 flags.go:64] FLAG: --reserved-memory="" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430308 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430311 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430314 2573 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 02:40:14.435942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430317 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430320 2573 flags.go:64] FLAG: --runonce="false" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430323 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430326 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430329 2573 flags.go:64] FLAG: --seccomp-default="false" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430332 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430335 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430338 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430341 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430344 2573 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430347 2573 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430350 2573 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430353 2573 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430356 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430359 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430362 2573 flags.go:64] FLAG: --system-cgroups="" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430365 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430371 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430374 2573 flags.go:64] FLAG: --tls-cert-file="" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430376 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430381 2573 flags.go:64] FLAG: --tls-min-version="" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430384 2573 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430387 2573 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430390 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430394 2573 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 02:40:14.436543 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430397 2573 flags.go:64] FLAG: --v="2" Oct 30 02:40:14.437405 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430402 2573 flags.go:64] FLAG: --version="false" Oct 30 02:40:14.437405 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430406 2573 flags.go:64] FLAG: --vmodule="" Oct 30 02:40:14.437405 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430410 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 02:40:14.437405 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430414 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 02:40:14.437405 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430581 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:40:14.437405 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430588 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:40:14.437405 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430592 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:40:14.437405 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430597 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:40:14.437405 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430600 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:40:14.437405 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430603 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:40:14.437405 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430606 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:40:14.437405 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430609 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:40:14.437405 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430612 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:40:14.437405 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430615 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:40:14.437405 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430619 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:40:14.437405 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430622 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:40:14.437405 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430625 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:40:14.437405 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430628 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:40:14.437405 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430630 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:40:14.437405 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430633 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:40:14.438169 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430636 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:40:14.438169 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430639 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:40:14.438169 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430641 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:40:14.438169 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430644 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:40:14.438169 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430646 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:40:14.438169 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430649 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:40:14.438169 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430651 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:40:14.438169 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430655 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:40:14.438169 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430659 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:40:14.438169 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430662 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:40:14.438169 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430667 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:40:14.438169 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430670 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:40:14.438169 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430673 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:40:14.438169 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430676 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:40:14.438169 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430678 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:40:14.438169 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430681 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:40:14.438169 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430683 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:40:14.438169 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430687 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:40:14.438169 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430690 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:40:14.438767 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430693 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:40:14.438767 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430695 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:40:14.438767 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430698 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:40:14.438767 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430701 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:40:14.438767 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430704 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:40:14.438767 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430706 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:40:14.438767 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430709 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:40:14.438767 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430711 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:40:14.438767 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430714 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:40:14.438767 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430717 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:40:14.438767 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430719 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:40:14.438767 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430722 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:40:14.438767 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430724 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:40:14.438767 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430727 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:40:14.438767 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430730 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:40:14.438767 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430733 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:40:14.438767 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430735 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:40:14.438767 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430738 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:40:14.438767 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430740 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:40:14.438767 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430743 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:40:14.439244 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430746 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:40:14.439244 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430748 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:40:14.439244 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430751 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:40:14.439244 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430754 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:40:14.439244 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430757 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:40:14.439244 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430760 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:40:14.439244 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430763 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:40:14.439244 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430765 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:40:14.439244 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430768 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:40:14.439244 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430770 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:40:14.439244 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430774 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:40:14.439244 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430777 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:40:14.439244 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430780 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:40:14.439244 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430782 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:40:14.439244 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430785 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:40:14.439244 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430788 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:40:14.439244 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430790 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:40:14.439244 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430793 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:40:14.439244 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430796 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:40:14.439244 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430799 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:40:14.439787 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430801 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:40:14.439787 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430804 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:40:14.439787 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430813 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:40:14.439787 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430817 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:40:14.439787 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430819 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:40:14.439787 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430822 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:40:14.439787 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430825 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:40:14.439787 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430828 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:40:14.439787 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430830 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:40:14.439787 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430833 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:40:14.439787 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430836 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:40:14.439787 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430838 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:40:14.439787 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430841 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:40:14.439787 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430844 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:40:14.439787 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430846 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:40:14.439787 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430851 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:40:14.439787 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430853 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:40:14.439787 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430856 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:40:14.439787 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430859 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:40:14.440229 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430861 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:40:14.440229 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430864 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:40:14.440229 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430867 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:40:14.440229 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430871 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:40:14.440229 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430874 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:40:14.440229 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430876 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:40:14.440229 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430879 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:40:14.440229 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430882 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:40:14.440229 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430884 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:40:14.440229 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430887 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:40:14.440229 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430890 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:40:14.440229 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430892 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:40:14.440229 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.430920 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 02:40:14.440229 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430963 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:40:14.440229 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430968 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:40:14.440229 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430971 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:40:14.440229 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430974 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:40:14.440639 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430977 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:40:14.440639 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430979 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:40:14.440639 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430983 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:40:14.440639 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430988 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:40:14.440639 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430991 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:40:14.440639 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430993 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:40:14.440639 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430996 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:40:14.440639 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.430999 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:40:14.440639 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431001 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:40:14.440639 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431004 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:40:14.440639 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431006 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:40:14.440639 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431010 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:40:14.440639 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431013 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:40:14.440639 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431016 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:40:14.440639 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431018 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:40:14.440639 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431021 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:40:14.440639 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431023 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:40:14.440639 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431026 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:40:14.440639 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431028 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:40:14.441086 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431032 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:40:14.441086 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431034 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:40:14.441086 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431037 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:40:14.441086 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431039 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:40:14.441086 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431042 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:40:14.441086 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431045 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:40:14.441086 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431047 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:40:14.441086 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431050 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:40:14.441086 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431052 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:40:14.441086 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431055 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:40:14.441086 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431057 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:40:14.441086 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431060 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:40:14.441086 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431062 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:40:14.441086 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431065 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:40:14.441086 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431067 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:40:14.441086 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431070 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:40:14.441086 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431072 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:40:14.441086 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431076 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:40:14.441086 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431078 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:40:14.441086 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431081 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:40:14.441543 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431083 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:40:14.441543 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431086 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:40:14.441543 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431089 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:40:14.441543 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431091 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:40:14.441543 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431096 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:40:14.441543 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431098 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:40:14.441543 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431101 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:40:14.441543 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431103 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:40:14.441543 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431106 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:40:14.441543 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431108 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:40:14.441543 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431111 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:40:14.441543 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431113 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:40:14.441543 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431117 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:40:14.441543 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431120 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:40:14.441543 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431123 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:40:14.441543 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431126 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:40:14.441543 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431128 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:40:14.441543 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431131 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:40:14.441543 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431134 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:40:14.441994 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431136 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:40:14.441994 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431139 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:40:14.441994 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431142 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:40:14.441994 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431144 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:40:14.441994 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431146 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:40:14.441994 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431149 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:40:14.441994 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431153 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:40:14.441994 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431156 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:40:14.441994 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431159 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:40:14.441994 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431161 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:40:14.441994 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431164 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:40:14.441994 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431167 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:40:14.441994 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431169 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:40:14.441994 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431172 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:40:14.441994 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431174 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:40:14.441994 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431176 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:40:14.441994 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431179 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:40:14.441994 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431183 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:40:14.441994 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431185 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:40:14.442446 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431188 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:40:14.442446 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431190 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:40:14.442446 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431193 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:40:14.442446 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431196 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:40:14.442446 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431198 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:40:14.442446 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431201 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:40:14.442446 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431204 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:40:14.442446 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431207 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:40:14.442446 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431210 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:40:14.442446 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431213 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:40:14.442446 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431217 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:40:14.442446 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431221 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:40:14.442446 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431224 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:40:14.442446 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431227 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:40:14.442446 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431230 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:40:14.442446 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431233 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:40:14.442446 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431235 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:40:14.442446 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431238 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:40:14.442446 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431240 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:40:14.442952 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431243 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:40:14.442952 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431245 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:40:14.442952 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431248 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:40:14.442952 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431250 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:40:14.442952 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431253 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:40:14.442952 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.431255 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:40:14.442952 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.439395 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 02:40:14.442952 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.439414 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 02:40:14.442952 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.439431 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 02:40:14.442952 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.439436 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 02:40:14.442952 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439498 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:40:14.442952 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439504 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:40:14.442952 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439507 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:40:14.442952 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439510 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:40:14.442952 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439513 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:40:14.442952 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439516 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:40:14.442952 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439519 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:40:14.442952 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439522 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:40:14.442952 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439524 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:40:14.443437 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439527 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:40:14.443437 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439530 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:40:14.443437 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439532 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:40:14.443437 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439535 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:40:14.443437 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439538 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:40:14.443437 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439540 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:40:14.443437 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439543 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:40:14.443437 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439546 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:40:14.443437 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439551 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:40:14.443437 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439573 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:40:14.443437 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439583 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:40:14.443437 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439586 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:40:14.443437 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439588 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:40:14.443437 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439591 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:40:14.443437 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439594 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:40:14.443437 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439596 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:40:14.443437 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439599 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:40:14.443437 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439602 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:40:14.443437 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439604 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:40:14.443437 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439607 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:40:14.443945 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439610 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:40:14.443945 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439612 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:40:14.443945 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439615 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:40:14.443945 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439618 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:40:14.443945 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439620 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:40:14.443945 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439623 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:40:14.443945 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439626 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:40:14.443945 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439628 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:40:14.443945 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439631 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:40:14.443945 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439633 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:40:14.443945 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439636 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:40:14.443945 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439639 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:40:14.443945 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439644 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:40:14.443945 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439648 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:40:14.443945 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439652 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:40:14.443945 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439656 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:40:14.443945 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439659 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:40:14.443945 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439663 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:40:14.443945 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439665 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:40:14.444513 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439668 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:40:14.444513 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439671 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:40:14.444513 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439673 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:40:14.444513 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439676 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:40:14.444513 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439678 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:40:14.444513 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439681 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:40:14.444513 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439683 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:40:14.444513 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439686 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:40:14.444513 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439689 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:40:14.444513 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439692 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:40:14.444513 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439696 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:40:14.444513 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439699 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:40:14.444513 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439702 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:40:14.444513 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439705 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:40:14.444513 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439708 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:40:14.444513 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439710 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:40:14.444513 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439713 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:40:14.444513 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439716 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:40:14.444513 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439719 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:40:14.444972 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439724 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:40:14.444972 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439728 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:40:14.444972 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439732 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:40:14.444972 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439735 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:40:14.444972 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439738 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:40:14.444972 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439741 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:40:14.444972 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439743 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:40:14.444972 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439746 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:40:14.444972 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439749 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:40:14.444972 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439751 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:40:14.444972 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439754 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:40:14.444972 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439756 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:40:14.444972 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439759 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:40:14.444972 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439762 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:40:14.444972 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439765 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:40:14.444972 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439767 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:40:14.444972 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439770 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:40:14.444972 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439772 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:40:14.444972 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439775 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:40:14.444972 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439777 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:40:14.445470 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439780 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:40:14.445470 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439782 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:40:14.445470 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439785 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:40:14.445470 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439787 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:40:14.445470 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439790 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:40:14.445470 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439793 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:40:14.445470 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439798 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:40:14.445470 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439804 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:40:14.445470 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439810 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:40:14.445470 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439814 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:40:14.445470 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439816 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:40:14.445470 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439819 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:40:14.445470 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439821 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:40:14.445470 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439825 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:40:14.445470 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439827 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:40:14.445470 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439830 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:40:14.445470 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439832 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:40:14.445470 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439835 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:40:14.445470 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.439837 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:40:14.445925 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.439866 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 02:40:14.445925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440007 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:40:14.445925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440012 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:40:14.445925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440015 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:40:14.445925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440018 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:40:14.445925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440020 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:40:14.445925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440024 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:40:14.445925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440026 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:40:14.445925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440029 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:40:14.445925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440034 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:40:14.445925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440041 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:40:14.445925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440046 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:40:14.445925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440050 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:40:14.445925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440052 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:40:14.445925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440055 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:40:14.445925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440058 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:40:14.446379 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440060 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:40:14.446379 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440063 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:40:14.446379 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440065 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:40:14.446379 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440068 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:40:14.446379 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440071 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:40:14.446379 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440073 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:40:14.446379 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440076 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:40:14.446379 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440079 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:40:14.446379 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440082 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:40:14.446379 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440084 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:40:14.446379 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440087 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:40:14.446379 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440090 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:40:14.446379 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440092 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:40:14.446379 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440095 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:40:14.446379 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440097 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:40:14.446379 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440100 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:40:14.446379 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440103 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:40:14.446379 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440105 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:40:14.446379 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440108 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:40:14.446379 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440111 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:40:14.447788 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440115 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:40:14.447788 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440119 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:40:14.447788 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440124 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:40:14.447788 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440128 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:40:14.447788 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440131 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:40:14.447788 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440133 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:40:14.447788 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440136 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:40:14.447788 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440138 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:40:14.447788 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440141 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:40:14.447788 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440144 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:40:14.447788 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440149 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:40:14.447788 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440152 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:40:14.447788 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440154 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:40:14.447788 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440157 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:40:14.447788 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440159 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:40:14.447788 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440162 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:40:14.447788 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440164 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:40:14.447788 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440167 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:40:14.447788 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440169 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:40:14.447788 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440172 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:40:14.448355 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440174 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:40:14.448355 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440177 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:40:14.448355 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440179 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:40:14.448355 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440182 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:40:14.448355 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440184 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:40:14.448355 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440187 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:40:14.448355 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440190 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:40:14.448355 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440195 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:40:14.448355 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440199 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:40:14.448355 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440203 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:40:14.448355 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440206 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:40:14.448355 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440209 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:40:14.448355 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440212 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:40:14.448355 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440214 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:40:14.448355 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440217 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:40:14.448355 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440219 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:40:14.448355 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440223 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:40:14.448355 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440226 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:40:14.448355 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440228 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:40:14.448925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440231 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:40:14.448925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440234 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:40:14.448925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440237 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:40:14.448925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440243 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:40:14.448925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440246 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:40:14.448925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440249 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:40:14.448925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440252 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:40:14.448925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440255 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:40:14.448925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440257 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:40:14.448925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440260 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:40:14.448925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440262 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:40:14.448925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440265 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:40:14.448925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440269 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:40:14.448925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440273 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:40:14.448925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440278 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:40:14.448925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440282 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:40:14.448925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440285 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:40:14.448925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440288 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:40:14.448925 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440290 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:40:14.449610 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440294 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:40:14.449610 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440297 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:40:14.449610 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440300 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:40:14.449610 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440302 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:40:14.449610 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440305 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:40:14.449610 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440307 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:40:14.449610 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440310 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:40:14.449610 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440313 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:40:14.449610 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440315 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:40:14.449610 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440318 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:40:14.449610 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440320 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:40:14.449610 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440324 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:40:14.449610 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440327 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:40:14.449610 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.440353 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 02:40:14.449610 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440388 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:40:14.449610 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440414 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:40:14.449992 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440419 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:40:14.449992 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440423 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:40:14.449992 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440427 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:40:14.449992 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440432 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:40:14.449992 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440438 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:40:14.449992 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440444 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:40:14.449992 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440447 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:40:14.449992 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440450 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:40:14.449992 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440453 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:40:14.449992 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440455 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:40:14.449992 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440458 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:40:14.449992 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440461 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:40:14.449992 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440463 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:40:14.449992 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440466 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:40:14.449992 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440469 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:40:14.449992 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440471 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:40:14.449992 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440474 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:40:14.449992 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440477 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:40:14.449992 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440480 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:40:14.450431 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440483 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:40:14.450431 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440485 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:40:14.450431 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440488 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:40:14.450431 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440490 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:40:14.450431 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440493 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:40:14.450431 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440495 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:40:14.450431 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440498 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:40:14.450431 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440500 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:40:14.450431 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440503 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:40:14.450431 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440507 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:40:14.450431 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440511 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:40:14.450431 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440515 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:40:14.450431 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440520 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:40:14.450431 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440524 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:40:14.450431 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440527 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:40:14.450431 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440530 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:40:14.450431 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440532 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:40:14.450431 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440535 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:40:14.450431 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440537 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:40:14.450431 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440540 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:40:14.450915 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440542 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:40:14.450915 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440545 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:40:14.450915 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440548 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:40:14.450915 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440552 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:40:14.450915 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440567 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:40:14.450915 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440570 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:40:14.450915 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440573 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:40:14.450915 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440575 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:40:14.450915 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440578 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:40:14.450915 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440581 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:40:14.450915 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440586 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:40:14.450915 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440590 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:40:14.450915 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440595 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:40:14.450915 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440600 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:40:14.450915 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440603 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:40:14.450915 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440606 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:40:14.450915 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440608 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:40:14.450915 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440612 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:40:14.450915 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440614 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:40:14.451391 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440617 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:40:14.451391 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440620 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:40:14.451391 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440622 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:40:14.451391 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440626 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:40:14.451391 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440629 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:40:14.451391 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440631 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:40:14.451391 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440635 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:40:14.451391 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440638 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:40:14.451391 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440640 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:40:14.451391 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440643 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:40:14.451391 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440646 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:40:14.451391 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440649 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:40:14.451391 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440651 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:40:14.451391 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440654 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:40:14.451391 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440657 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:40:14.451391 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440659 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:40:14.451391 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440662 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:40:14.451391 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440665 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:40:14.451391 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440670 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:40:14.451391 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440674 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:40:14.451931 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440678 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:40:14.451931 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440682 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:40:14.451931 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440684 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:40:14.451931 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440689 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:40:14.451931 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440692 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:40:14.451931 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440695 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:40:14.451931 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440697 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:40:14.451931 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440700 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:40:14.451931 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440702 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:40:14.451931 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440705 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:40:14.451931 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440707 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:40:14.451931 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440710 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:40:14.451931 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440713 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:40:14.451931 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440715 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:40:14.451931 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440718 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:40:14.451931 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440720 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:40:14.451931 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440724 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:40:14.451931 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440726 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:40:14.451931 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440730 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:40:14.452402 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440733 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:40:14.452402 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440736 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:40:14.452402 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440738 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:40:14.452402 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440741 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:40:14.452402 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440744 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:40:14.452402 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440749 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:40:14.452402 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:14.440753 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:40:14.452402 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.440807 2573 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 02:40:14.452402 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.443093 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 02:40:14.452402 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.444085 2573 server.go:1014] "Starting client certificate rotation" Oct 30 02:40:14.452402 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.444197 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 02:40:14.452402 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.444249 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 02:40:14.452402 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.446301 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 02:40:14.452402 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.449312 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 02:40:14.461819 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.461803 2573 log.go:25] "Validated CRI v1 runtime API" Oct 30 02:40:14.466402 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.466388 2573 log.go:25] "Validated CRI v1 image API" Oct 30 02:40:14.467183 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.467161 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 02:40:14.470078 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.470052 2573 fs.go:135] Filesystem UUIDs: map[3732df26-ba27-476d-9822-735e2b79e213:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8093e980-7c04-4d52-829e-7815d40d5cb7:/dev/nvme0n1p4] Oct 30 02:40:14.470136 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.470079 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 02:40:14.478073 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.477944 2573 manager.go:217] Machine: {Timestamp:2026-10-30 02:40:14.47632991 +0000 UTC m=+0.379428733 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100827 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f2b591c7a88874aaa82913ca59e2e SystemUUID:ec2f2b59-1c7a-8887-4aaa-82913ca59e2e BootID:1a15ae50-fc30-480e-88dd-9d244d9262e9 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cb:40:0f:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cb:40:0f:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:a5:dd:54:8b:d0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 02:40:14.478842 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.478831 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 02:40:14.478933 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.478920 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 02:40:14.482278 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.482246 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 02:40:14.482410 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.482279 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-200.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 02:40:14.483848 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.483837 2573 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 02:40:14.483893 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.483850 2573 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 02:40:14.483893 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.483864 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 02:40:14.483893 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.483875 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 02:40:14.485078 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.485068 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 02:40:14.489432 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.489420 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 02:40:14.489567 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.489549 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 02:40:14.490398 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.490389 2573 kubelet.go:493] "Attempting to sync node with API server" Oct 30 02:40:14.490437 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.490402 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 02:40:14.490437 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.490416 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 02:40:14.490437 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.490428 2573 kubelet.go:405] "Adding apiserver pod source" Oct 30 02:40:14.490511 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.490440 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 02:40:14.490833 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.490819 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 02:40:14.493216 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.493180 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 02:40:14.500588 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.500575 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 02:40:14.501541 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.501529 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 02:40:14.501591 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.501574 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 02:40:14.501737 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.501726 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 02:40:14.501766 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.501744 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 02:40:14.501766 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.501750 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 02:40:14.501766 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.501756 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 02:40:14.501766 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.501762 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 02:40:14.501869 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.501771 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 02:40:14.502549 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.502540 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 02:40:14.502589 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.502550 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 02:40:14.502589 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.502586 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 02:40:14.502643 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.502592 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 02:40:14.502643 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.502602 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 02:40:14.502643 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.502611 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 02:40:14.503597 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.503583 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 02:40:14.503641 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.503609 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 02:40:14.508095 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.508081 2573 server.go:1263] "Started kubelet" Oct 30 02:40:14.508181 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.508145 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 02:40:14.508272 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.508193 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 02:40:14.508309 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.508295 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 02:40:14.508860 ip-10-0-141-200 systemd[1]: Started Kubernetes Kubelet. Oct 30 02:40:14.509619 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.509596 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 02:40:14.511782 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.511763 2573 server.go:317] "Adding debug handlers to kubelet server" Oct 30 02:40:14.516044 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:14.516021 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 02:40:14.518178 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.518159 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 02:40:14.519494 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.519479 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 02:40:14.519671 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:14.519648 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-200.ec2.internal\" not found" Oct 30 02:40:14.520245 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.520228 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 02:40:14.520245 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.520225 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 02:40:14.520358 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.520253 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 02:40:14.520414 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.520361 2573 reconstruct.go:97] "Volume reconstruction finished" Oct 30 02:40:14.520414 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.520371 2573 reconciler.go:29] "Reconciler: start to sync state" Oct 30 02:40:14.520489 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.520413 2573 factory.go:55] Registering systemd factory Oct 30 02:40:14.520489 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.520442 2573 factory.go:223] Registration of the systemd container factory successfully Oct 30 02:40:14.520693 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.520678 2573 factory.go:153] Registering CRI-O factory Oct 30 02:40:14.520746 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.520697 2573 factory.go:223] Registration of the crio container factory successfully Oct 30 02:40:14.520780 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.520750 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 02:40:14.520780 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.520767 2573 factory.go:103] Registering Raw factory Oct 30 02:40:14.520838 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.520784 2573 manager.go:1199] Started watching for new ooms in manager Oct 30 02:40:14.521866 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.521845 2573 manager.go:319] Starting recovery of all containers Oct 30 02:40:14.532696 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.532677 2573 manager.go:324] Recovery completed Oct 30 02:40:14.536615 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.536593 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-200.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 02:40:14.536746 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:14.536654 2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-200.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 02:40:14.536985 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.536964 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:40:14.538049 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:14.536937 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-200.ec2.internal.18e32e53e6825a05 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-200.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-200.ec2.internal,},FirstTimestamp:2026-10-30 02:40:14.508055045 +0000 UTC m=+0.411153868,LastTimestamp:2026-10-30 02:40:14.508055045 +0000 UTC m=+0.411153868,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-200.ec2.internal,}" Oct 30 02:40:14.539088 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.539068 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:40:14.539152 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.539096 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:40:14.539152 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.539106 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:40:14.539617 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.539600 2573 cpu_manager.go:226] "Starting" policy="none" Oct 30 02:40:14.539617 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.539612 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 02:40:14.539749 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.539619 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:40:14.539749 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.539628 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 02:40:14.540338 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.540317 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:40:14.540407 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.540352 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:40:14.540407 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.540367 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:40:14.540939 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.540922 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 02:40:14.540994 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.540981 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tn5dd" Oct 30 02:40:14.541694 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.541678 2573 policy_none.go:50] "Start" Oct 30 02:40:14.541694 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.541694 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 02:40:14.541809 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.541704 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 02:40:14.543616 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.543603 2573 policy_none.go:44] "Start" Oct 30 02:40:14.548370 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.548352 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tn5dd" Oct 30 02:40:14.557479 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:14.557455 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-200.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 02:40:14.557479 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:14.557458 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 02:40:14.557638 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:14.557431 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-200.ec2.internal.18e32e53e85bd97b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-200.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-200.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-200.ec2.internal,},FirstTimestamp:2026-10-30 02:40:14.539086203 +0000 UTC m=+0.442185026,LastTimestamp:2026-10-30 02:40:14.539086203 +0000 UTC m=+0.442185026,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-200.ec2.internal,}" Oct 30 02:40:14.557819 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:14.557800 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 02:40:14.583143 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.583111 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 02:40:14.585741 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.585727 2573 manager.go:341] "Starting Device Plugin manager" Oct 30 02:40:14.585827 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:14.585757 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 02:40:14.585827 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.585766 2573 server.go:87] "Starting device plugin registration server" Oct 30 02:40:14.586046 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.586033 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 02:40:14.586076 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.586049 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 02:40:14.586150 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.586136 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 02:40:14.586244 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.586208 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 02:40:14.586244 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.586215 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 02:40:14.586798 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:14.586779 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 02:40:14.586888 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:14.586837 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-200.ec2.internal\" not found" Oct 30 02:40:14.677969 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.677912 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 02:40:14.679758 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.679089 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 02:40:14.679758 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.679109 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 02:40:14.679758 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.679127 2573 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 02:40:14.679758 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:14.679158 2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 02:40:14.686605 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.686588 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:40:14.687359 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.687344 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:40:14.687423 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.687371 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:40:14.687423 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.687381 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:40:14.687423 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.687404 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-200.ec2.internal" Oct 30 02:40:14.693845 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.693833 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:40:14.705128 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.704985 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-200.ec2.internal" Oct 30 02:40:14.705214 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:14.705132 2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-200.ec2.internal\": node \"ip-10-0-141-200.ec2.internal\" not found" Oct 30 02:40:14.743447 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:14.743426 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-200.ec2.internal\" not found" Oct 30 02:40:14.779980 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.779958 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-200.ec2.internal"] Oct 30 02:40:14.780070 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.780016 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:40:14.780816 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.780803 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:40:14.780890 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.780829 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:40:14.780890 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.780839 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:40:14.782946 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.782934 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:40:14.783099 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.783086 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" Oct 30 02:40:14.783145 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.783112 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:40:14.783606 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.783593 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:40:14.783606 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.783616 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:40:14.783776 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.783642 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:40:14.783776 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.783656 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:40:14.783776 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.783624 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:40:14.783776 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.783720 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:40:14.785685 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.785669 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-200.ec2.internal" Oct 30 02:40:14.785773 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.785696 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:40:14.786361 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.786346 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:40:14.786436 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.786379 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:40:14.786436 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.786394 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:40:14.809593 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:14.809576 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-200.ec2.internal\" not found" node="ip-10-0-141-200.ec2.internal" Oct 30 02:40:14.811692 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:14.811677 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-200.ec2.internal\" not found" node="ip-10-0-141-200.ec2.internal" Oct 30 02:40:14.821858 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.821843 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09b727cb32d7ac800f6bb9bd152bad1f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-200.ec2.internal\" (UID: \"09b727cb32d7ac800f6bb9bd152bad1f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-200.ec2.internal" Oct 30 02:40:14.821915 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.821870 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6496276491b4aff226133cbf4e8c9798-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal\" (UID: \"6496276491b4aff226133cbf4e8c9798\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" Oct 30 02:40:14.821915 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.821889 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6496276491b4aff226133cbf4e8c9798-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal\" (UID: \"6496276491b4aff226133cbf4e8c9798\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" Oct 30 02:40:14.844407 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:14.844389 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-200.ec2.internal\" not found" Oct 30 02:40:14.922699 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.922666 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6496276491b4aff226133cbf4e8c9798-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal\" (UID: \"6496276491b4aff226133cbf4e8c9798\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" Oct 30 02:40:14.922699 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.922699 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6496276491b4aff226133cbf4e8c9798-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal\" (UID: \"6496276491b4aff226133cbf4e8c9798\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" Oct 30 02:40:14.922887 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.922715 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09b727cb32d7ac800f6bb9bd152bad1f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-200.ec2.internal\" (UID: \"09b727cb32d7ac800f6bb9bd152bad1f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-200.ec2.internal" Oct 30 02:40:14.922887 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.922778 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6496276491b4aff226133cbf4e8c9798-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal\" (UID: \"6496276491b4aff226133cbf4e8c9798\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" Oct 30 02:40:14.922887 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.922800 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09b727cb32d7ac800f6bb9bd152bad1f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-200.ec2.internal\" (UID: \"09b727cb32d7ac800f6bb9bd152bad1f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-200.ec2.internal" Oct 30 02:40:14.922887 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:14.922778 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6496276491b4aff226133cbf4e8c9798-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal\" (UID: \"6496276491b4aff226133cbf4e8c9798\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" Oct 30 02:40:14.945097 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:14.945053 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-200.ec2.internal\" not found" Oct 30 02:40:15.045967 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:15.045936 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-200.ec2.internal\" not found" Oct 30 02:40:15.113168 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:15.113142 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" Oct 30 02:40:15.115277 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:15.115261 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-200.ec2.internal" Oct 30 02:40:15.146724 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:15.146691 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-200.ec2.internal\" not found" Oct 30 02:40:17.287348 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:17.287321 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-200.ec2.internal\" not found" Oct 30 02:40:17.387816 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:17.387784 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-200.ec2.internal\" not found" Oct 30 02:40:17.484500 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:17.484469 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 02:40:17.484998 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:17.484654 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 02:40:17.488762 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:17.488741 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-200.ec2.internal\" not found" Oct 30 02:40:17.589464 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:17.589434 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-200.ec2.internal\" not found" Oct 30 02:40:17.594255 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:17.594214 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:35:16 +0000 UTC" deadline="2028-05-12 01:04:35.442480884 +0000 UTC" Oct 30 02:40:17.594255 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:17.594253 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13438h24m17.848232183s" Oct 30 02:40:17.689552 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:17.689529 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-200.ec2.internal\" not found" Oct 30 02:40:17.701594 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:17.701557 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6496276491b4aff226133cbf4e8c9798.slice/crio-4a23671b801854ce8d1c37d6fd1db927e2551c9dc7ae06b31bfb9ebbece18d70 WatchSource:0}: Error finding container 4a23671b801854ce8d1c37d6fd1db927e2551c9dc7ae06b31bfb9ebbece18d70: Status 404 returned error can't find the container with id 4a23671b801854ce8d1c37d6fd1db927e2551c9dc7ae06b31bfb9ebbece18d70 Oct 30 02:40:17.702837 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:17.702814 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b727cb32d7ac800f6bb9bd152bad1f.slice/crio-d55a5603b6cd1e4f6a135c34bc6ede909abe96f23baea1fcea1d56ec925d2ef5 WatchSource:0}: Error finding container d55a5603b6cd1e4f6a135c34bc6ede909abe96f23baea1fcea1d56ec925d2ef5: Status 404 returned error can't find the container with id d55a5603b6cd1e4f6a135c34bc6ede909abe96f23baea1fcea1d56ec925d2ef5 Oct 30 02:40:17.706679 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:17.706665 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:40:17.708799 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:17.708771 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:40:17.723587 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:17.723548 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-200.ec2.internal" event={"ID":"09b727cb32d7ac800f6bb9bd152bad1f","Type":"ContainerStarted","Data":"d55a5603b6cd1e4f6a135c34bc6ede909abe96f23baea1fcea1d56ec925d2ef5"} Oct 30 02:40:17.724631 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:17.724593 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" event={"ID":"6496276491b4aff226133cbf4e8c9798","Type":"ContainerStarted","Data":"4a23671b801854ce8d1c37d6fd1db927e2551c9dc7ae06b31bfb9ebbece18d70"} Oct 30 02:40:17.759979 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:17.759951 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" Oct 30 02:40:17.781373 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:17.781348 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 02:40:17.783195 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:17.783181 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-200.ec2.internal" Oct 30 02:40:17.800164 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:17.800142 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 02:40:18.027206 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.027124 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:40:18.106784 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.106758 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:40:18.531540 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.531511 2573 apiserver.go:52] "Watching apiserver" Oct 30 02:40:18.559850 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.559823 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 02:40:18.574159 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.574126 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 02:40:18.574522 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.574501 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-cd6rj","kube-system/konnectivity-agent-xkkvm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh","openshift-network-operator/iptables-alerter-llk8x","openshift-multus/network-metrics-daemon-xg5ng","openshift-cluster-node-tuning-operator/tuned-v4kfc","openshift-image-registry/node-ca-s5lq6","openshift-multus/multus-d7vl2","openshift-network-diagnostics/network-check-target-t765c","kube-system/kube-apiserver-proxy-ip-10-0-141-200.ec2.internal","openshift-dns/node-resolver-86gjv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal","openshift-multus/multus-additional-cni-plugins-ttd2z","openshift-ovn-kubernetes/ovnkube-node-j9v4r"] Oct 30 02:40:18.577162 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.577138 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s5lq6" Oct 30 02:40:18.579253 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.579224 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.581736 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.581428 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.583666 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.583647 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-llk8x" Oct 30 02:40:18.586631 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.586154 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:18.586631 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:18.586224 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:18.588460 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.588440 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.591028 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.590996 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:18.591143 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:18.591076 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:18.591234 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.591216 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 02:40:18.591579 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.591522 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 02:40:18.591838 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.591822 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dfk55\"" Oct 30 02:40:18.591929 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.591885 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 02:40:18.592070 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.592055 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 02:40:18.592181 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.592166 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 02:40:18.592282 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.592258 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 02:40:18.592465 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.592407 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 02:40:18.592594 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.592561 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2trkp\"" Oct 30 02:40:18.596575 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.596507 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:18.596683 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:18.596584 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:18.597566 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.597547 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xkkvm" Oct 30 02:40:18.597929 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.597909 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 02:40:18.598715 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.598693 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 02:40:18.598715 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.598709 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cb5f2\"" Oct 30 02:40:18.598853 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.598755 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:40:18.598853 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.598811 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 02:40:18.598853 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.598830 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 02:40:18.599007 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.598843 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nd26x\"" Oct 30 02:40:18.599007 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.598904 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 02:40:18.599007 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.598936 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 02:40:18.600690 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.600671 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.602494 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.602474 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 02:40:18.602938 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.602920 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-86gjv" Oct 30 02:40:18.603926 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.603908 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7l8jj" Oct 30 02:40:18.605151 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.605135 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.612485 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.612467 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7l8jj" Oct 30 02:40:18.661733 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.661714 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 02:40:18.671747 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.671717 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stmvr\" (UniqueName: \"kubernetes.io/projected/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1-kube-api-access-stmvr\") pod \"node-ca-s5lq6\" (UID: \"e754f0c3-6ac7-4e3e-ab0e-c343b673abc1\") " pod="openshift-image-registry/node-ca-s5lq6" Oct 30 02:40:18.671871 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.671758 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs\") pod \"network-metrics-daemon-xg5ng\" (UID: \"5f79e66d-ae26-455c-b111-82f211ffe507\") " pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:18.671871 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.671789 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq248\" (UniqueName: \"kubernetes.io/projected/5f79e66d-ae26-455c-b111-82f211ffe507-kube-api-access-xq248\") pod \"network-metrics-daemon-xg5ng\" (UID: \"5f79e66d-ae26-455c-b111-82f211ffe507\") " pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:18.671871 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.671812 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-modprobe-d\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.671871 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.671834 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-run-openvswitch\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.671871 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.671856 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b33f658-c455-4aa4-ab62-7b5e3de65be5-tmp-dir\") pod \"node-resolver-86gjv\" (UID: \"8b33f658-c455-4aa4-ab62-7b5e3de65be5\") " pod="openshift-dns/node-resolver-86gjv" Oct 30 02:40:18.672140 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.671927 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-cni-netd\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.672140 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.671966 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stdbm\" (UniqueName: \"kubernetes.io/projected/7f749cb9-f4b4-4db4-a580-2877108e718c-kube-api-access-stdbm\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.672140 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672011 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb7cc1ad-e964-4602-9ce5-c838212826d2-iptables-alerter-script\") pod \"iptables-alerter-llk8x\" (UID: \"fb7cc1ad-e964-4602-9ce5-c838212826d2\") " pod="openshift-network-operator/iptables-alerter-llk8x" Oct 30 02:40:18.672140 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672047 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-var-lib-cni-bin\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.672140 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672095 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1-host\") pod \"node-ca-s5lq6\" (UID: \"e754f0c3-6ac7-4e3e-ab0e-c343b673abc1\") " pod="openshift-image-registry/node-ca-s5lq6" Oct 30 02:40:18.672140 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672121 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/913d7a7f-bba5-4c75-bd3d-1727790ca220-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.672491 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672168 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f749cb9-f4b4-4db4-a580-2877108e718c-ovnkube-config\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.672491 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672195 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-system-cni-dir\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.672491 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672219 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-run-multus-certs\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.672491 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672254 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-etc-kubernetes\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.672491 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672307 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-registration-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.672491 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672330 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-systemd\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.672491 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672352 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-var-lib-kubelet\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.672491 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672378 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-slash\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.672491 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672402 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-etc-openvswitch\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.672491 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672425 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-multus-socket-dir-parent\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.672491 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672462 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-multus-conf-dir\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.672491 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672484 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb-agent-certs\") pod \"konnectivity-agent-xkkvm\" (UID: \"6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb\") " pod="kube-system/konnectivity-agent-xkkvm" Oct 30 02:40:18.673075 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672545 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f749cb9-f4b4-4db4-a580-2877108e718c-env-overrides\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.673075 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672570 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f749cb9-f4b4-4db4-a580-2877108e718c-ovnkube-script-lib\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.673075 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672605 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8x7w\" (UniqueName: \"kubernetes.io/projected/fb7cc1ad-e964-4602-9ce5-c838212826d2-kube-api-access-g8x7w\") pod \"iptables-alerter-llk8x\" (UID: \"fb7cc1ad-e964-4602-9ce5-c838212826d2\") " pod="openshift-network-operator/iptables-alerter-llk8x" Oct 30 02:40:18.673075 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672648 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b33f658-c455-4aa4-ab62-7b5e3de65be5-hosts-file\") pod \"node-resolver-86gjv\" (UID: \"8b33f658-c455-4aa4-ab62-7b5e3de65be5\") " pod="openshift-dns/node-resolver-86gjv" Oct 30 02:40:18.673075 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672681 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/913d7a7f-bba5-4c75-bd3d-1727790ca220-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.673075 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672709 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-sys-fs\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.673075 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672731 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-run\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.673075 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672752 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-sys\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.673075 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672779 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-kubelet\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.673075 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672803 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-systemd-units\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.673075 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672826 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-var-lib-openvswitch\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.673075 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672850 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f749cb9-f4b4-4db4-a580-2877108e718c-ovn-node-metrics-cert\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.673075 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672875 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/913d7a7f-bba5-4c75-bd3d-1727790ca220-system-cni-dir\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.673075 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672898 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-device-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.673075 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672921 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm8sz\" (UniqueName: \"kubernetes.io/projected/5077c63e-73b2-4a23-95f0-51edd0875a47-kube-api-access-jm8sz\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.673075 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.672944 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb7cc1ad-e964-4602-9ce5-c838212826d2-host-slash\") pod \"iptables-alerter-llk8x\" (UID: \"fb7cc1ad-e964-4602-9ce5-c838212826d2\") " pod="openshift-network-operator/iptables-alerter-llk8x" Oct 30 02:40:18.673876 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673051 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/107bc8a5-c01b-4b78-aa29-765aab336604-kubelet-config\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:18.673876 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673075 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-cnibin\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.673876 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673097 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-hostroot\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.673876 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673121 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.673876 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673145 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-socket-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.673876 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673169 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-etc-selinux\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.673876 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673193 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-run-k8s-cni-cncf-io\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.673876 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673216 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-var-lib-cni-multus\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.673876 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673239 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bflr\" (UniqueName: \"kubernetes.io/projected/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-kube-api-access-6bflr\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.673876 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673262 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-run-systemd\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.673876 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673286 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-run-ovn-kubernetes\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.673876 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673309 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-cni-bin\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.673876 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673333 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8bsm\" (UniqueName: \"kubernetes.io/projected/8b33f658-c455-4aa4-ab62-7b5e3de65be5-kube-api-access-v8bsm\") pod \"node-resolver-86gjv\" (UID: \"8b33f658-c455-4aa4-ab62-7b5e3de65be5\") " pod="openshift-dns/node-resolver-86gjv" Oct 30 02:40:18.673876 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673357 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-multus-cni-dir\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.673876 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673381 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1-serviceca\") pod \"node-ca-s5lq6\" (UID: \"e754f0c3-6ac7-4e3e-ab0e-c343b673abc1\") " pod="openshift-image-registry/node-ca-s5lq6" Oct 30 02:40:18.673876 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673404 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-run-ovn\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.674583 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673429 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-log-socket\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.674583 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673453 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48721522-308a-4472-b687-d08caa72978d-cni-binary-copy\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.674583 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673476 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-run-netns\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.674583 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673499 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-var-lib-kubelet\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.674583 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673523 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/913d7a7f-bba5-4c75-bd3d-1727790ca220-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.674583 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673546 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-sysctl-d\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.674583 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673569 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-run-netns\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.674583 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673605 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d98p\" (UniqueName: \"kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p\") pod \"network-check-target-t765c\" (UID: \"fb1a01bf-dc4f-430b-8b6d-d5569f85c981\") " pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:18.674583 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673647 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/913d7a7f-bba5-4c75-bd3d-1727790ca220-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.674583 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673672 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-sysconfig\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.674583 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673701 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-kubernetes\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.674583 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673733 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-lib-modules\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.674583 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673765 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-tuned\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.674583 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673790 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:18.674583 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673940 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/913d7a7f-bba5-4c75-bd3d-1727790ca220-cnibin\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.674583 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673956 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-host\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.675320 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673972 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb-konnectivity-ca\") pod \"konnectivity-agent-xkkvm\" (UID: \"6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb\") " pod="kube-system/konnectivity-agent-xkkvm" Oct 30 02:40:18.675320 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.673994 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74x77\" (UniqueName: \"kubernetes.io/projected/913d7a7f-bba5-4c75-bd3d-1727790ca220-kube-api-access-74x77\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.675320 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.674013 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-tmp\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.675320 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.674038 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-node-log\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.675320 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.674054 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/107bc8a5-c01b-4b78-aa29-765aab336604-dbus\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:18.675320 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.674073 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/48721522-308a-4472-b687-d08caa72978d-multus-daemon-config\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.675320 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.674088 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89h27\" (UniqueName: \"kubernetes.io/projected/48721522-308a-4472-b687-d08caa72978d-kube-api-access-89h27\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.675320 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.674104 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-sysctl-conf\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.675320 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.674120 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.690596 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.690532 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 02:40:18.690596 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.690590 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:40:18.690878 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.690842 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 02:40:18.690878 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.690852 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jvpkf\"" Oct 30 02:40:18.690998 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.690904 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 02:40:18.690998 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.690911 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-74jzq\"" Oct 30 02:40:18.690998 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.690984 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 02:40:18.691145 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.690998 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 02:40:18.691145 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.691026 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 02:40:18.691145 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.691002 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 02:40:18.691145 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.691044 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 02:40:18.691145 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.691002 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 02:40:18.691145 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.691051 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xkr2x\"" Oct 30 02:40:18.691769 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.691746 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 02:40:18.691905 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.691886 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q9zqs\"" Oct 30 02:40:18.693870 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.693852 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nmhn4\"" Oct 30 02:40:18.701585 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.701561 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 02:40:18.701720 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.701642 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 02:40:18.775312 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.775282 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/48721522-308a-4472-b687-d08caa72978d-multus-daemon-config\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.775312 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.775315 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89h27\" (UniqueName: \"kubernetes.io/projected/48721522-308a-4472-b687-d08caa72978d-kube-api-access-89h27\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.775563 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.775331 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-sysctl-conf\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.775563 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.775350 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.775563 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.775367 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stmvr\" (UniqueName: \"kubernetes.io/projected/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1-kube-api-access-stmvr\") pod \"node-ca-s5lq6\" (UID: \"e754f0c3-6ac7-4e3e-ab0e-c343b673abc1\") " pod="openshift-image-registry/node-ca-s5lq6" Oct 30 02:40:18.775563 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.775386 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs\") pod \"network-metrics-daemon-xg5ng\" (UID: \"5f79e66d-ae26-455c-b111-82f211ffe507\") " pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:18.775563 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.775514 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-sysctl-conf\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.775563 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.775551 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.775893 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:18.775746 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:18.775893 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:18.775804 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs podName:5f79e66d-ae26-455c-b111-82f211ffe507 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:19.275787686 +0000 UTC m=+3.138826807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs") pod "network-metrics-daemon-xg5ng" (UID: "5f79e66d-ae26-455c-b111-82f211ffe507") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:18.775893 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.775847 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq248\" (UniqueName: \"kubernetes.io/projected/5f79e66d-ae26-455c-b111-82f211ffe507-kube-api-access-xq248\") pod \"network-metrics-daemon-xg5ng\" (UID: \"5f79e66d-ae26-455c-b111-82f211ffe507\") " pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:18.775893 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.775873 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-modprobe-d\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.775893 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.775894 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-run-openvswitch\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.776140 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.775918 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b33f658-c455-4aa4-ab62-7b5e3de65be5-tmp-dir\") pod \"node-resolver-86gjv\" (UID: \"8b33f658-c455-4aa4-ab62-7b5e3de65be5\") " pod="openshift-dns/node-resolver-86gjv" Oct 30 02:40:18.776140 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.775937 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-cni-netd\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.776140 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.775960 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stdbm\" (UniqueName: \"kubernetes.io/projected/7f749cb9-f4b4-4db4-a580-2877108e718c-kube-api-access-stdbm\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.776140 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.775980 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb7cc1ad-e964-4602-9ce5-c838212826d2-iptables-alerter-script\") pod \"iptables-alerter-llk8x\" (UID: \"fb7cc1ad-e964-4602-9ce5-c838212826d2\") " pod="openshift-network-operator/iptables-alerter-llk8x" Oct 30 02:40:18.776140 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.775996 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-var-lib-cni-bin\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.776140 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.775997 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-run-openvswitch\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.776140 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1-host\") pod \"node-ca-s5lq6\" (UID: \"e754f0c3-6ac7-4e3e-ab0e-c343b673abc1\") " pod="openshift-image-registry/node-ca-s5lq6" Oct 30 02:40:18.776140 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.775994 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/48721522-308a-4472-b687-d08caa72978d-multus-daemon-config\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.776140 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776025 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/913d7a7f-bba5-4c75-bd3d-1727790ca220-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.776140 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776047 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-cni-netd\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.776603 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776279 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-modprobe-d\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.776603 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776327 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b33f658-c455-4aa4-ab62-7b5e3de65be5-tmp-dir\") pod \"node-resolver-86gjv\" (UID: \"8b33f658-c455-4aa4-ab62-7b5e3de65be5\") " pod="openshift-dns/node-resolver-86gjv" Oct 30 02:40:18.776603 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776053 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f749cb9-f4b4-4db4-a580-2877108e718c-ovnkube-config\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.776603 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776388 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-system-cni-dir\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.776603 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776416 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-run-multus-certs\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.776603 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776441 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-etc-kubernetes\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.776603 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776484 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-registration-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.776603 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776497 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/913d7a7f-bba5-4c75-bd3d-1727790ca220-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.776603 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776508 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-systemd\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.776603 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776531 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-var-lib-kubelet\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.776603 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776555 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-slash\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.776603 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776580 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-etc-openvswitch\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.776603 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776604 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-multus-socket-dir-parent\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.777223 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776653 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-multus-conf-dir\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.777223 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776679 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb-agent-certs\") pod \"konnectivity-agent-xkkvm\" (UID: \"6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb\") " pod="kube-system/konnectivity-agent-xkkvm" Oct 30 02:40:18.777223 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776700 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f749cb9-f4b4-4db4-a580-2877108e718c-ovnkube-config\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.777223 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776705 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f749cb9-f4b4-4db4-a580-2877108e718c-env-overrides\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.777223 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776731 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f749cb9-f4b4-4db4-a580-2877108e718c-ovnkube-script-lib\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.777223 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776756 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8x7w\" (UniqueName: \"kubernetes.io/projected/fb7cc1ad-e964-4602-9ce5-c838212826d2-kube-api-access-g8x7w\") pod \"iptables-alerter-llk8x\" (UID: \"fb7cc1ad-e964-4602-9ce5-c838212826d2\") " pod="openshift-network-operator/iptables-alerter-llk8x" Oct 30 02:40:18.777223 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776759 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-var-lib-cni-bin\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.777223 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776779 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b33f658-c455-4aa4-ab62-7b5e3de65be5-hosts-file\") pod \"node-resolver-86gjv\" (UID: \"8b33f658-c455-4aa4-ab62-7b5e3de65be5\") " pod="openshift-dns/node-resolver-86gjv" Oct 30 02:40:18.777223 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776805 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1-host\") pod \"node-ca-s5lq6\" (UID: \"e754f0c3-6ac7-4e3e-ab0e-c343b673abc1\") " pod="openshift-image-registry/node-ca-s5lq6" Oct 30 02:40:18.777223 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776805 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/913d7a7f-bba5-4c75-bd3d-1727790ca220-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.777223 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-sys-fs\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.777223 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776849 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-run\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.777223 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776862 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-sys\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.777223 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776878 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-kubelet\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.777223 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776892 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-systemd-units\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.777223 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776907 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-var-lib-openvswitch\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.777223 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776921 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f749cb9-f4b4-4db4-a580-2877108e718c-ovn-node-metrics-cert\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.778019 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776938 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/913d7a7f-bba5-4c75-bd3d-1727790ca220-system-cni-dir\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.778019 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776952 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-device-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.778019 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm8sz\" (UniqueName: \"kubernetes.io/projected/5077c63e-73b2-4a23-95f0-51edd0875a47-kube-api-access-jm8sz\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.778019 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.776995 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb7cc1ad-e964-4602-9ce5-c838212826d2-host-slash\") pod \"iptables-alerter-llk8x\" (UID: \"fb7cc1ad-e964-4602-9ce5-c838212826d2\") " pod="openshift-network-operator/iptables-alerter-llk8x" Oct 30 02:40:18.778019 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777015 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/107bc8a5-c01b-4b78-aa29-765aab336604-kubelet-config\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:18.778019 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777029 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-cnibin\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.778019 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777043 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-hostroot\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.778019 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777058 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.778019 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777073 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-socket-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.778019 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777074 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb7cc1ad-e964-4602-9ce5-c838212826d2-iptables-alerter-script\") pod \"iptables-alerter-llk8x\" (UID: \"fb7cc1ad-e964-4602-9ce5-c838212826d2\") " pod="openshift-network-operator/iptables-alerter-llk8x" Oct 30 02:40:18.778019 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777088 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-etc-selinux\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.778019 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777123 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-run-k8s-cni-cncf-io\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.778019 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777143 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-var-lib-cni-multus\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.778019 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777151 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-etc-selinux\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.778019 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777160 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bflr\" (UniqueName: \"kubernetes.io/projected/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-kube-api-access-6bflr\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.778019 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777178 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-run-systemd\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.778019 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777182 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-sys-fs\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.778905 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777205 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-run-ovn-kubernetes\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.778905 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777223 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-cni-bin\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.778905 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777239 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8bsm\" (UniqueName: \"kubernetes.io/projected/8b33f658-c455-4aa4-ab62-7b5e3de65be5-kube-api-access-v8bsm\") pod \"node-resolver-86gjv\" (UID: \"8b33f658-c455-4aa4-ab62-7b5e3de65be5\") " pod="openshift-dns/node-resolver-86gjv" Oct 30 02:40:18.778905 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777249 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/913d7a7f-bba5-4c75-bd3d-1727790ca220-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.778905 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777260 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-run\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.778905 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777255 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-multus-cni-dir\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.778905 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777292 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-sys\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.778905 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777299 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-multus-cni-dir\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.778905 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777302 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1-serviceca\") pod \"node-ca-s5lq6\" (UID: \"e754f0c3-6ac7-4e3e-ab0e-c343b673abc1\") " pod="openshift-image-registry/node-ca-s5lq6" Oct 30 02:40:18.778905 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777326 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-kubelet\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.778905 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777321 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-systemd-units\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.778905 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777329 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-run-ovn\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.778905 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777345 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-var-lib-openvswitch\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.778905 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777355 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-run-k8s-cni-cncf-io\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.778905 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777358 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-log-socket\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.778905 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777387 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-var-lib-cni-multus\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.778905 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777385 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48721522-308a-4472-b687-d08caa72978d-cni-binary-copy\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.778905 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-run-netns\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.779682 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777426 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-var-lib-kubelet\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.779682 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777449 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/913d7a7f-bba5-4c75-bd3d-1727790ca220-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.779682 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-sysctl-d\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.779682 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777494 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-run-netns\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.779682 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d98p\" (UniqueName: \"kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p\") pod \"network-check-target-t765c\" (UID: \"fb1a01bf-dc4f-430b-8b6d-d5569f85c981\") " pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:18.779682 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777535 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/913d7a7f-bba5-4c75-bd3d-1727790ca220-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.779682 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777560 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-sysconfig\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.779682 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777585 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-kubernetes\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.779682 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777608 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-lib-modules\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.779682 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777651 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-tuned\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.779682 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777657 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-cnibin\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.779682 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777657 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/107bc8a5-c01b-4b78-aa29-765aab336604-kubelet-config\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:18.779682 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777683 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-hostroot\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.779682 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777720 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-cni-bin\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.779682 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:18.777731 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:18.779682 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:18.777763 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret podName:107bc8a5-c01b-4b78-aa29-765aab336604 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:19.277752768 +0000 UTC m=+3.140791885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret") pod "global-pull-secret-syncer-cd6rj" (UID: "107bc8a5-c01b-4b78-aa29-765aab336604") : object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:18.779682 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777669 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:18.780404 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777796 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-socket-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.780404 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777829 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48721522-308a-4472-b687-d08caa72978d-cni-binary-copy\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.780404 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777835 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-run-ovn-kubernetes\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.780404 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777605 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 02:40:18.780404 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777942 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-run-ovn\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.780404 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777977 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/913d7a7f-bba5-4c75-bd3d-1727790ca220-system-cni-dir\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.780404 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777979 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb7cc1ad-e964-4602-9ce5-c838212826d2-host-slash\") pod \"iptables-alerter-llk8x\" (UID: \"fb7cc1ad-e964-4602-9ce5-c838212826d2\") " pod="openshift-network-operator/iptables-alerter-llk8x" Oct 30 02:40:18.780404 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778010 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-log-socket\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.780404 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.777944 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-run-systemd\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.780404 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778026 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-device-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.780404 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778061 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-multus-conf-dir\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.780404 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778069 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-system-cni-dir\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.780404 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778094 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-run-multus-certs\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.780404 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778128 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-etc-kubernetes\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.780404 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778236 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-multus-socket-dir-parent\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.780404 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778288 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-slash\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.780404 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778295 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.780404 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778331 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/913d7a7f-bba5-4c75-bd3d-1727790ca220-cnibin\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.781236 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778369 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-host\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.781236 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778397 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb-konnectivity-ca\") pod \"konnectivity-agent-xkkvm\" (UID: \"6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb\") " pod="kube-system/konnectivity-agent-xkkvm" Oct 30 02:40:18.781236 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778429 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74x77\" (UniqueName: \"kubernetes.io/projected/913d7a7f-bba5-4c75-bd3d-1727790ca220-kube-api-access-74x77\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.781236 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778454 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-tmp\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.781236 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778481 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-node-log\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.781236 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778505 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/107bc8a5-c01b-4b78-aa29-765aab336604-dbus\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:18.781236 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778533 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-registration-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.781236 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778505 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-systemd\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.781236 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778674 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f749cb9-f4b4-4db4-a580-2877108e718c-ovnkube-script-lib\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.781236 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778729 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-var-lib-kubelet\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.781236 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778747 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-run-netns\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.781236 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778817 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-sysctl-d\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.781236 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778885 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f749cb9-f4b4-4db4-a580-2877108e718c-env-overrides\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.781236 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.778898 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-var-lib-kubelet\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.781236 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.779040 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/913d7a7f-bba5-4c75-bd3d-1727790ca220-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.781236 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.779049 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/913d7a7f-bba5-4c75-bd3d-1727790ca220-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.781236 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.779082 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-etc-openvswitch\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.781236 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.779107 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-node-log\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.782023 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.779159 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-host\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.782023 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.779164 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-sysconfig\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.782023 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.779215 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/107bc8a5-c01b-4b78-aa29-765aab336604-dbus\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:18.782023 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.779263 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-lib-modules\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.782023 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.779275 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-kubernetes\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.782023 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.779301 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-run-netns\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.782023 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.779311 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/913d7a7f-bba5-4c75-bd3d-1727790ca220-cnibin\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.782023 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.779342 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b33f658-c455-4aa4-ab62-7b5e3de65be5-hosts-file\") pod \"node-resolver-86gjv\" (UID: \"8b33f658-c455-4aa4-ab62-7b5e3de65be5\") " pod="openshift-dns/node-resolver-86gjv" Oct 30 02:40:18.782023 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.779541 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb-konnectivity-ca\") pod \"konnectivity-agent-xkkvm\" (UID: \"6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb\") " pod="kube-system/konnectivity-agent-xkkvm" Oct 30 02:40:18.782023 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.779686 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1-serviceca\") pod \"node-ca-s5lq6\" (UID: \"e754f0c3-6ac7-4e3e-ab0e-c343b673abc1\") " pod="openshift-image-registry/node-ca-s5lq6" Oct 30 02:40:18.782819 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.782795 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-tmp\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.782968 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.782837 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-tuned\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.782968 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.782876 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f749cb9-f4b4-4db4-a580-2877108e718c-ovn-node-metrics-cert\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.783089 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.783012 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb-agent-certs\") pod \"konnectivity-agent-xkkvm\" (UID: \"6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb\") " pod="kube-system/konnectivity-agent-xkkvm" Oct 30 02:40:18.797983 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.797959 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq248\" (UniqueName: \"kubernetes.io/projected/5f79e66d-ae26-455c-b111-82f211ffe507-kube-api-access-xq248\") pod \"network-metrics-daemon-xg5ng\" (UID: \"5f79e66d-ae26-455c-b111-82f211ffe507\") " pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:18.798144 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.798118 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89h27\" (UniqueName: \"kubernetes.io/projected/48721522-308a-4472-b687-d08caa72978d-kube-api-access-89h27\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.799209 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.799187 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stmvr\" (UniqueName: \"kubernetes.io/projected/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1-kube-api-access-stmvr\") pod \"node-ca-s5lq6\" (UID: \"e754f0c3-6ac7-4e3e-ab0e-c343b673abc1\") " pod="openshift-image-registry/node-ca-s5lq6" Oct 30 02:40:18.804598 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:18.804576 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 02:40:18.804598 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:18.804601 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 02:40:18.804760 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:18.804626 2573 projected.go:196] Error preparing data for projected volume kube-api-access-7d98p for pod openshift-network-diagnostics/network-check-target-t765c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:18.804760 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:18.804702 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p podName:fb1a01bf-dc4f-430b-8b6d-d5569f85c981 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:19.304682755 +0000 UTC m=+3.167721892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7d98p" (UniqueName: "kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p") pod "network-check-target-t765c" (UID: "fb1a01bf-dc4f-430b-8b6d-d5569f85c981") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:18.807160 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.807139 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm8sz\" (UniqueName: \"kubernetes.io/projected/5077c63e-73b2-4a23-95f0-51edd0875a47-kube-api-access-jm8sz\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.807718 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.807446 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bflr\" (UniqueName: \"kubernetes.io/projected/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-kube-api-access-6bflr\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.808018 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.807996 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8bsm\" (UniqueName: \"kubernetes.io/projected/8b33f658-c455-4aa4-ab62-7b5e3de65be5-kube-api-access-v8bsm\") pod \"node-resolver-86gjv\" (UID: \"8b33f658-c455-4aa4-ab62-7b5e3de65be5\") " pod="openshift-dns/node-resolver-86gjv" Oct 30 02:40:18.808093 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.808060 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stdbm\" (UniqueName: \"kubernetes.io/projected/7f749cb9-f4b4-4db4-a580-2877108e718c-kube-api-access-stdbm\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.811796 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.811774 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74x77\" (UniqueName: \"kubernetes.io/projected/913d7a7f-bba5-4c75-bd3d-1727790ca220-kube-api-access-74x77\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.812053 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.812037 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8x7w\" (UniqueName: \"kubernetes.io/projected/fb7cc1ad-e964-4602-9ce5-c838212826d2-kube-api-access-g8x7w\") pod \"iptables-alerter-llk8x\" (UID: \"fb7cc1ad-e964-4602-9ce5-c838212826d2\") " pod="openshift-network-operator/iptables-alerter-llk8x" Oct 30 02:40:18.892411 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.892376 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s5lq6" Oct 30 02:40:18.903792 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:18.903761 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode754f0c3_6ac7_4e3e_ab0e_c343b673abc1.slice/crio-9a4b4cb3bc647cf0fc8c9caf8163268b564c3c437487f98c820765b01fa125db WatchSource:0}: Error finding container 9a4b4cb3bc647cf0fc8c9caf8163268b564c3c437487f98c820765b01fa125db: Status 404 returned error can't find the container with id 9a4b4cb3bc647cf0fc8c9caf8163268b564c3c437487f98c820765b01fa125db Oct 30 02:40:18.918738 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.918714 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:40:18.926734 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.926713 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:40:18.926992 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:18.926969 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913d7a7f_bba5_4c75_bd3d_1727790ca220.slice/crio-b00353db60ca1c907bcb72884ec92d7cc53c6fbc767b74875275535caab0190d WatchSource:0}: Error finding container b00353db60ca1c907bcb72884ec92d7cc53c6fbc767b74875275535caab0190d: Status 404 returned error can't find the container with id b00353db60ca1c907bcb72884ec92d7cc53c6fbc767b74875275535caab0190d Oct 30 02:40:18.927519 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.927478 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-llk8x" Oct 30 02:40:18.931509 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.931486 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:40:18.939013 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.938994 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xkkvm" Oct 30 02:40:18.961952 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.961927 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-86gjv" Oct 30 02:40:18.962099 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.962072 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d7vl2" Oct 30 02:40:18.964735 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:18.964688 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5077c63e_73b2_4a23_95f0_51edd0875a47.slice/crio-04c75f0ca595be3dd141be5bdb280111218a73c872d38ce9aabd85c9b0055303 WatchSource:0}: Error finding container 04c75f0ca595be3dd141be5bdb280111218a73c872d38ce9aabd85c9b0055303: Status 404 returned error can't find the container with id 04c75f0ca595be3dd141be5bdb280111218a73c872d38ce9aabd85c9b0055303 Oct 30 02:40:18.965693 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:18.965678 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:18.966427 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:18.966335 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7cc1ad_e964_4602_9ce5_c838212826d2.slice/crio-77b23e1fbe8c6d00890c298ddb1d734a5cda96d8d22d38085a5fe134dc0cb2e3 WatchSource:0}: Error finding container 77b23e1fbe8c6d00890c298ddb1d734a5cda96d8d22d38085a5fe134dc0cb2e3: Status 404 returned error can't find the container with id 77b23e1fbe8c6d00890c298ddb1d734a5cda96d8d22d38085a5fe134dc0cb2e3 Oct 30 02:40:18.968037 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:18.968013 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2896c8_e75c_4565_b5fe_3320f26fdd8a.slice/crio-dbdec34d8f0e60ebd7275d2454f2b60d39cb162589560f3fae2e56e5c8c1c578 WatchSource:0}: Error finding container dbdec34d8f0e60ebd7275d2454f2b60d39cb162589560f3fae2e56e5c8c1c578: Status 404 returned error can't find the container with id dbdec34d8f0e60ebd7275d2454f2b60d39cb162589560f3fae2e56e5c8c1c578 Oct 30 02:40:18.969587 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:18.969563 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9c24e9_4f7d_4a79_9459_a3e2cf1129bb.slice/crio-e65862c0dd5dda56c009b98387b19e171a3c6cc152f0395933c41dbca985b313 WatchSource:0}: Error finding container e65862c0dd5dda56c009b98387b19e171a3c6cc152f0395933c41dbca985b313: Status 404 returned error can't find the container with id e65862c0dd5dda56c009b98387b19e171a3c6cc152f0395933c41dbca985b313 Oct 30 02:40:18.977982 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:18.977938 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b33f658_c455_4aa4_ab62_7b5e3de65be5.slice/crio-ccbdacb6e094ffeacd537808abbcbacea482495ffdccda28718be721036e7aad WatchSource:0}: Error finding container ccbdacb6e094ffeacd537808abbcbacea482495ffdccda28718be721036e7aad: Status 404 returned error can't find the container with id ccbdacb6e094ffeacd537808abbcbacea482495ffdccda28718be721036e7aad Oct 30 02:40:18.981931 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:18.981898 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f749cb9_f4b4_4db4_a580_2877108e718c.slice/crio-d1b4d081681ce4b2aad6ac3cec34e6c4985e180ce1cec7b99aa5cea2831de066 WatchSource:0}: Error finding container d1b4d081681ce4b2aad6ac3cec34e6c4985e180ce1cec7b99aa5cea2831de066: Status 404 returned error can't find the container with id d1b4d081681ce4b2aad6ac3cec34e6c4985e180ce1cec7b99aa5cea2831de066 Oct 30 02:40:18.983611 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:40:18.983588 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48721522_308a_4472_b687_d08caa72978d.slice/crio-4496a3998e096416a749b67d66e8be8e63555a21349198a0e13abd6e02d41f42 WatchSource:0}: Error finding container 4496a3998e096416a749b67d66e8be8e63555a21349198a0e13abd6e02d41f42: Status 404 returned error can't find the container with id 4496a3998e096416a749b67d66e8be8e63555a21349198a0e13abd6e02d41f42 Oct 30 02:40:19.013945 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:19.013920 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:40:19.281861 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:19.281612 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:19.281992 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:19.281892 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs\") pod \"network-metrics-daemon-xg5ng\" (UID: \"5f79e66d-ae26-455c-b111-82f211ffe507\") " pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:19.281992 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:19.281729 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:19.282093 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:19.282001 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:19.282093 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:19.282004 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret podName:107bc8a5-c01b-4b78-aa29-765aab336604 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:20.281983064 +0000 UTC m=+4.145022204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret") pod "global-pull-secret-syncer-cd6rj" (UID: "107bc8a5-c01b-4b78-aa29-765aab336604") : object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:19.282093 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:19.282064 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs podName:5f79e66d-ae26-455c-b111-82f211ffe507 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:20.282047833 +0000 UTC m=+4.145086959 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs") pod "network-metrics-daemon-xg5ng" (UID: "5f79e66d-ae26-455c-b111-82f211ffe507") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:19.382818 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:19.382780 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d98p\" (UniqueName: \"kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p\") pod \"network-check-target-t765c\" (UID: \"fb1a01bf-dc4f-430b-8b6d-d5569f85c981\") " pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:19.383016 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:19.382993 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 02:40:19.383078 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:19.383018 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 02:40:19.383078 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:19.383031 2573 projected.go:196] Error preparing data for projected volume kube-api-access-7d98p for pod openshift-network-diagnostics/network-check-target-t765c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:19.383151 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:19.383093 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p podName:fb1a01bf-dc4f-430b-8b6d-d5569f85c981 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:20.383074205 +0000 UTC m=+4.246113327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7d98p" (UniqueName: "kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p") pod "network-check-target-t765c" (UID: "fb1a01bf-dc4f-430b-8b6d-d5569f85c981") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:19.616136 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:19.615999 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:35:18 +0000 UTC" deadline="2028-05-03 12:53:29.343797057 +0000 UTC" Oct 30 02:40:19.616136 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:19.616061 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13234h13m9.727740515s" Oct 30 02:40:19.720251 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:19.720215 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:19.720426 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:19.720359 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:19.732793 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:19.732760 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-86gjv" event={"ID":"8b33f658-c455-4aa4-ab62-7b5e3de65be5","Type":"ContainerStarted","Data":"ccbdacb6e094ffeacd537808abbcbacea482495ffdccda28718be721036e7aad"} Oct 30 02:40:19.734390 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:19.734366 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" event={"ID":"9d2896c8-e75c-4565-b5fe-3320f26fdd8a","Type":"ContainerStarted","Data":"dbdec34d8f0e60ebd7275d2454f2b60d39cb162589560f3fae2e56e5c8c1c578"} Oct 30 02:40:19.735877 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:19.735852 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-llk8x" event={"ID":"fb7cc1ad-e964-4602-9ce5-c838212826d2","Type":"ContainerStarted","Data":"77b23e1fbe8c6d00890c298ddb1d734a5cda96d8d22d38085a5fe134dc0cb2e3"} Oct 30 02:40:19.741255 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:19.741231 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" event={"ID":"5077c63e-73b2-4a23-95f0-51edd0875a47","Type":"ContainerStarted","Data":"04c75f0ca595be3dd141be5bdb280111218a73c872d38ce9aabd85c9b0055303"} Oct 30 02:40:19.742823 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:19.742733 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttd2z" event={"ID":"913d7a7f-bba5-4c75-bd3d-1727790ca220","Type":"ContainerStarted","Data":"b00353db60ca1c907bcb72884ec92d7cc53c6fbc767b74875275535caab0190d"} Oct 30 02:40:19.748756 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:19.748019 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-200.ec2.internal" event={"ID":"09b727cb32d7ac800f6bb9bd152bad1f","Type":"ContainerStarted","Data":"25357029410b453a1058c5e40df1945c3e08fed76b8509c54c929332a079da9a"} Oct 30 02:40:19.754637 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:19.754557 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xkkvm" event={"ID":"6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb","Type":"ContainerStarted","Data":"e65862c0dd5dda56c009b98387b19e171a3c6cc152f0395933c41dbca985b313"} Oct 30 02:40:19.759119 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:19.759091 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s5lq6" event={"ID":"e754f0c3-6ac7-4e3e-ab0e-c343b673abc1","Type":"ContainerStarted","Data":"9a4b4cb3bc647cf0fc8c9caf8163268b564c3c437487f98c820765b01fa125db"} Oct 30 02:40:19.762210 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:19.762185 2573 generic.go:356] "Generic (PLEG): container finished" podID="6496276491b4aff226133cbf4e8c9798" containerID="4972329a555a16e7f846265b3e466e660d22139f947c63691d4259b3b469346c" exitCode=0 Oct 30 02:40:19.762315 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:19.762278 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" event={"ID":"6496276491b4aff226133cbf4e8c9798","Type":"ContainerDied","Data":"4972329a555a16e7f846265b3e466e660d22139f947c63691d4259b3b469346c"} Oct 30 02:40:19.769099 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:19.766547 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" event={"ID":"7f749cb9-f4b4-4db4-a580-2877108e718c","Type":"ContainerStarted","Data":"d1b4d081681ce4b2aad6ac3cec34e6c4985e180ce1cec7b99aa5cea2831de066"} Oct 30 02:40:19.769099 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:19.768432 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d7vl2" event={"ID":"48721522-308a-4472-b687-d08caa72978d","Type":"ContainerStarted","Data":"4496a3998e096416a749b67d66e8be8e63555a21349198a0e13abd6e02d41f42"} Oct 30 02:40:19.803429 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:19.803364 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-200.ec2.internal" podStartSLOduration=2.803347658 podStartE2EDuration="2.803347658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:19.8029452 +0000 UTC m=+3.665984364" watchObservedRunningTime="2026-10-30 02:40:19.803347658 +0000 UTC m=+3.666386798" Oct 30 02:40:20.291636 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:20.291356 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:20.291636 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:20.291409 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs\") pod \"network-metrics-daemon-xg5ng\" (UID: \"5f79e66d-ae26-455c-b111-82f211ffe507\") " pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:20.291636 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:20.291517 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:20.291636 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:20.291529 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:20.291636 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:20.291585 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret podName:107bc8a5-c01b-4b78-aa29-765aab336604 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:22.291565651 +0000 UTC m=+6.154604791 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret") pod "global-pull-secret-syncer-cd6rj" (UID: "107bc8a5-c01b-4b78-aa29-765aab336604") : object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:20.291636 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:20.291606 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs podName:5f79e66d-ae26-455c-b111-82f211ffe507 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:22.291596276 +0000 UTC m=+6.154635396 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs") pod "network-metrics-daemon-xg5ng" (UID: "5f79e66d-ae26-455c-b111-82f211ffe507") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:20.391786 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:20.391753 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d98p\" (UniqueName: \"kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p\") pod \"network-check-target-t765c\" (UID: \"fb1a01bf-dc4f-430b-8b6d-d5569f85c981\") " pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:20.391944 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:20.391923 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 02:40:20.391944 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:20.391943 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 02:40:20.392046 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:20.391956 2573 projected.go:196] Error preparing data for projected volume kube-api-access-7d98p for pod openshift-network-diagnostics/network-check-target-t765c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:20.392046 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:20.392010 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p podName:fb1a01bf-dc4f-430b-8b6d-d5569f85c981 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:22.391991954 +0000 UTC m=+6.255031078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7d98p" (UniqueName: "kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p") pod "network-check-target-t765c" (UID: "fb1a01bf-dc4f-430b-8b6d-d5569f85c981") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:20.616328 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:20.616226 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:35:18 +0000 UTC" deadline="2028-07-22 10:15:01.027410638 +0000 UTC" Oct 30 02:40:20.616328 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:20.616285 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15151h34m40.411129384s" Oct 30 02:40:20.727350 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:20.727076 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:20.727350 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:20.727204 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:20.727350 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:20.727240 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:20.727350 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:20.727313 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:20.780869 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:20.780795 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" event={"ID":"6496276491b4aff226133cbf4e8c9798","Type":"ContainerStarted","Data":"48d489eb7b5c2cafc71f1bed529d66e769a95d60df16f4b548c1743f77c59de9"} Oct 30 02:40:21.720019 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:21.719985 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:21.720410 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:21.720134 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:21.797347 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:21.795976 2573 generic.go:356] "Generic (PLEG): container finished" podID="913d7a7f-bba5-4c75-bd3d-1727790ca220" containerID="c9d85b1f0972e6497edfa002179a68141edbda4bf07cbe91884ac3111fb36f11" exitCode=0 Oct 30 02:40:21.797347 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:21.797108 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttd2z" event={"ID":"913d7a7f-bba5-4c75-bd3d-1727790ca220","Type":"ContainerDied","Data":"c9d85b1f0972e6497edfa002179a68141edbda4bf07cbe91884ac3111fb36f11"} Oct 30 02:40:21.866393 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:21.866331 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" podStartSLOduration=4.86630919 podStartE2EDuration="4.86630919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:20.818500459 +0000 UTC m=+4.681539597" watchObservedRunningTime="2026-10-30 02:40:21.86630919 +0000 UTC m=+5.729348329" Oct 30 02:40:22.306607 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:22.306572 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:22.306838 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:22.306632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs\") pod \"network-metrics-daemon-xg5ng\" (UID: \"5f79e66d-ae26-455c-b111-82f211ffe507\") " pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:22.306838 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:22.306762 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:22.306838 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:22.306818 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs podName:5f79e66d-ae26-455c-b111-82f211ffe507 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:26.306802458 +0000 UTC m=+10.169841576 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs") pod "network-metrics-daemon-xg5ng" (UID: "5f79e66d-ae26-455c-b111-82f211ffe507") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:22.307218 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:22.307191 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:22.307302 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:22.307242 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret podName:107bc8a5-c01b-4b78-aa29-765aab336604 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:26.307227841 +0000 UTC m=+10.170266960 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret") pod "global-pull-secret-syncer-cd6rj" (UID: "107bc8a5-c01b-4b78-aa29-765aab336604") : object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:22.407884 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:22.407229 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d98p\" (UniqueName: \"kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p\") pod \"network-check-target-t765c\" (UID: \"fb1a01bf-dc4f-430b-8b6d-d5569f85c981\") " pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:22.407884 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:22.407408 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 02:40:22.407884 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:22.407429 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 02:40:22.407884 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:22.407441 2573 projected.go:196] Error preparing data for projected volume kube-api-access-7d98p for pod openshift-network-diagnostics/network-check-target-t765c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:22.407884 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:22.407496 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p podName:fb1a01bf-dc4f-430b-8b6d-d5569f85c981 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:26.407480036 +0000 UTC m=+10.270519156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7d98p" (UniqueName: "kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p") pod "network-check-target-t765c" (UID: "fb1a01bf-dc4f-430b-8b6d-d5569f85c981") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:22.723064 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:22.722399 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:22.723064 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:22.722525 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:22.723064 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:22.722899 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:22.723064 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:22.722992 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:23.719987 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:23.719439 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:23.719987 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:23.719586 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:23.803401 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:23.803356 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xkkvm" event={"ID":"6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb","Type":"ContainerStarted","Data":"543bdee88b4ca2c7f6837f247d5dd4e9285714524671950c3905ae6b6e557e5f"} Oct 30 02:40:23.807797 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:23.807758 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s5lq6" event={"ID":"e754f0c3-6ac7-4e3e-ab0e-c343b673abc1","Type":"ContainerStarted","Data":"87c9b7496ea3d56a6dd0d0f4fdb2ddafb85b5db84673a8e2e0fabd09c7ee4867"} Oct 30 02:40:23.841580 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:23.841442 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xkkvm" podStartSLOduration=4.125767594 podStartE2EDuration="7.841428887s" totalImagesPullingTime="3.715661293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:23.840502948 +0000 UTC m=+7.703542087" watchObservedRunningTime="2026-10-30 02:40:23.841428887 +0000 UTC m=+7.704468024" Oct 30 02:40:24.324887 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:24.324853 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xkkvm" Oct 30 02:40:24.722219 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:24.722187 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:24.722384 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:24.722300 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:24.722384 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:24.722371 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:24.722489 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:24.722427 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:24.818547 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:24.818516 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" event={"ID":"5077c63e-73b2-4a23-95f0-51edd0875a47","Type":"ContainerStarted","Data":"d8e28d987c33e3fcc59317c00a5830415389933b11327dec0eb403f857cc309e"} Oct 30 02:40:25.720669 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:25.720325 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:25.720669 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:25.720472 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:26.153165 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:26.153103 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xkkvm" Oct 30 02:40:26.153942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:26.153765 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xkkvm" Oct 30 02:40:26.191607 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:26.191545 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s5lq6" podStartSLOduration=5.627927484 podStartE2EDuration="10.191527549s" totalImagesPullingTime="4.563600065s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:23.87593201 +0000 UTC m=+7.738971148" watchObservedRunningTime="2026-10-30 02:40:26.191527549 +0000 UTC m=+10.054566693" Oct 30 02:40:26.276367 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:26.276337 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 02:40:26.348644 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:26.346679 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:26.348644 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:26.346724 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs\") pod \"network-metrics-daemon-xg5ng\" (UID: \"5f79e66d-ae26-455c-b111-82f211ffe507\") " pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:26.348644 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:26.346869 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:26.348644 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:26.346927 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs podName:5f79e66d-ae26-455c-b111-82f211ffe507 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:34.346909545 +0000 UTC m=+18.209948664 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs") pod "network-metrics-daemon-xg5ng" (UID: "5f79e66d-ae26-455c-b111-82f211ffe507") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:26.348644 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:26.347312 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:26.348644 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:26.347362 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret podName:107bc8a5-c01b-4b78-aa29-765aab336604 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:34.347347084 +0000 UTC m=+18.210386205 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret") pod "global-pull-secret-syncer-cd6rj" (UID: "107bc8a5-c01b-4b78-aa29-765aab336604") : object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:26.447481 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:26.447447 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d98p\" (UniqueName: \"kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p\") pod \"network-check-target-t765c\" (UID: \"fb1a01bf-dc4f-430b-8b6d-d5569f85c981\") " pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:26.447669 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:26.447648 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 02:40:26.447733 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:26.447673 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 02:40:26.447733 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:26.447686 2573 projected.go:196] Error preparing data for projected volume kube-api-access-7d98p for pod openshift-network-diagnostics/network-check-target-t765c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:26.447818 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:26.447753 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p podName:fb1a01bf-dc4f-430b-8b6d-d5569f85c981 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:34.447733573 +0000 UTC m=+18.310772705 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7d98p" (UniqueName: "kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p") pod "network-check-target-t765c" (UID: "fb1a01bf-dc4f-430b-8b6d-d5569f85c981") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:26.631465 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:26.630710 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:40:26.276357043Z","UUID":"05866ab7-6a76-48a2-afbd-9893665c1c2e","Handler":null,"Name":"","Endpoint":""} Oct 30 02:40:26.632769 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:26.632745 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 02:40:26.632904 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:26.632778 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 02:40:26.721082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:26.721054 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:26.721199 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:26.721169 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:26.721254 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:26.721218 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:26.721367 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:26.721346 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:26.825478 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:26.825449 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d7vl2" event={"ID":"48721522-308a-4472-b687-d08caa72978d","Type":"ContainerStarted","Data":"87c98ab68d21420bcd90f7f9beedf6ef0fff86b4b4320e662f8ffe109cff681c"} Oct 30 02:40:26.828633 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:26.827441 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-86gjv" event={"ID":"8b33f658-c455-4aa4-ab62-7b5e3de65be5","Type":"ContainerStarted","Data":"d5d60c59749fd9ae42b03f8b12ee8b3cde378728a596da90bc96df4e9e1406a2"} Oct 30 02:40:26.830511 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:26.830488 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" event={"ID":"5077c63e-73b2-4a23-95f0-51edd0875a47","Type":"ContainerStarted","Data":"01c0ab6a2db20f013bc5912f4032030350f7d4139c50c3c3cb9a5f6472a301e5"} Oct 30 02:40:26.833251 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:26.833215 2573 generic.go:356] "Generic (PLEG): container finished" podID="913d7a7f-bba5-4c75-bd3d-1727790ca220" containerID="8f24fc7ec6ab978325ed17f02a396648cb53e63d0430eb73ce3c97e15df3db44" exitCode=0 Oct 30 02:40:26.834190 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:26.834167 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttd2z" event={"ID":"913d7a7f-bba5-4c75-bd3d-1727790ca220","Type":"ContainerDied","Data":"8f24fc7ec6ab978325ed17f02a396648cb53e63d0430eb73ce3c97e15df3db44"} Oct 30 02:40:26.834754 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:26.834727 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xkkvm" Oct 30 02:40:26.911328 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:26.909730 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-d7vl2" podStartSLOduration=3.606109291 podStartE2EDuration="10.909708677s" totalImagesPullingTime="7.303599386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:26.86543352 +0000 UTC m=+10.728472658" watchObservedRunningTime="2026-10-30 02:40:26.909708677 +0000 UTC m=+10.772747830" Oct 30 02:40:26.982704 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:26.982642 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-86gjv" podStartSLOduration=3.355080913 podStartE2EDuration="10.982605574s" totalImagesPullingTime="7.627524661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:26.982448949 +0000 UTC m=+10.845488114" watchObservedRunningTime="2026-10-30 02:40:26.982605574 +0000 UTC m=+10.845644713" Oct 30 02:40:27.719532 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:27.719496 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:27.720072 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:27.719645 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:27.838490 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:27.838419 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" event={"ID":"5077c63e-73b2-4a23-95f0-51edd0875a47","Type":"ContainerStarted","Data":"28581b4520fdfde92be33cafa3fc7e8d1bc56ac9a3a4921e934833fe0e67584b"} Oct 30 02:40:27.841539 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:27.841512 2573 generic.go:356] "Generic (PLEG): container finished" podID="913d7a7f-bba5-4c75-bd3d-1727790ca220" containerID="e62d03f5c935645288cc58e84e92bc4c4f8f8cc54f1c2f3535c11e67d8bd2ea8" exitCode=0 Oct 30 02:40:27.842254 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:27.842229 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttd2z" event={"ID":"913d7a7f-bba5-4c75-bd3d-1727790ca220","Type":"ContainerDied","Data":"e62d03f5c935645288cc58e84e92bc4c4f8f8cc54f1c2f3535c11e67d8bd2ea8"} Oct 30 02:40:27.877800 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:27.877746 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" podStartSLOduration=3.93817854 podStartE2EDuration="11.87773072s" totalImagesPullingTime="7.93955218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:27.877611099 +0000 UTC m=+11.740650241" watchObservedRunningTime="2026-10-30 02:40:27.87773072 +0000 UTC m=+11.740769860" Oct 30 02:40:28.720366 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:28.720335 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:28.720904 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:28.720380 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:28.720904 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:28.720483 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:28.720904 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:28.720652 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:28.845817 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:28.845774 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-llk8x" event={"ID":"fb7cc1ad-e964-4602-9ce5-c838212826d2","Type":"ContainerStarted","Data":"ed094eac6debef3e126bb970b9e923d852bb5f4a92f1db99e0c301236966cc84"} Oct 30 02:40:28.849370 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:28.849341 2573 generic.go:356] "Generic (PLEG): container finished" podID="913d7a7f-bba5-4c75-bd3d-1727790ca220" containerID="abd59f82eb8c5ba752fb7ac5a968802829ab0c3e341593d05d2df4a4d628fe64" exitCode=0 Oct 30 02:40:28.849501 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:28.849419 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttd2z" event={"ID":"913d7a7f-bba5-4c75-bd3d-1727790ca220","Type":"ContainerDied","Data":"abd59f82eb8c5ba752fb7ac5a968802829ab0c3e341593d05d2df4a4d628fe64"} Oct 30 02:40:28.882263 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:28.882214 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-llk8x" podStartSLOduration=5.030648254 podStartE2EDuration="12.882196077s" totalImagesPullingTime="7.851547823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:28.882122253 +0000 UTC m=+12.745161393" watchObservedRunningTime="2026-10-30 02:40:28.882196077 +0000 UTC m=+12.745235218" Oct 30 02:40:29.719458 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:29.719421 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:29.719646 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:29.719561 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:29.853981 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:29.853948 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" event={"ID":"9d2896c8-e75c-4565-b5fe-3320f26fdd8a","Type":"ContainerStarted","Data":"d1294274c9a0668eeb87c5b4bab0e60431e90ecd1e1ee47388da264638d70fc1"} Oct 30 02:40:29.894440 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:29.894381 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" podStartSLOduration=4.086966772 podStartE2EDuration="13.894362316s" totalImagesPullingTime="9.807395544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:29.893644918 +0000 UTC m=+13.756684058" watchObservedRunningTime="2026-10-30 02:40:29.894362316 +0000 UTC m=+13.757401457" Oct 30 02:40:30.719761 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:30.719728 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:30.719923 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:30.719858 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:30.719923 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:30.719896 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:30.720061 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:30.720037 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:30.859486 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:30.859212 2573 generic.go:356] "Generic (PLEG): container finished" podID="913d7a7f-bba5-4c75-bd3d-1727790ca220" containerID="b37278b8239e3875c96e11b030f206b58b509b5f29a246aa5ef64b3a5e93386b" exitCode=0 Oct 30 02:40:30.859930 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:30.859383 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttd2z" event={"ID":"913d7a7f-bba5-4c75-bd3d-1727790ca220","Type":"ContainerDied","Data":"b37278b8239e3875c96e11b030f206b58b509b5f29a246aa5ef64b3a5e93386b"} Oct 30 02:40:31.719812 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:31.719781 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:31.720004 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:31.719917 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:31.865516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:31.865485 2573 generic.go:356] "Generic (PLEG): container finished" podID="913d7a7f-bba5-4c75-bd3d-1727790ca220" containerID="d4b1947d35f3f81b39f13f8975784b7523dfb1bf2a4f4ee8ddb35f734cd17409" exitCode=0 Oct 30 02:40:31.865962 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:31.865539 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttd2z" event={"ID":"913d7a7f-bba5-4c75-bd3d-1727790ca220","Type":"ContainerDied","Data":"d4b1947d35f3f81b39f13f8975784b7523dfb1bf2a4f4ee8ddb35f734cd17409"} Oct 30 02:40:32.719986 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:32.719960 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:32.720073 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:32.719995 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:32.720106 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:32.720074 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:32.720207 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:32.720182 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:32.874093 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:32.874060 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttd2z" event={"ID":"913d7a7f-bba5-4c75-bd3d-1727790ca220","Type":"ContainerStarted","Data":"a444fd0045b24fd46967e0ee84f10c978b56668b770edb0d23960363d36a4a9b"} Oct 30 02:40:32.875644 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:32.875608 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 02:40:32.875947 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:32.875926 2573 generic.go:356] "Generic (PLEG): container finished" podID="7f749cb9-f4b4-4db4-a580-2877108e718c" containerID="5a2ae849a76af523e6d52f9d57c5b9ec835538192fd42df0b6ae7fe9d16cedc8" exitCode=1 Oct 30 02:40:32.876020 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:32.875959 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" event={"ID":"7f749cb9-f4b4-4db4-a580-2877108e718c","Type":"ContainerDied","Data":"5a2ae849a76af523e6d52f9d57c5b9ec835538192fd42df0b6ae7fe9d16cedc8"} Oct 30 02:40:32.876020 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:32.876004 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" event={"ID":"7f749cb9-f4b4-4db4-a580-2877108e718c","Type":"ContainerStarted","Data":"1e5e8c87f47dff67ddb5d7fa45e63a1a763dfa4d97ccab95714b8666bd54751c"} Oct 30 02:40:32.938317 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:32.938164 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ttd2z" podStartSLOduration=6.871630795 podStartE2EDuration="16.93815186s" totalImagesPullingTime="10.066521065s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:32.927333503 +0000 UTC m=+16.790372642" watchObservedRunningTime="2026-10-30 02:40:32.93815186 +0000 UTC m=+16.801190998" Oct 30 02:40:33.719688 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:33.719657 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:33.719889 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:33.719770 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:33.880670 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:33.880645 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 02:40:33.881161 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:33.881036 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" event={"ID":"7f749cb9-f4b4-4db4-a580-2877108e718c","Type":"ContainerStarted","Data":"db31bd1d7e9cbaa750653c0b7413a164ef916391157b37b2c12aae999525524f"} Oct 30 02:40:33.881161 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:33.881121 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" event={"ID":"7f749cb9-f4b4-4db4-a580-2877108e718c","Type":"ContainerStarted","Data":"0056fa5f2523e86200ab977cce16f4cb79c79765a4b687ea8ebbe21dd74733b2"} Oct 30 02:40:33.881283 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:33.881167 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" event={"ID":"7f749cb9-f4b4-4db4-a580-2877108e718c","Type":"ContainerStarted","Data":"0297cba653e0fc9b083314822c27e89ed87d7ec5f72f05f29077aa4b6c2c38fc"} Oct 30 02:40:33.881283 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:33.881223 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" event={"ID":"7f749cb9-f4b4-4db4-a580-2877108e718c","Type":"ContainerStarted","Data":"b50e464b1cf54b9ee5bba5c5107d27c7bb36818cb3fef57283ae3858357a36ba"} Oct 30 02:40:34.404886 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:34.404848 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:34.404886 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:34.404891 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs\") pod \"network-metrics-daemon-xg5ng\" (UID: \"5f79e66d-ae26-455c-b111-82f211ffe507\") " pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:34.405101 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:34.404995 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:34.405101 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:34.405007 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:34.405101 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:34.405049 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs podName:5f79e66d-ae26-455c-b111-82f211ffe507 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:50.405036045 +0000 UTC m=+34.268075163 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs") pod "network-metrics-daemon-xg5ng" (UID: "5f79e66d-ae26-455c-b111-82f211ffe507") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:34.405101 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:34.405064 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret podName:107bc8a5-c01b-4b78-aa29-765aab336604 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:50.405056373 +0000 UTC m=+34.268095490 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret") pod "global-pull-secret-syncer-cd6rj" (UID: "107bc8a5-c01b-4b78-aa29-765aab336604") : object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:34.505723 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:34.505684 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d98p\" (UniqueName: \"kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p\") pod \"network-check-target-t765c\" (UID: \"fb1a01bf-dc4f-430b-8b6d-d5569f85c981\") " pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:34.505858 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:34.505835 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 02:40:34.505858 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:34.505851 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 02:40:34.505937 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:34.505861 2573 projected.go:196] Error preparing data for projected volume kube-api-access-7d98p for pod openshift-network-diagnostics/network-check-target-t765c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:34.505937 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:34.505910 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p podName:fb1a01bf-dc4f-430b-8b6d-d5569f85c981 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:50.505895957 +0000 UTC m=+34.368935079 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7d98p" (UniqueName: "kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p") pod "network-check-target-t765c" (UID: "fb1a01bf-dc4f-430b-8b6d-d5569f85c981") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:34.720242 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:34.720153 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:34.720391 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:34.720311 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:34.720391 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:34.720365 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:34.720508 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:34.720452 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:35.720398 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:35.720306 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:35.720869 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:35.720430 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:35.887817 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:35.887793 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 02:40:35.888127 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:35.888104 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" event={"ID":"7f749cb9-f4b4-4db4-a580-2877108e718c","Type":"ContainerStarted","Data":"5330f7ad9ea94316ef0065ceb1d1c7ab4c17568b5be938035ba104a56e81c81b"} Oct 30 02:40:36.720541 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:36.720510 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:36.720928 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:36.720586 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:36.720928 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:36.720686 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:36.720928 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:36.720816 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:37.719686 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:37.719447 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:37.719844 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:37.719733 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:37.895279 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:37.895248 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 02:40:37.895898 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:37.895576 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" event={"ID":"7f749cb9-f4b4-4db4-a580-2877108e718c","Type":"ContainerStarted","Data":"a888627efcf182e19d1d1489578bc3135ab088410c0355bef165918538559a6c"} Oct 30 02:40:37.895984 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:37.895926 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:37.896032 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:37.896020 2573 scope.go:122] "RemoveContainer" containerID="5a2ae849a76af523e6d52f9d57c5b9ec835538192fd42df0b6ae7fe9d16cedc8" Oct 30 02:40:37.911432 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:37.911414 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:38.719772 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:38.719741 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:38.719772 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:38.719775 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:38.719997 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:38.719850 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:38.720045 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:38.719986 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:38.901260 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:38.901232 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 02:40:38.901682 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:38.901596 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" event={"ID":"7f749cb9-f4b4-4db4-a580-2877108e718c","Type":"ContainerStarted","Data":"d5023d011867fecaa4ebdb4ace21f0e0e779f76ab3c74adc4bf71c79a3b4c586"} Oct 30 02:40:38.901818 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:38.901803 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:38.901887 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:38.901865 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:38.916434 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:38.916412 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:40:38.965397 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:38.965347 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" podStartSLOduration=9.270974107 podStartE2EDuration="22.965333625s" totalImagesPullingTime="13.694359518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:38.964186555 +0000 UTC m=+22.827225694" watchObservedRunningTime="2026-10-30 02:40:38.965333625 +0000 UTC m=+22.828372764" Oct 30 02:40:39.719398 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:39.719368 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:39.719568 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:39.719488 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:40.366854 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:40.366822 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-86gjv_8b33f658-c455-4aa4-ab62-7b5e3de65be5/dns-node-resolver/0.log" Oct 30 02:40:40.719930 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:40.719842 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:40.719930 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:40.719874 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:40.720125 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:40.719954 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:40.720125 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:40.720064 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:41.278216 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:41.278192 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s5lq6_e754f0c3-6ac7-4e3e-ab0e-c343b673abc1/node-ca/0.log" Oct 30 02:40:41.719915 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:41.719837 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:41.720268 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:41.719940 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:42.719901 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:42.719870 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:42.720091 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:42.719967 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:42.720091 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:42.720005 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:42.720091 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:42.720061 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:43.719358 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:43.719328 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:43.719608 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:43.719427 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:44.719881 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:44.719852 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:44.720254 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:44.719964 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:44.720254 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:44.720009 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:44.720254 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:44.720080 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:45.720034 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:45.720004 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:45.720415 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:45.720103 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:46.721056 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:46.721026 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:46.721422 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:46.721151 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:46.721422 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:46.721214 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:46.721422 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:46.721313 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:47.719697 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:47.719667 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:47.719879 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:47.719768 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:48.719874 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:48.719839 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:48.720382 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:48.719944 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:48.720382 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:48.720033 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:48.720545 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:48.720504 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:49.720377 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:49.720343 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:49.720802 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:49.720446 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:50.422499 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:50.422465 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:50.422499 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:50.422501 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs\") pod \"network-metrics-daemon-xg5ng\" (UID: \"5f79e66d-ae26-455c-b111-82f211ffe507\") " pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:50.422711 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:50.422609 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:50.422711 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:50.422669 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:50.422711 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:50.422710 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret podName:107bc8a5-c01b-4b78-aa29-765aab336604 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:22.422694972 +0000 UTC m=+66.285734094 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret") pod "global-pull-secret-syncer-cd6rj" (UID: "107bc8a5-c01b-4b78-aa29-765aab336604") : object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:50.422813 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:50.422725 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs podName:5f79e66d-ae26-455c-b111-82f211ffe507 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:22.422717985 +0000 UTC m=+66.285757101 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs") pod "network-metrics-daemon-xg5ng" (UID: "5f79e66d-ae26-455c-b111-82f211ffe507") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:50.523627 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:50.523577 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d98p\" (UniqueName: \"kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p\") pod \"network-check-target-t765c\" (UID: \"fb1a01bf-dc4f-430b-8b6d-d5569f85c981\") " pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:50.523775 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:50.523738 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 02:40:50.523775 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:50.523757 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 02:40:50.523775 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:50.523766 2573 projected.go:196] Error preparing data for projected volume kube-api-access-7d98p for pod openshift-network-diagnostics/network-check-target-t765c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:50.523871 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:50.523815 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p podName:fb1a01bf-dc4f-430b-8b6d-d5569f85c981 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:22.523804141 +0000 UTC m=+66.386843270 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7d98p" (UniqueName: "kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p") pod "network-check-target-t765c" (UID: "fb1a01bf-dc4f-430b-8b6d-d5569f85c981") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:50.720170 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:50.720088 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:50.720170 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:50.720111 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:50.720335 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:50.720192 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:50.720335 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:50.720250 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:51.720034 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:51.720001 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:51.720416 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:51.720106 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:52.720052 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:52.720020 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:52.720453 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:52.720028 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:52.720453 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:52.720119 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:52.720453 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:52.720188 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:53.720095 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:53.720057 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:53.720548 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:53.720189 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:54.719388 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:54.719353 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:54.719560 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:54.719490 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:54.719560 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:54.719531 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:54.719677 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:54.719600 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:55.720048 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:55.720015 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:55.720439 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:55.720142 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:56.721359 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:56.721323 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:56.721831 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:56.721439 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:56.721831 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:56.721490 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:56.721831 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:56.721654 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:57.720026 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:57.719994 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:57.720217 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:57.720118 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:58.719960 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:58.719926 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:58.719960 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:58.719957 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:58.720489 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:58.720027 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:58.720489 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:58.720134 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:40:59.365523 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:59.365286 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cd6rj"] Oct 30 02:40:59.365696 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:59.365653 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:40:59.365791 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:59.365765 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:40:59.373984 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:59.373952 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xg5ng"] Oct 30 02:40:59.374111 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:59.374085 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:40:59.374237 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:59.374215 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:40:59.374684 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:59.374662 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t765c"] Oct 30 02:40:59.374779 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:40:59.374739 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:40:59.374857 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:40:59.374834 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:41:00.720373 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:00.720295 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:41:00.720835 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:41:00.720399 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:41:00.720835 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:00.720436 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:41:00.720835 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:41:00.720481 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:41:01.720011 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:01.719968 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:41:01.720194 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:41:01.720082 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:41:02.722873 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:02.722845 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:41:02.723475 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:02.722873 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:41:02.723475 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:41:02.722958 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cd6rj" podUID="107bc8a5-c01b-4b78-aa29-765aab336604" Oct 30 02:41:02.723475 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:41:02.723083 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t765c" podUID="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" Oct 30 02:41:03.719830 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:03.719806 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:41:03.719937 ip-10-0-141-200 kubenswrapper[2573]: E1030 02:41:03.719919 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg5ng" podUID="5f79e66d-ae26-455c-b111-82f211ffe507" Oct 30 02:41:03.988019 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:03.987950 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeReady" Oct 30 02:41:03.988435 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:03.988084 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 02:41:04.105276 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.105238 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2xmlx"] Oct 30 02:41:04.107471 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.107449 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:04.108058 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.108036 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-q9hsv"] Oct 30 02:41:04.109809 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.109795 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q9hsv" Oct 30 02:41:04.131083 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.131060 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8td9f\"" Oct 30 02:41:04.131406 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.131384 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2xmlx"] Oct 30 02:41:04.136799 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.136780 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 02:41:04.136963 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.136945 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q9hsv"] Oct 30 02:41:04.140220 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.140199 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 02:41:04.140468 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.140435 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 02:41:04.140468 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.140459 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 02:41:04.140468 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.140461 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ksjrs\"" Oct 30 02:41:04.140693 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.140471 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 02:41:04.187975 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.187948 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tfqlk"] Oct 30 02:41:04.189819 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.189805 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:04.215871 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.215822 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tfqlk"] Oct 30 02:41:04.216025 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.216004 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-config-volume\") pod \"dns-default-2xmlx\" (UID: \"b2f64d21-ae78-4fdd-b2e5-83c7629e563a\") " pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:04.216083 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.216035 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be9a9474-03c1-489e-ab01-0e8e83d79daf-cert\") pod \"ingress-canary-q9hsv\" (UID: \"be9a9474-03c1-489e-ab01-0e8e83d79daf\") " pod="openshift-ingress-canary/ingress-canary-q9hsv" Oct 30 02:41:04.216083 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.216057 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84khn\" (UniqueName: \"kubernetes.io/projected/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-kube-api-access-84khn\") pod \"dns-default-2xmlx\" (UID: \"b2f64d21-ae78-4fdd-b2e5-83c7629e563a\") " pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:04.216183 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.216159 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-metrics-tls\") pod \"dns-default-2xmlx\" (UID: \"b2f64d21-ae78-4fdd-b2e5-83c7629e563a\") " pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:04.216233 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.216188 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-tmp-dir\") pod \"dns-default-2xmlx\" (UID: \"b2f64d21-ae78-4fdd-b2e5-83c7629e563a\") " pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:04.216233 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.216212 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77cp8\" (UniqueName: \"kubernetes.io/projected/be9a9474-03c1-489e-ab01-0e8e83d79daf-kube-api-access-77cp8\") pod \"ingress-canary-q9hsv\" (UID: \"be9a9474-03c1-489e-ab01-0e8e83d79daf\") " pod="openshift-ingress-canary/ingress-canary-q9hsv" Oct 30 02:41:04.229747 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.229727 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 02:41:04.229747 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.229736 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p7m4t\"" Oct 30 02:41:04.239191 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.239158 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 02:41:04.239236 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.239185 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 02:41:04.239427 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.239414 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 02:41:04.317502 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.317466 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e93a8c5-35f7-459b-a8f1-f282130116cb-crio-socket\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:04.317502 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.317506 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e93a8c5-35f7-459b-a8f1-f282130116cb-data-volume\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:04.317740 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.317531 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-metrics-tls\") pod \"dns-default-2xmlx\" (UID: \"b2f64d21-ae78-4fdd-b2e5-83c7629e563a\") " pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:04.317740 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.317604 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-tmp-dir\") pod \"dns-default-2xmlx\" (UID: \"b2f64d21-ae78-4fdd-b2e5-83c7629e563a\") " pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:04.317740 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.317652 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tccft\" (UniqueName: \"kubernetes.io/projected/7e93a8c5-35f7-459b-a8f1-f282130116cb-kube-api-access-tccft\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:04.317740 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.317672 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77cp8\" (UniqueName: \"kubernetes.io/projected/be9a9474-03c1-489e-ab01-0e8e83d79daf-kube-api-access-77cp8\") pod \"ingress-canary-q9hsv\" (UID: \"be9a9474-03c1-489e-ab01-0e8e83d79daf\") " pod="openshift-ingress-canary/ingress-canary-q9hsv" Oct 30 02:41:04.317880 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.317740 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-config-volume\") pod \"dns-default-2xmlx\" (UID: \"b2f64d21-ae78-4fdd-b2e5-83c7629e563a\") " pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:04.317880 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.317761 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e93a8c5-35f7-459b-a8f1-f282130116cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:04.317880 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.317778 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be9a9474-03c1-489e-ab01-0e8e83d79daf-cert\") pod \"ingress-canary-q9hsv\" (UID: \"be9a9474-03c1-489e-ab01-0e8e83d79daf\") " pod="openshift-ingress-canary/ingress-canary-q9hsv" Oct 30 02:41:04.317880 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.317795 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e93a8c5-35f7-459b-a8f1-f282130116cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:04.318017 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.317926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84khn\" (UniqueName: \"kubernetes.io/projected/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-kube-api-access-84khn\") pod \"dns-default-2xmlx\" (UID: \"b2f64d21-ae78-4fdd-b2e5-83c7629e563a\") " pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:04.318017 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.317943 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-tmp-dir\") pod \"dns-default-2xmlx\" (UID: \"b2f64d21-ae78-4fdd-b2e5-83c7629e563a\") " pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:04.318274 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.318254 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-config-volume\") pod \"dns-default-2xmlx\" (UID: \"b2f64d21-ae78-4fdd-b2e5-83c7629e563a\") " pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:04.321411 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.321386 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-metrics-tls\") pod \"dns-default-2xmlx\" (UID: \"b2f64d21-ae78-4fdd-b2e5-83c7629e563a\") " pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:04.321663 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.321645 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be9a9474-03c1-489e-ab01-0e8e83d79daf-cert\") pod \"ingress-canary-q9hsv\" (UID: \"be9a9474-03c1-489e-ab01-0e8e83d79daf\") " pod="openshift-ingress-canary/ingress-canary-q9hsv" Oct 30 02:41:04.339191 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.339165 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84khn\" (UniqueName: \"kubernetes.io/projected/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-kube-api-access-84khn\") pod \"dns-default-2xmlx\" (UID: \"b2f64d21-ae78-4fdd-b2e5-83c7629e563a\") " pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:04.339905 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.339882 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77cp8\" (UniqueName: \"kubernetes.io/projected/be9a9474-03c1-489e-ab01-0e8e83d79daf-kube-api-access-77cp8\") pod \"ingress-canary-q9hsv\" (UID: \"be9a9474-03c1-489e-ab01-0e8e83d79daf\") " pod="openshift-ingress-canary/ingress-canary-q9hsv" Oct 30 02:41:04.418758 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.418722 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e93a8c5-35f7-459b-a8f1-f282130116cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:04.418758 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.418759 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e93a8c5-35f7-459b-a8f1-f282130116cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:04.418928 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.418895 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e93a8c5-35f7-459b-a8f1-f282130116cb-crio-socket\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:04.418975 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.418929 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e93a8c5-35f7-459b-a8f1-f282130116cb-data-volume\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:04.418975 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.418960 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tccft\" (UniqueName: \"kubernetes.io/projected/7e93a8c5-35f7-459b-a8f1-f282130116cb-kube-api-access-tccft\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:04.419099 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.419079 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e93a8c5-35f7-459b-a8f1-f282130116cb-crio-socket\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:04.419210 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.419189 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:04.419328 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.419270 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e93a8c5-35f7-459b-a8f1-f282130116cb-data-volume\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:04.419443 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.419425 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e93a8c5-35f7-459b-a8f1-f282130116cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:04.421060 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.421033 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e93a8c5-35f7-459b-a8f1-f282130116cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:04.423456 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.423439 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q9hsv" Oct 30 02:41:04.441313 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.441275 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tccft\" (UniqueName: \"kubernetes.io/projected/7e93a8c5-35f7-459b-a8f1-f282130116cb-kube-api-access-tccft\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:04.501962 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.501729 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:04.719432 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.719398 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:41:04.719936 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.719654 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:41:04.722924 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.722905 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q9hsv"] Oct 30 02:41:04.724982 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:41:04.724964 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9a9474_03c1_489e_ab01_0e8e83d79daf.slice/crio-3821ebc90c34eb201d35c59d28586577e4e8e1b9449e09e43abadb4790b26bfe WatchSource:0}: Error finding container 3821ebc90c34eb201d35c59d28586577e4e8e1b9449e09e43abadb4790b26bfe: Status 404 returned error can't find the container with id 3821ebc90c34eb201d35c59d28586577e4e8e1b9449e09e43abadb4790b26bfe Oct 30 02:41:04.732123 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.732087 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2xmlx"] Oct 30 02:41:04.733331 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.733058 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 02:41:04.733331 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.733124 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 02:41:04.733331 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.733148 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2qbtl\"" Oct 30 02:41:04.733331 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.733205 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7tzfh\"" Oct 30 02:41:04.733331 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.733244 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 02:41:04.734555 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:41:04.734535 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f64d21_ae78_4fdd_b2e5_83c7629e563a.slice/crio-d12148f0921e69bfc84db330ac55a37d131909162ea932bd68035e64b3960f9b WatchSource:0}: Error finding container d12148f0921e69bfc84db330ac55a37d131909162ea932bd68035e64b3960f9b: Status 404 returned error can't find the container with id d12148f0921e69bfc84db330ac55a37d131909162ea932bd68035e64b3960f9b Oct 30 02:41:04.790709 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.790635 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tfqlk"] Oct 30 02:41:04.793730 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:41:04.793707 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e93a8c5_35f7_459b_a8f1_f282130116cb.slice/crio-fbe884666880840186124d47864fe6002ab614ab816e77422b0c73277819e344 WatchSource:0}: Error finding container fbe884666880840186124d47864fe6002ab614ab816e77422b0c73277819e344: Status 404 returned error can't find the container with id fbe884666880840186124d47864fe6002ab614ab816e77422b0c73277819e344 Oct 30 02:41:04.958856 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.958814 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q9hsv" event={"ID":"be9a9474-03c1-489e-ab01-0e8e83d79daf","Type":"ContainerStarted","Data":"3821ebc90c34eb201d35c59d28586577e4e8e1b9449e09e43abadb4790b26bfe"} Oct 30 02:41:04.959990 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.959960 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2xmlx" event={"ID":"b2f64d21-ae78-4fdd-b2e5-83c7629e563a","Type":"ContainerStarted","Data":"d12148f0921e69bfc84db330ac55a37d131909162ea932bd68035e64b3960f9b"} Oct 30 02:41:04.961276 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.961255 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tfqlk" event={"ID":"7e93a8c5-35f7-459b-a8f1-f282130116cb","Type":"ContainerStarted","Data":"bbd8a59a2a2fa6994f97d4e60f838066f3db10423353a6198be9e418e71bd5a8"} Oct 30 02:41:04.961374 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:04.961296 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tfqlk" event={"ID":"7e93a8c5-35f7-459b-a8f1-f282130116cb","Type":"ContainerStarted","Data":"fbe884666880840186124d47864fe6002ab614ab816e77422b0c73277819e344"} Oct 30 02:41:05.720270 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:05.720236 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:41:05.734047 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:05.733989 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lz8p4\"" Oct 30 02:41:05.734047 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:05.734010 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 02:41:05.966956 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:05.966919 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tfqlk" event={"ID":"7e93a8c5-35f7-459b-a8f1-f282130116cb","Type":"ContainerStarted","Data":"ff5ef78be73afde935cceee6696376948b116d91644dd28bb6f4a2d53527e605"} Oct 30 02:41:06.971756 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:06.971719 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2xmlx" event={"ID":"b2f64d21-ae78-4fdd-b2e5-83c7629e563a","Type":"ContainerStarted","Data":"5cd73201297771768678e6ece73919d5bc58b1ada132fa7867bfcdc54d02f133"} Oct 30 02:41:06.972156 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:06.971782 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2xmlx" event={"ID":"b2f64d21-ae78-4fdd-b2e5-83c7629e563a","Type":"ContainerStarted","Data":"9b75e19c91f28216a83f21133ecca1f57c582c2065ad4ef188b6480f02ed8e18"} Oct 30 02:41:06.972156 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:06.971871 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:07.014516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.014464 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-2xmlx" podStartSLOduration=1.408583039 podStartE2EDuration="3.014445937s" totalImagesPullingTime="1.605862898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:07.013653633 +0000 UTC m=+50.876692773" watchObservedRunningTime="2026-10-30 02:41:07.014445937 +0000 UTC m=+50.877485077" Oct 30 02:41:07.088969 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.088937 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl"] Oct 30 02:41:07.091019 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.091000 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:07.117433 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.117408 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 02:41:07.117433 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.117411 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 02:41:07.118577 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.118556 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8cdxd\"" Oct 30 02:41:07.119283 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.119249 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 02:41:07.119283 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.119278 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 02:41:07.119425 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.119278 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 02:41:07.129402 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.128730 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl"] Oct 30 02:41:07.132167 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.132142 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck"] Oct 30 02:41:07.135907 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.135885 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c5f6v"] Oct 30 02:41:07.136061 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.136043 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:07.138830 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.138802 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.139190 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.139159 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-z5vhl\" (UID: \"0c71f4fe-cc89-4856-accb-5e6c7c9452e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:07.139278 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.139230 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-z5vhl\" (UID: \"0c71f4fe-cc89-4856-accb-5e6c7c9452e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:07.139278 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.139262 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-z5vhl\" (UID: \"0c71f4fe-cc89-4856-accb-5e6c7c9452e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:07.139392 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.139304 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vwtj\" (UniqueName: \"kubernetes.io/projected/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-kube-api-access-4vwtj\") pod \"openshift-state-metrics-7bd7489957-z5vhl\" (UID: \"0c71f4fe-cc89-4856-accb-5e6c7c9452e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:07.162832 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.162801 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck"] Oct 30 02:41:07.213329 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.213294 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 02:41:07.213512 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.213385 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 02:41:07.213800 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.213762 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 02:41:07.213888 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.213828 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mmm8g\"" Oct 30 02:41:07.213888 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.213829 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 02:41:07.213977 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.213898 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rw9z8\"" Oct 30 02:41:07.218437 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.218413 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 02:41:07.218544 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.218498 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 02:41:07.240156 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.240083 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-z5vhl\" (UID: \"0c71f4fe-cc89-4856-accb-5e6c7c9452e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:07.240156 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.240123 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-php67\" (UniqueName: \"kubernetes.io/projected/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-api-access-php67\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:07.240156 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.240143 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-tls\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.240360 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.240196 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:07.240360 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.240332 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7cecaabc-af83-4bf9-b3e5-d446b1302f12-root\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.240360 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.240357 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-accelerators-collector-config\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.240457 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.240376 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:07.240457 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.240394 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:07.240535 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.240469 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cecaabc-af83-4bf9-b3e5-d446b1302f12-sys\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.240535 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.240502 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-textfile\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.240651 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.240584 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:07.240703 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.240647 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tvzr\" (UniqueName: \"kubernetes.io/projected/7cecaabc-af83-4bf9-b3e5-d446b1302f12-kube-api-access-9tvzr\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.240703 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.240679 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:07.240811 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.240712 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-z5vhl\" (UID: \"0c71f4fe-cc89-4856-accb-5e6c7c9452e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:07.240811 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.240753 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vwtj\" (UniqueName: \"kubernetes.io/projected/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-kube-api-access-4vwtj\") pod \"openshift-state-metrics-7bd7489957-z5vhl\" (UID: \"0c71f4fe-cc89-4856-accb-5e6c7c9452e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:07.240923 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.240905 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-wtmp\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.241032 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.240947 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.241032 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.240981 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cecaabc-af83-4bf9-b3e5-d446b1302f12-metrics-client-ca\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.241105 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.241066 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-z5vhl\" (UID: \"0c71f4fe-cc89-4856-accb-5e6c7c9452e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:07.241335 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.241319 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-z5vhl\" (UID: \"0c71f4fe-cc89-4856-accb-5e6c7c9452e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:07.242966 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.242942 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-z5vhl\" (UID: \"0c71f4fe-cc89-4856-accb-5e6c7c9452e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:07.243671 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.243650 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-z5vhl\" (UID: \"0c71f4fe-cc89-4856-accb-5e6c7c9452e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:07.265384 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.265357 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vwtj\" (UniqueName: \"kubernetes.io/projected/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-kube-api-access-4vwtj\") pod \"openshift-state-metrics-7bd7489957-z5vhl\" (UID: \"0c71f4fe-cc89-4856-accb-5e6c7c9452e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:07.341405 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.341382 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-tls\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.341476 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.341410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:07.341476 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.341436 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7cecaabc-af83-4bf9-b3e5-d446b1302f12-root\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.341476 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.341458 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-accelerators-collector-config\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.341607 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.341485 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:07.341607 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.341510 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:07.341607 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.341531 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7cecaabc-af83-4bf9-b3e5-d446b1302f12-root\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.341607 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.341549 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cecaabc-af83-4bf9-b3e5-d446b1302f12-sys\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.341607 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.341595 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cecaabc-af83-4bf9-b3e5-d446b1302f12-sys\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.341861 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.341664 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-textfile\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.341861 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.341702 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:07.341861 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.341746 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tvzr\" (UniqueName: \"kubernetes.io/projected/7cecaabc-af83-4bf9-b3e5-d446b1302f12-kube-api-access-9tvzr\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.341861 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.341774 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:07.341861 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.341804 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:07.341861 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.341826 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-wtmp\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.342117 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.341879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.342117 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.341911 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cecaabc-af83-4bf9-b3e5-d446b1302f12-metrics-client-ca\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.342117 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.341942 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-wtmp\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.342117 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.341962 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-php67\" (UniqueName: \"kubernetes.io/projected/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-api-access-php67\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:07.342317 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.342170 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:07.342938 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.342877 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cecaabc-af83-4bf9-b3e5-d446b1302f12-metrics-client-ca\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.342938 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.342873 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-accelerators-collector-config\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.343266 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.343227 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-textfile\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.343401 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.343382 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:07.343895 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.343868 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-tls\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.344237 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.344219 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:07.344568 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.344546 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.344639 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.344604 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:07.367327 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.367300 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-php67\" (UniqueName: \"kubernetes.io/projected/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-api-access-php67\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:07.369190 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.369161 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tvzr\" (UniqueName: \"kubernetes.io/projected/7cecaabc-af83-4bf9-b3e5-d446b1302f12-kube-api-access-9tvzr\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.405364 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.405328 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:07.455069 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.454377 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:07.466874 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.465743 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:07.487498 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:41:07.487469 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cecaabc_af83_4bf9_b3e5_d446b1302f12.slice/crio-06f1fc5f300539b25f978d613245bd45f3002512ea9895cbbc01030dc677a6fb WatchSource:0}: Error finding container 06f1fc5f300539b25f978d613245bd45f3002512ea9895cbbc01030dc677a6fb: Status 404 returned error can't find the container with id 06f1fc5f300539b25f978d613245bd45f3002512ea9895cbbc01030dc677a6fb Oct 30 02:41:07.716157 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.716125 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl"] Oct 30 02:41:07.718309 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:41:07.718281 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c71f4fe_cc89_4856_accb_5e6c7c9452e2.slice/crio-73dcebfdcf6a6cd70831cf2f336d84c18d7ee6b04438d24fc3b27bb82e6950b4 WatchSource:0}: Error finding container 73dcebfdcf6a6cd70831cf2f336d84c18d7ee6b04438d24fc3b27bb82e6950b4: Status 404 returned error can't find the container with id 73dcebfdcf6a6cd70831cf2f336d84c18d7ee6b04438d24fc3b27bb82e6950b4 Oct 30 02:41:07.752517 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.752487 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck"] Oct 30 02:41:07.754410 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:41:07.754386 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode06d9d5b_be5f_46aa_a633_8ce9c9c37589.slice/crio-d43deb655200ec4bd4fab804d41f694a5038e3837f234a33f49e5a9130df168e WatchSource:0}: Error finding container d43deb655200ec4bd4fab804d41f694a5038e3837f234a33f49e5a9130df168e: Status 404 returned error can't find the container with id d43deb655200ec4bd4fab804d41f694a5038e3837f234a33f49e5a9130df168e Oct 30 02:41:07.977705 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.977273 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q9hsv" event={"ID":"be9a9474-03c1-489e-ab01-0e8e83d79daf","Type":"ContainerStarted","Data":"a8aac288073438857368de93254e7036a128ae11cc503b7d107b453bed8179c7"} Oct 30 02:41:07.979362 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.979330 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" event={"ID":"0c71f4fe-cc89-4856-accb-5e6c7c9452e2","Type":"ContainerStarted","Data":"8922a63a6f140e7ad75441710253764301c3dd58c515954ba493d13f1d271eb9"} Oct 30 02:41:07.979481 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.979391 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" event={"ID":"0c71f4fe-cc89-4856-accb-5e6c7c9452e2","Type":"ContainerStarted","Data":"9b0546aa80668042fca747c2bacafdc61c829e1c01c8da0ad8aad5de973ec5e9"} Oct 30 02:41:07.979481 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.979417 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" event={"ID":"0c71f4fe-cc89-4856-accb-5e6c7c9452e2","Type":"ContainerStarted","Data":"73dcebfdcf6a6cd70831cf2f336d84c18d7ee6b04438d24fc3b27bb82e6950b4"} Oct 30 02:41:07.981532 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.981503 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tfqlk" event={"ID":"7e93a8c5-35f7-459b-a8f1-f282130116cb","Type":"ContainerStarted","Data":"fa0840426c49d42a4b583976ea8946daef1051a039c7a5d8a98d646a360b6518"} Oct 30 02:41:07.982817 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.982791 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c5f6v" event={"ID":"7cecaabc-af83-4bf9-b3e5-d446b1302f12","Type":"ContainerStarted","Data":"06f1fc5f300539b25f978d613245bd45f3002512ea9895cbbc01030dc677a6fb"} Oct 30 02:41:07.984046 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:07.984022 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" event={"ID":"e06d9d5b-be5f-46aa-a633-8ce9c9c37589","Type":"ContainerStarted","Data":"d43deb655200ec4bd4fab804d41f694a5038e3837f234a33f49e5a9130df168e"} Oct 30 02:41:08.014073 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.014026 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-q9hsv" podStartSLOduration=1.441345968 podStartE2EDuration="4.014012474s" totalImagesPullingTime="2.572666506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:08.01319708 +0000 UTC m=+51.876236220" watchObservedRunningTime="2026-10-30 02:41:08.014012474 +0000 UTC m=+51.877051603" Oct 30 02:41:08.572993 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.572894 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tfqlk" podStartSLOduration=2.484576787 podStartE2EDuration="4.572875883s" totalImagesPullingTime="2.088299096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:08.05404142 +0000 UTC m=+51.917080559" watchObservedRunningTime="2026-10-30 02:41:08.572875883 +0000 UTC m=+52.435915025" Oct 30 02:41:08.573424 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.573400 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 02:41:08.576394 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.576369 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.591569 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.591543 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 02:41:08.591805 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.591786 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 02:41:08.591901 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.591840 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 02:41:08.606278 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.606252 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 02:41:08.606392 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.606354 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 02:41:08.607815 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.607729 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 02:41:08.653888 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.653855 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.654066 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.653923 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.654066 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.653949 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/992d8669-c914-486f-83ea-688dc69ca5a0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.654066 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.653971 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kndrd\" (UniqueName: \"kubernetes.io/projected/992d8669-c914-486f-83ea-688dc69ca5a0-kube-api-access-kndrd\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.654066 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.653994 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.654066 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.654021 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-web-config\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.654313 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.654066 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/992d8669-c914-486f-83ea-688dc69ca5a0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.654313 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.654095 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.654313 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.654124 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/992d8669-c914-486f-83ea-688dc69ca5a0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.654313 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.654156 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/992d8669-c914-486f-83ea-688dc69ca5a0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.654313 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.654205 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-config-volume\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.654313 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.654233 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.654313 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.654266 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/992d8669-c914-486f-83ea-688dc69ca5a0-config-out\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.692652 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.692607 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 02:41:08.692814 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.692698 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9rd8b\"" Oct 30 02:41:08.692814 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.692729 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 02:41:08.692814 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.692784 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 02:41:08.692921 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.692846 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 02:41:08.755506 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.755473 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/992d8669-c914-486f-83ea-688dc69ca5a0-config-out\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.755676 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.755536 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.755676 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.755580 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.755676 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.755606 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/992d8669-c914-486f-83ea-688dc69ca5a0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.755676 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.755650 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kndrd\" (UniqueName: \"kubernetes.io/projected/992d8669-c914-486f-83ea-688dc69ca5a0-kube-api-access-kndrd\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.755676 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.755674 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.755918 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.755701 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-web-config\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.755918 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.755742 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/992d8669-c914-486f-83ea-688dc69ca5a0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.755918 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.755768 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.755918 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.755793 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/992d8669-c914-486f-83ea-688dc69ca5a0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.755918 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.755807 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/992d8669-c914-486f-83ea-688dc69ca5a0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.755918 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.755846 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-config-volume\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.755918 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.755870 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.756820 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.756792 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/992d8669-c914-486f-83ea-688dc69ca5a0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.757331 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.757034 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/992d8669-c914-486f-83ea-688dc69ca5a0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.757697 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.757486 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/992d8669-c914-486f-83ea-688dc69ca5a0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.758353 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.758326 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/992d8669-c914-486f-83ea-688dc69ca5a0-config-out\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.759051 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.758981 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.759507 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.759479 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-web-config\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.760514 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.760490 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.760602 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.760581 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.760839 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.760815 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.760947 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.760911 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-config-volume\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.762138 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.762113 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/992d8669-c914-486f-83ea-688dc69ca5a0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.762248 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.762226 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.778360 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.778331 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kndrd\" (UniqueName: \"kubernetes.io/projected/992d8669-c914-486f-83ea-688dc69ca5a0-kube-api-access-kndrd\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.898089 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.897911 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:08.989040 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.989005 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" event={"ID":"e06d9d5b-be5f-46aa-a633-8ce9c9c37589","Type":"ContainerStarted","Data":"e98b4821fe29c7f96077c62c565455a0310d793eee19df6b6418ae1c56317369"} Oct 30 02:41:08.989384 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.989065 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" event={"ID":"e06d9d5b-be5f-46aa-a633-8ce9c9c37589","Type":"ContainerStarted","Data":"6d2c5a35920ee6b10cd5c083c617878b141c885f367255312c34872747c2df35"} Oct 30 02:41:08.991106 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.991078 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" event={"ID":"0c71f4fe-cc89-4856-accb-5e6c7c9452e2","Type":"ContainerStarted","Data":"2da138ff4e658f4423aef72c9c503f103985fc5f0c3ad3446f22817e4817d77d"} Oct 30 02:41:08.993591 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.993470 2573 generic.go:356] "Generic (PLEG): container finished" podID="7cecaabc-af83-4bf9-b3e5-d446b1302f12" containerID="03dba5fd4f38b874d3c102b40a79af1a1c01bfb20d57608b4a1e64a4ce225891" exitCode=0 Oct 30 02:41:08.993591 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:08.993560 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c5f6v" event={"ID":"7cecaabc-af83-4bf9-b3e5-d446b1302f12","Type":"ContainerDied","Data":"03dba5fd4f38b874d3c102b40a79af1a1c01bfb20d57608b4a1e64a4ce225891"} Oct 30 02:41:09.031118 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.031070 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" podStartSLOduration=1.079983842 podStartE2EDuration="2.031056776s" totalImagesPullingTime="951.072934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:09.029752152 +0000 UTC m=+52.892791290" watchObservedRunningTime="2026-10-30 02:41:09.031056776 +0000 UTC m=+52.894095915" Oct 30 02:41:09.216741 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.216711 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 02:41:09.218472 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:41:09.218444 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992d8669_c914_486f_83ea_688dc69ca5a0.slice/crio-1fac83ee3517ccc5d3939b215e63adc0f51a43f4ef74247f00322b61fd72cf77 WatchSource:0}: Error finding container 1fac83ee3517ccc5d3939b215e63adc0f51a43f4ef74247f00322b61fd72cf77: Status 404 returned error can't find the container with id 1fac83ee3517ccc5d3939b215e63adc0f51a43f4ef74247f00322b61fd72cf77 Oct 30 02:41:09.350401 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.350323 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-67fdc77d96-6slgz"] Oct 30 02:41:09.353716 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.353695 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.367351 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.367329 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 02:41:09.367574 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.367533 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 02:41:09.367859 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.367834 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 02:41:09.367859 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.367859 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 02:41:09.368033 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.367902 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-19g27kfjqkpbt\"" Oct 30 02:41:09.368033 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.367935 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 02:41:09.368033 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.367992 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pjcnc\"" Oct 30 02:41:09.376520 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.376500 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67fdc77d96-6slgz"] Oct 30 02:41:09.468576 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.468544 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.468576 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.468577 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-grpc-tls\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.468824 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.468596 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-tls\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.468824 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.468763 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6vgd\" (UniqueName: \"kubernetes.io/projected/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-kube-api-access-p6vgd\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.468824 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.468794 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-metrics-client-ca\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.468921 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.468821 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.468965 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.468945 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-tmp\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.469016 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.469001 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.469120 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.469098 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.570392 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.570358 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.570392 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.570392 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-grpc-tls\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.570576 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.570410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-tls\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.570576 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.570494 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6vgd\" (UniqueName: \"kubernetes.io/projected/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-kube-api-access-p6vgd\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.570576 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.570515 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-metrics-client-ca\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.570576 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.570547 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.570806 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.570634 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-tmp\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.570806 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.570709 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.570806 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.570773 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.571281 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.571249 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-tmp\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.571934 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.571863 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-metrics-client-ca\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.573685 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.573607 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.573766 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.573651 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-tls\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.573766 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.573702 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.573991 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.573971 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.574099 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.574079 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.574135 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.574097 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-grpc-tls\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.592301 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.592281 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6vgd\" (UniqueName: \"kubernetes.io/projected/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-kube-api-access-p6vgd\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.679376 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.679300 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:09.993599 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:09.993572 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67fdc77d96-6slgz"] Oct 30 02:41:09.996824 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:41:09.996791 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec808c2_5a3a_46f1_b8ea_95d24ee6cdb0.slice/crio-0f063e2868918936ecd1e77ae7ede121a90f16a34ba53793387629cc2b64a9dc WatchSource:0}: Error finding container 0f063e2868918936ecd1e77ae7ede121a90f16a34ba53793387629cc2b64a9dc: Status 404 returned error can't find the container with id 0f063e2868918936ecd1e77ae7ede121a90f16a34ba53793387629cc2b64a9dc Oct 30 02:41:10.004886 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:10.004855 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c5f6v" event={"ID":"7cecaabc-af83-4bf9-b3e5-d446b1302f12","Type":"ContainerStarted","Data":"5d53ae4d7dbc4321f09f01b83dfab42244cc8371d0ddcdf1db35f81851ded917"} Oct 30 02:41:10.004992 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:10.004957 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c5f6v" event={"ID":"7cecaabc-af83-4bf9-b3e5-d446b1302f12","Type":"ContainerStarted","Data":"064f2e1b4bf1ffb360be0af8b4e9422faa546952791498e7b6b53006a2c0e3da"} Oct 30 02:41:10.007835 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:10.007809 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" event={"ID":"e06d9d5b-be5f-46aa-a633-8ce9c9c37589","Type":"ContainerStarted","Data":"bee12f687f3e50a634afc3ccab688e256c232e28ed4a9719540fcbe53d8c775d"} Oct 30 02:41:10.009127 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:10.009102 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"992d8669-c914-486f-83ea-688dc69ca5a0","Type":"ContainerStarted","Data":"1fac83ee3517ccc5d3939b215e63adc0f51a43f4ef74247f00322b61fd72cf77"} Oct 30 02:41:10.048287 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:10.048239 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c5f6v" podStartSLOduration=2.136189933 podStartE2EDuration="3.048227542s" totalImagesPullingTime="833.371518ms" totalInitContainerRuntime="78.666091ms" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:10.046998864 +0000 UTC m=+53.910038004" watchObservedRunningTime="2026-10-30 02:41:10.048227542 +0000 UTC m=+53.911266681" Oct 30 02:41:10.094410 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:10.094362 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" podStartSLOduration=2.032411197 podStartE2EDuration="3.094349624s" totalImagesPullingTime="1.061938427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:10.088985406 +0000 UTC m=+53.952024546" watchObservedRunningTime="2026-10-30 02:41:10.094349624 +0000 UTC m=+53.957388763" Oct 30 02:41:10.918443 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:10.918417 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:11.014864 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.014819 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" event={"ID":"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0","Type":"ContainerStarted","Data":"0f063e2868918936ecd1e77ae7ede121a90f16a34ba53793387629cc2b64a9dc"} Oct 30 02:41:11.016632 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.016594 2573 generic.go:356] "Generic (PLEG): container finished" podID="992d8669-c914-486f-83ea-688dc69ca5a0" containerID="8171d08cc27a8278b5dfe6c057eb59c5a9992947a7555c85c48f4780cf193488" exitCode=0 Oct 30 02:41:11.016757 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.016649 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"992d8669-c914-486f-83ea-688dc69ca5a0","Type":"ContainerDied","Data":"8171d08cc27a8278b5dfe6c057eb59c5a9992947a7555c85c48f4780cf193488"} Oct 30 02:41:11.632075 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.632038 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-dc789c958-vff7h"] Oct 30 02:41:11.634701 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.634674 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.652739 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.652711 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 02:41:11.658163 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.658135 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-dc789c958-vff7h"] Oct 30 02:41:11.659828 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.659794 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 02:41:11.666880 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.666857 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 02:41:11.667000 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.666923 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bdpn101jv6nnb\"" Oct 30 02:41:11.667000 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.666988 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 02:41:11.667110 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.667031 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-227fv\"" Oct 30 02:41:11.688444 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.688410 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fd320963-1dae-4be2-9352-15a810104d69-audit-log\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.688444 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.688447 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fd320963-1dae-4be2-9352-15a810104d69-client-ca-bundle\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.688697 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.688515 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fd320963-1dae-4be2-9352-15a810104d69-secret-metrics-server-tls\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.688697 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.688539 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fd320963-1dae-4be2-9352-15a810104d69-metrics-server-audit-profiles\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.688697 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.688564 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fd320963-1dae-4be2-9352-15a810104d69-secret-metrics-server-client-certs\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.688697 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.688604 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd320963-1dae-4be2-9352-15a810104d69-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.688697 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.688657 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqcx7\" (UniqueName: \"kubernetes.io/projected/fd320963-1dae-4be2-9352-15a810104d69-kube-api-access-fqcx7\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.789176 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.789141 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fd320963-1dae-4be2-9352-15a810104d69-secret-metrics-server-tls\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.789334 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.789188 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fd320963-1dae-4be2-9352-15a810104d69-metrics-server-audit-profiles\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.789398 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.789332 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fd320963-1dae-4be2-9352-15a810104d69-secret-metrics-server-client-certs\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.789455 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.789412 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd320963-1dae-4be2-9352-15a810104d69-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.789455 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.789443 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqcx7\" (UniqueName: \"kubernetes.io/projected/fd320963-1dae-4be2-9352-15a810104d69-kube-api-access-fqcx7\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.789895 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.789559 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fd320963-1dae-4be2-9352-15a810104d69-audit-log\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.789895 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.789588 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fd320963-1dae-4be2-9352-15a810104d69-client-ca-bundle\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.790067 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.790045 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fd320963-1dae-4be2-9352-15a810104d69-audit-log\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.790131 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.790109 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd320963-1dae-4be2-9352-15a810104d69-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.790269 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.790243 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fd320963-1dae-4be2-9352-15a810104d69-metrics-server-audit-profiles\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.793198 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.792387 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fd320963-1dae-4be2-9352-15a810104d69-secret-metrics-server-tls\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.793198 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.792802 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fd320963-1dae-4be2-9352-15a810104d69-client-ca-bundle\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.793198 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.793171 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fd320963-1dae-4be2-9352-15a810104d69-secret-metrics-server-client-certs\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.813592 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.813563 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqcx7\" (UniqueName: \"kubernetes.io/projected/fd320963-1dae-4be2-9352-15a810104d69-kube-api-access-fqcx7\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.842143 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.842112 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-647f7b9c85-ldfjx"] Oct 30 02:41:11.845361 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.845337 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.859847 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.859821 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 02:41:11.860285 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.860250 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 02:41:11.860389 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.860378 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kclfx\"" Oct 30 02:41:11.860447 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.860420 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 02:41:11.860447 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.860424 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 02:41:11.860548 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.860460 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 02:41:11.871140 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.871108 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 02:41:11.871607 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.871585 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 02:41:11.871724 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.871714 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 02:41:11.872802 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.872778 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-647f7b9c85-ldfjx"] Oct 30 02:41:11.893851 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.890652 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.893851 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.890710 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.893851 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.890786 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.893851 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.890815 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl2vg\" (UniqueName: \"kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.893851 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.890843 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49f725c1-f41a-403d-b729-a170fc1fe687-tmp\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.893851 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.890866 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.893851 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.890922 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.893851 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.890952 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.950082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.949849 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:11.951896 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.951771 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-shbmn"] Oct 30 02:41:11.953876 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.953859 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-shbmn" Oct 30 02:41:11.975315 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.975288 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cjkxs\"" Oct 30 02:41:11.975447 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.975363 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 02:41:11.980632 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.978232 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-shbmn"] Oct 30 02:41:11.996653 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.991378 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwcpz\" (UniqueName: \"kubernetes.io/projected/c42eb03a-a0d4-4509-8d16-ae82e1ac3c26-kube-api-access-mwcpz\") pod \"monitoring-plugin-77f4f76594-shbmn\" (UID: \"c42eb03a-a0d4-4509-8d16-ae82e1ac3c26\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-shbmn" Oct 30 02:41:11.996653 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.991423 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.996653 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.991522 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.996653 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.991591 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2vg\" (UniqueName: \"kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.996653 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.991657 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49f725c1-f41a-403d-b729-a170fc1fe687-tmp\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.996653 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.991691 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c42eb03a-a0d4-4509-8d16-ae82e1ac3c26-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-shbmn\" (UID: \"c42eb03a-a0d4-4509-8d16-ae82e1ac3c26\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-shbmn" Oct 30 02:41:11.996653 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.991743 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.996653 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.991904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.996653 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.991941 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.996653 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.991994 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.996653 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.993431 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.996653 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.994241 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.996653 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.995807 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49f725c1-f41a-403d-b729-a170fc1fe687-tmp\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.996653 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.995877 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.996653 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.996235 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:11.998524 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:11.998501 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:12.000939 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:12.000917 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:12.030411 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:12.025171 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl2vg\" (UniqueName: \"kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:12.030411 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:12.027661 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" event={"ID":"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0","Type":"ContainerStarted","Data":"b3dd993705b835e65023e1f91fb204eea4fabb3186085236473ecfd85a8a6028"} Oct 30 02:41:12.030411 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:12.027741 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" event={"ID":"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0","Type":"ContainerStarted","Data":"9fc80230e7a5527fd1371414c36e539508031ae95572b3433c8959f8d8152e8c"} Oct 30 02:41:12.030411 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:12.027783 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" event={"ID":"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0","Type":"ContainerStarted","Data":"c055b2b39362f129f235679161c16a61561a2e68eec467ce9d5b73b61066eafb"} Oct 30 02:41:12.093451 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:12.093401 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwcpz\" (UniqueName: \"kubernetes.io/projected/c42eb03a-a0d4-4509-8d16-ae82e1ac3c26-kube-api-access-mwcpz\") pod \"monitoring-plugin-77f4f76594-shbmn\" (UID: \"c42eb03a-a0d4-4509-8d16-ae82e1ac3c26\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-shbmn" Oct 30 02:41:12.093634 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:12.093557 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c42eb03a-a0d4-4509-8d16-ae82e1ac3c26-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-shbmn\" (UID: \"c42eb03a-a0d4-4509-8d16-ae82e1ac3c26\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-shbmn" Oct 30 02:41:12.100652 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:12.096631 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c42eb03a-a0d4-4509-8d16-ae82e1ac3c26-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-shbmn\" (UID: \"c42eb03a-a0d4-4509-8d16-ae82e1ac3c26\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-shbmn" Oct 30 02:41:12.115961 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:12.115934 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwcpz\" (UniqueName: \"kubernetes.io/projected/c42eb03a-a0d4-4509-8d16-ae82e1ac3c26-kube-api-access-mwcpz\") pod \"monitoring-plugin-77f4f76594-shbmn\" (UID: \"c42eb03a-a0d4-4509-8d16-ae82e1ac3c26\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-shbmn" Oct 30 02:41:12.169865 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:12.169776 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:12.267253 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:12.267217 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-shbmn" Oct 30 02:41:12.289382 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:12.289352 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-dc789c958-vff7h"] Oct 30 02:41:12.291797 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:41:12.291770 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd320963_1dae_4be2_9352_15a810104d69.slice/crio-51dca0b80913a1012976309f7599242c791c00fc8be56b17497b1c2b320bfcfa WatchSource:0}: Error finding container 51dca0b80913a1012976309f7599242c791c00fc8be56b17497b1c2b320bfcfa: Status 404 returned error can't find the container with id 51dca0b80913a1012976309f7599242c791c00fc8be56b17497b1c2b320bfcfa Oct 30 02:41:12.465638 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:12.465594 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-647f7b9c85-ldfjx"] Oct 30 02:41:12.470195 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:41:12.470153 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f725c1_f41a_403d_b729_a170fc1fe687.slice/crio-719b462c9bbf11e8fa2bba324a94d6311f2196d28e35106f50f564ce80743ec1 WatchSource:0}: Error finding container 719b462c9bbf11e8fa2bba324a94d6311f2196d28e35106f50f564ce80743ec1: Status 404 returned error can't find the container with id 719b462c9bbf11e8fa2bba324a94d6311f2196d28e35106f50f564ce80743ec1 Oct 30 02:41:12.567506 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:12.567482 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-shbmn"] Oct 30 02:41:13.036826 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:13.036785 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" event={"ID":"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0","Type":"ContainerStarted","Data":"6bd85430fed20dab1a2c33d4afef12911add4aa064dae3ea8fb2f3d61155c395"} Oct 30 02:41:13.037290 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:13.036872 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" event={"ID":"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0","Type":"ContainerStarted","Data":"837e2e907754533488c7ac6981f69ab0422532182486823dce57fac7185e8d50"} Oct 30 02:41:13.037290 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:13.036912 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" event={"ID":"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0","Type":"ContainerStarted","Data":"a30f254178e08c3b4500c0232300912bef7fb29612eef753cf199bb1f94567d8"} Oct 30 02:41:13.038026 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:13.037920 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:13.039933 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:13.039907 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-shbmn" event={"ID":"c42eb03a-a0d4-4509-8d16-ae82e1ac3c26","Type":"ContainerStarted","Data":"3f2def1ae94b30dfa54bff0a45d786d2122a87be552737268dabff27b5074954"} Oct 30 02:41:13.041331 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:13.041274 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-647f7b9c85-ldfjx" event={"ID":"49f725c1-f41a-403d-b729-a170fc1fe687","Type":"ContainerStarted","Data":"719b462c9bbf11e8fa2bba324a94d6311f2196d28e35106f50f564ce80743ec1"} Oct 30 02:41:13.043982 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:13.043919 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-dc789c958-vff7h" event={"ID":"fd320963-1dae-4be2-9352-15a810104d69","Type":"ContainerStarted","Data":"51dca0b80913a1012976309f7599242c791c00fc8be56b17497b1c2b320bfcfa"} Oct 30 02:41:13.050638 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:13.050129 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"992d8669-c914-486f-83ea-688dc69ca5a0","Type":"ContainerStarted","Data":"cfdba29c45f9d63d76dc9654306b899c4d640075a5dc791c02b01392b7d16914"} Oct 30 02:41:13.050638 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:13.050190 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"992d8669-c914-486f-83ea-688dc69ca5a0","Type":"ContainerStarted","Data":"d8544c73aa52ed57a312182780a274c1313b7d1433b37282644dc43e40c38000"} Oct 30 02:41:13.050638 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:13.050231 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"992d8669-c914-486f-83ea-688dc69ca5a0","Type":"ContainerStarted","Data":"7277f4a31564fee3139b0db16e3efe98870b42fff7421f6d1f7db3046282f39c"} Oct 30 02:41:13.050638 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:13.050269 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"992d8669-c914-486f-83ea-688dc69ca5a0","Type":"ContainerStarted","Data":"dc5b32e7376c2369d0e322c9d544b1a3d954a5ff21be5b69c11ed00dc3ea1ddd"} Oct 30 02:41:13.050638 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:13.050305 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"992d8669-c914-486f-83ea-688dc69ca5a0","Type":"ContainerStarted","Data":"9da09420916edc7f979b2d38c165556fbff94ac53a75c645c315df70ab9edc2d"} Oct 30 02:41:13.050638 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:13.050343 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"992d8669-c914-486f-83ea-688dc69ca5a0","Type":"ContainerStarted","Data":"b275acc06ff965bb21d8a827ff90709765488dc1693af7ff8796e7f71839284a"} Oct 30 02:41:13.083882 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:13.083827 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" podStartSLOduration=1.663583199 podStartE2EDuration="4.083812534s" totalImagesPullingTime="2.420229335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:13.083672073 +0000 UTC m=+56.946711213" watchObservedRunningTime="2026-10-30 02:41:13.083812534 +0000 UTC m=+56.946851674" Oct 30 02:41:13.138015 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:13.137958 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.827656666 podStartE2EDuration="5.137942162s" totalImagesPullingTime="2.296547139s" totalInitContainerRuntime="13.738357ms" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:13.137038865 +0000 UTC m=+57.000078005" watchObservedRunningTime="2026-10-30 02:41:13.137942162 +0000 UTC m=+57.000981302" Oct 30 02:41:14.058605 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.058130 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-shbmn" event={"ID":"c42eb03a-a0d4-4509-8d16-ae82e1ac3c26","Type":"ContainerStarted","Data":"82d999c47612270cba02c78bb96872d46931639003ab62e555c5ac34048951da"} Oct 30 02:41:14.058605 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.058307 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-shbmn" Oct 30 02:41:14.060825 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.060484 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-dc789c958-vff7h" event={"ID":"fd320963-1dae-4be2-9352-15a810104d69","Type":"ContainerStarted","Data":"c1f6972eff63ab5bff02028d56973e5dccd03b214871bb2f491f44da5b4d5d2c"} Oct 30 02:41:14.063895 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.063874 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-shbmn" Oct 30 02:41:14.147990 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.146396 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-shbmn" podStartSLOduration=1.747624251 podStartE2EDuration="3.146379964s" totalImagesPullingTime="1.398755713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:14.104572918 +0000 UTC m=+57.967612057" watchObservedRunningTime="2026-10-30 02:41:14.146379964 +0000 UTC m=+58.009419102" Oct 30 02:41:14.157820 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.157788 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 02:41:14.162011 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.161986 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.194891 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.194859 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 02:41:14.197011 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.196959 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-dc789c958-vff7h" podStartSLOduration=1.600369507 podStartE2EDuration="3.196945345s" totalImagesPullingTime="1.596575838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:14.196801349 +0000 UTC m=+58.059840489" watchObservedRunningTime="2026-10-30 02:41:14.196945345 +0000 UTC m=+58.059984490" Oct 30 02:41:14.204297 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.204269 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 02:41:14.204422 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.204362 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 02:41:14.204507 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.204473 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-q4z4m\"" Oct 30 02:41:14.204623 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.204601 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 02:41:14.204864 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.204735 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-96il72fi4oust\"" Oct 30 02:41:14.204864 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.204841 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 02:41:14.204864 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.204846 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 02:41:14.204982 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.204914 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 02:41:14.204982 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.204957 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 02:41:14.205041 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.204989 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 02:41:14.205041 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.205027 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 02:41:14.213315 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.213289 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.213440 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.213341 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.213440 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.213376 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.213541 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.213488 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.213541 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.213524 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-config\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.213631 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.213547 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.213631 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.213591 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43ccba25-b0ea-47c8-b664-105feea298a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.213716 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.213641 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.213716 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.213666 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.213716 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.213696 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl9n4\" (UniqueName: \"kubernetes.io/projected/43ccba25-b0ea-47c8-b664-105feea298a3-kube-api-access-pl9n4\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.213801 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.213726 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.213801 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.213766 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.213859 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.213807 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.213859 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.213835 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.213917 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.213864 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.213917 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.213887 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.213917 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.213913 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43ccba25-b0ea-47c8-b664-105feea298a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.213998 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.213945 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.213998 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.213969 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.214058 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.213995 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.270240 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.270210 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 02:41:14.270407 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.270321 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 02:41:14.270856 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.270833 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 02:41:14.280461 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.280435 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 02:41:14.281815 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.281789 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 02:41:14.314744 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.314708 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.314908 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.314753 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-config\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.314908 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.314776 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.314908 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.314826 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43ccba25-b0ea-47c8-b664-105feea298a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.314908 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.314863 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.315115 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.315038 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.315115 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.315071 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl9n4\" (UniqueName: \"kubernetes.io/projected/43ccba25-b0ea-47c8-b664-105feea298a3-kube-api-access-pl9n4\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.315115 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.315103 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.315248 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.315145 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.315248 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.315187 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.315248 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.315187 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.315248 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.315216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.315428 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.315246 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.315428 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.315270 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.315428 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.315300 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43ccba25-b0ea-47c8-b664-105feea298a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.315428 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.315335 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.315428 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.315359 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.315428 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.315382 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.315428 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.315422 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.315798 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.315466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.315798 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.315502 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.315798 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.315778 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.316379 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.316348 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.317149 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.317116 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.318321 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.318299 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.319191 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.319158 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.319307 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.319095 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-config\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.319441 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.319421 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.320025 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.320003 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.322180 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.322094 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.328901 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.328876 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43ccba25-b0ea-47c8-b664-105feea298a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.329065 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.329044 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43ccba25-b0ea-47c8-b664-105feea298a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.329345 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.329326 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.329474 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.329452 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.329775 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.329728 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.329876 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.329813 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.329943 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.329889 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.329943 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.329900 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.330116 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.330096 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.339082 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.339063 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl9n4\" (UniqueName: \"kubernetes.io/projected/43ccba25-b0ea-47c8-b664-105feea298a3-kube-api-access-pl9n4\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.505233 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.505197 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:14.845295 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:14.845070 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 02:41:14.848396 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:41:14.848365 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ccba25_b0ea_47c8_b664_105feea298a3.slice/crio-9372225da62eb113e6c7f903c1f76b737db097a185b64543f4398c2771a34eb3 WatchSource:0}: Error finding container 9372225da62eb113e6c7f903c1f76b737db097a185b64543f4398c2771a34eb3: Status 404 returned error can't find the container with id 9372225da62eb113e6c7f903c1f76b737db097a185b64543f4398c2771a34eb3 Oct 30 02:41:15.066404 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:15.066362 2573 generic.go:356] "Generic (PLEG): container finished" podID="43ccba25-b0ea-47c8-b664-105feea298a3" containerID="53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb" exitCode=0 Oct 30 02:41:15.067492 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:15.067466 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43ccba25-b0ea-47c8-b664-105feea298a3","Type":"ContainerDied","Data":"53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb"} Oct 30 02:41:15.067637 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:15.067536 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43ccba25-b0ea-47c8-b664-105feea298a3","Type":"ContainerStarted","Data":"9372225da62eb113e6c7f903c1f76b737db097a185b64543f4398c2771a34eb3"} Oct 30 02:41:16.072451 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:16.072417 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-647f7b9c85-ldfjx" event={"ID":"49f725c1-f41a-403d-b729-a170fc1fe687","Type":"ContainerStarted","Data":"9f0c8ff7cee660c75133b779bbf875fb9a1b5daf1d5305e937b6d63788e57071"} Oct 30 02:41:16.121156 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:16.121100 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-647f7b9c85-ldfjx" podStartSLOduration=2.221226306 podStartE2EDuration="5.121085225s" totalImagesPullingTime="2.899858919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:16.11957939 +0000 UTC m=+59.982618530" watchObservedRunningTime="2026-10-30 02:41:16.121085225 +0000 UTC m=+59.984124363" Oct 30 02:41:16.986874 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:16.986838 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:18.083869 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:18.083782 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43ccba25-b0ea-47c8-b664-105feea298a3","Type":"ContainerStarted","Data":"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757"} Oct 30 02:41:19.090840 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:19.090806 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43ccba25-b0ea-47c8-b664-105feea298a3","Type":"ContainerStarted","Data":"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd"} Oct 30 02:41:19.091237 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:19.090880 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43ccba25-b0ea-47c8-b664-105feea298a3","Type":"ContainerStarted","Data":"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c"} Oct 30 02:41:19.091237 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:19.090907 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43ccba25-b0ea-47c8-b664-105feea298a3","Type":"ContainerStarted","Data":"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8"} Oct 30 02:41:19.091237 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:19.090936 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43ccba25-b0ea-47c8-b664-105feea298a3","Type":"ContainerStarted","Data":"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae"} Oct 30 02:41:19.091237 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:19.090961 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43ccba25-b0ea-47c8-b664-105feea298a3","Type":"ContainerStarted","Data":"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e"} Oct 30 02:41:19.155683 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:19.155611 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.24179954 podStartE2EDuration="5.155597118s" totalImagesPullingTime="2.894812978s" totalInitContainerRuntime="18.9846ms" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:19.155181385 +0000 UTC m=+63.018220538" watchObservedRunningTime="2026-10-30 02:41:19.155597118 +0000 UTC m=+63.018636257" Oct 30 02:41:19.505520 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:19.505488 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:20.075417 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:20.075389 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:22.170680 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:22.170642 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:22.171183 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:22.170742 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:22.175300 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:22.175278 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:22.502774 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:22.502736 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:41:22.502943 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:22.502793 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs\") pod \"network-metrics-daemon-xg5ng\" (UID: \"5f79e66d-ae26-455c-b111-82f211ffe507\") " pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:41:22.516484 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:22.516457 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 02:41:22.516605 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:22.516519 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 02:41:22.525803 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:22.525781 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:41:22.525901 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:22.525871 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs\") pod \"network-metrics-daemon-xg5ng\" (UID: \"5f79e66d-ae26-455c-b111-82f211ffe507\") " pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:41:22.546000 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:22.545980 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lz8p4\"" Oct 30 02:41:22.554177 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:22.554160 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:41:22.603990 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:22.603960 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d98p\" (UniqueName: \"kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p\") pod \"network-check-target-t765c\" (UID: \"fb1a01bf-dc4f-430b-8b6d-d5569f85c981\") " pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:41:22.616232 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:22.616187 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 02:41:22.636937 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:22.636914 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 02:41:22.647301 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:22.647279 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d98p\" (UniqueName: \"kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p\") pod \"network-check-target-t765c\" (UID: \"fb1a01bf-dc4f-430b-8b6d-d5569f85c981\") " pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:41:22.743974 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:22.743946 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2qbtl\"" Oct 30 02:41:22.749527 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:22.749503 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7tzfh\"" Oct 30 02:41:22.752623 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:22.752599 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:41:22.757991 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:22.757938 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:41:22.847942 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:22.847919 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xg5ng"] Oct 30 02:41:22.855417 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:41:22.855386 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f79e66d_ae26_455c_b111_82f211ffe507.slice/crio-ec4b55722b73847c22ccfa652ecfe716abc269944e17f3e3338b73b01c8a0a2b WatchSource:0}: Error finding container ec4b55722b73847c22ccfa652ecfe716abc269944e17f3e3338b73b01c8a0a2b: Status 404 returned error can't find the container with id ec4b55722b73847c22ccfa652ecfe716abc269944e17f3e3338b73b01c8a0a2b Oct 30 02:41:23.058078 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:23.057969 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t765c"] Oct 30 02:41:23.059516 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:23.059490 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cd6rj"] Oct 30 02:41:23.061026 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:41:23.061004 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1a01bf_dc4f_430b_8b6d_d5569f85c981.slice/crio-f34ccfce0afc3dc0ebf9c79c7878c9f45f2b22f7d6c9a9b7edb556a88938acbf WatchSource:0}: Error finding container f34ccfce0afc3dc0ebf9c79c7878c9f45f2b22f7d6c9a9b7edb556a88938acbf: Status 404 returned error can't find the container with id f34ccfce0afc3dc0ebf9c79c7878c9f45f2b22f7d6c9a9b7edb556a88938acbf Oct 30 02:41:23.061566 ip-10-0-141-200 kubenswrapper[2573]: W1030 02:41:23.061544 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107bc8a5_c01b_4b78_aa29_765aab336604.slice/crio-8ae40967f9880f262525f70dbad8fba75d576cec200aacf35a642824b53ad1e6 WatchSource:0}: Error finding container 8ae40967f9880f262525f70dbad8fba75d576cec200aacf35a642824b53ad1e6: Status 404 returned error can't find the container with id 8ae40967f9880f262525f70dbad8fba75d576cec200aacf35a642824b53ad1e6 Oct 30 02:41:23.106424 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:23.106390 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cd6rj" event={"ID":"107bc8a5-c01b-4b78-aa29-765aab336604","Type":"ContainerStarted","Data":"8ae40967f9880f262525f70dbad8fba75d576cec200aacf35a642824b53ad1e6"} Oct 30 02:41:23.107656 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:23.107630 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t765c" event={"ID":"fb1a01bf-dc4f-430b-8b6d-d5569f85c981","Type":"ContainerStarted","Data":"f34ccfce0afc3dc0ebf9c79c7878c9f45f2b22f7d6c9a9b7edb556a88938acbf"} Oct 30 02:41:23.108818 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:23.108796 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xg5ng" event={"ID":"5f79e66d-ae26-455c-b111-82f211ffe507","Type":"ContainerStarted","Data":"ec4b55722b73847c22ccfa652ecfe716abc269944e17f3e3338b73b01c8a0a2b"} Oct 30 02:41:23.112776 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:23.112747 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:24.116140 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:24.116058 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xg5ng" event={"ID":"5f79e66d-ae26-455c-b111-82f211ffe507","Type":"ContainerStarted","Data":"ba1430389af55c65092c05b0c55d08631bd1c78e085e4c0086286ccba9f1acd2"} Oct 30 02:41:24.116140 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:24.116125 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xg5ng" event={"ID":"5f79e66d-ae26-455c-b111-82f211ffe507","Type":"ContainerStarted","Data":"7fd9a36c38dcef8dd09b354ccc03ff83b2a55aa71f6e25d7dddfcdf8f12554eb"} Oct 30 02:41:24.156383 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:24.156318 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xg5ng" podStartSLOduration=67.254507818 podStartE2EDuration="1m8.156300197s" totalImagesPullingTime="901.792379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:24.154200652 +0000 UTC m=+68.017239792" watchObservedRunningTime="2026-10-30 02:41:24.156300197 +0000 UTC m=+68.019339337" Oct 30 02:41:25.211408 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:25.211377 2573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-647f7b9c85-ldfjx"] Oct 30 02:41:27.130037 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:27.130002 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t765c" event={"ID":"fb1a01bf-dc4f-430b-8b6d-d5569f85c981","Type":"ContainerStarted","Data":"620b0a650bad69475e55ab3548321a804b685aa3325e59f1b89c18232be01ba5"} Oct 30 02:41:27.130497 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:27.130164 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:41:27.169657 ip-10-0-141-200 kubenswrapper[2573]: I1030 02:41:27.169569 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t765c" podStartSLOduration=68.074588477 podStartE2EDuration="1m11.169515849s" totalImagesPullingTime="3.094927372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:27.169515458 +0000 UTC m=+71.032554601" watchObservedRunningTime="2026-10-30 02:41:27.169515849 +0000 UTC m=+71.032554989" Oct 30 02:41:27.827435 ip-10-0-141-200 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 02:41:27.843021 ip-10-0-141-200 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 02:41:27.843190 ip-10-0-141-200 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 02:41:27.843238 ip-10-0-141-200 systemd[1]: kubelet.service: Consumed 4.738s CPU time. Oct 30 02:41:27.854326 ip-10-0-141-200 systemd[1]: Starting Kubernetes Kubelet... Oct 30 02:41:27.915563 ip-10-0-141-200 kubenswrapper[5882]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 02:41:27.915563 ip-10-0-141-200 kubenswrapper[5882]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 02:41:27.915563 ip-10-0-141-200 kubenswrapper[5882]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 02:41:27.915563 ip-10-0-141-200 kubenswrapper[5882]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 02:41:27.917593 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917512 5882 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:41:27.917593 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917593 5882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:41:27.917671 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917597 5882 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:41:27.917671 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917601 5882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:41:27.917671 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917604 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:41:27.917671 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917608 5882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:41:27.917671 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917611 5882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:41:27.917671 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917633 5882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:41:27.917671 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917638 5882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:41:27.917671 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917641 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:41:27.917671 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917643 5882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:41:27.917671 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917647 5882 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:41:27.917671 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917650 5882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:41:27.917671 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917653 5882 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:41:27.917671 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917656 5882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:41:27.917671 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917658 5882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:41:27.917671 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917661 5882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:41:27.917671 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917664 5882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:41:27.917671 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917666 5882 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:41:27.917671 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917669 5882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:41:27.917671 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917673 5882 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:41:27.917671 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917676 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:41:27.918162 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917679 5882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:41:27.918162 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917682 5882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:41:27.918162 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917684 5882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:41:27.918162 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917694 5882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:41:27.918162 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917697 5882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:41:27.918162 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917700 5882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:41:27.918162 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917703 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:41:27.918162 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917706 5882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:41:27.918162 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917708 5882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:41:27.918162 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917711 5882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:41:27.918162 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917714 5882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:41:27.918162 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917716 5882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:41:27.918162 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917719 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:41:27.918162 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917722 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:41:27.918162 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917724 5882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:41:27.918162 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917727 5882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:41:27.918162 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917731 5882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:41:27.918162 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917735 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:41:27.918162 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917738 5882 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:41:27.918637 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917741 5882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:41:27.918637 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917744 5882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:41:27.918637 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917746 5882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:41:27.918637 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917749 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:41:27.918637 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917752 5882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:41:27.918637 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917755 5882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:41:27.918637 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917757 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:41:27.918637 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917760 5882 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:41:27.918637 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917764 5882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:41:27.918637 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917767 5882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:41:27.918637 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917771 5882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:41:27.918637 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917774 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:41:27.918637 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917777 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:41:27.918637 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917780 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:41:27.918637 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917783 5882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:41:27.918637 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917786 5882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:41:27.918637 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917788 5882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:41:27.918637 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917791 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:41:27.918637 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917795 5882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:41:27.919101 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917797 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:41:27.919101 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917801 5882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:41:27.919101 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917803 5882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:41:27.919101 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917806 5882 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:41:27.919101 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917808 5882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:41:27.919101 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917811 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:41:27.919101 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917814 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:41:27.919101 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917816 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:41:27.919101 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917819 5882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:41:27.919101 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917822 5882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:41:27.919101 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917824 5882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:41:27.919101 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917827 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:41:27.919101 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917830 5882 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:41:27.919101 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917832 5882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:41:27.919101 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917835 5882 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:41:27.919101 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917838 5882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:41:27.919101 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917840 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:41:27.919101 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917843 5882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:41:27.919101 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917846 5882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:41:27.919568 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917848 5882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:41:27.919568 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917851 5882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:41:27.919568 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917854 5882 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:41:27.919568 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917857 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:41:27.919568 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917859 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:41:27.919568 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917862 5882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:41:27.919568 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917865 5882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:41:27.919568 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917867 5882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:41:27.919568 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917871 5882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:41:27.919568 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917874 5882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:41:27.919568 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917877 5882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:41:27.919568 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917879 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:41:27.919568 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917883 5882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:41:27.919568 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917886 5882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:41:27.919568 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917891 5882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:41:27.919568 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917894 5882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:41:27.919568 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917896 5882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:41:27.919568 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917899 5882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:41:27.919568 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917902 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:41:27.919568 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917904 5882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:41:27.920065 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917907 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:41:27.920065 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917910 5882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:41:27.920065 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917912 5882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:41:27.920065 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917915 5882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:41:27.920065 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917918 5882 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:41:27.920065 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917920 5882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:41:27.920065 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.917923 5882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:41:27.920065 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918668 5882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:41:27.920065 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918676 5882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:41:27.920065 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918679 5882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:41:27.920065 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918682 5882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:41:27.920065 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918684 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:41:27.920065 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918687 5882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:41:27.920065 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918690 5882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:41:27.920065 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918692 5882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:41:27.920065 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918695 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:41:27.920065 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918698 5882 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:41:27.920065 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918701 5882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:41:27.920065 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918704 5882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:41:27.920529 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918707 5882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:41:27.920529 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918710 5882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:41:27.920529 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918713 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:41:27.920529 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918715 5882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:41:27.920529 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918718 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:41:27.920529 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918721 5882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:41:27.920529 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918724 5882 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:41:27.920529 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918727 5882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:41:27.920529 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918730 5882 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:41:27.920529 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918732 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:41:27.920529 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918735 5882 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:41:27.920529 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918738 5882 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:41:27.920529 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918740 5882 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:41:27.920529 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918743 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:41:27.920529 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918745 5882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:41:27.920529 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918748 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:41:27.920529 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918751 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:41:27.920529 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918753 5882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:41:27.920529 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918756 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:41:27.920529 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918759 5882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:41:27.921074 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918761 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:41:27.921074 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918764 5882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:41:27.921074 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918767 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:41:27.921074 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918770 5882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:41:27.921074 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918772 5882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:41:27.921074 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918775 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:41:27.921074 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918777 5882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:41:27.921074 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918780 5882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:41:27.921074 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918783 5882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:41:27.921074 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918785 5882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:41:27.921074 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918788 5882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:41:27.921074 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918791 5882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:41:27.921074 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918793 5882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:41:27.921074 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918796 5882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:41:27.921074 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918798 5882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:41:27.921074 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918801 5882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:41:27.921074 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918804 5882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:41:27.921074 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918807 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:41:27.921074 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918809 5882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:41:27.921547 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918812 5882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:41:27.921547 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918815 5882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:41:27.921547 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918818 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:41:27.921547 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918820 5882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:41:27.921547 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918823 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:41:27.921547 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918826 5882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:41:27.921547 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918828 5882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:41:27.921547 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918831 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:41:27.921547 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918833 5882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:41:27.921547 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918836 5882 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:41:27.921547 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918839 5882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:41:27.921547 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918842 5882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:41:27.921547 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918846 5882 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:41:27.921547 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918848 5882 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:41:27.921547 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918851 5882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:41:27.921547 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918854 5882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:41:27.921547 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918856 5882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:41:27.921547 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918859 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:41:27.921547 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918861 5882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:41:27.921547 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918864 5882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:41:27.922050 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918874 5882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:41:27.922050 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918877 5882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:41:27.922050 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918880 5882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:41:27.922050 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918882 5882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:41:27.922050 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918885 5882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:41:27.922050 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918888 5882 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:41:27.922050 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918891 5882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:41:27.922050 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918893 5882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:41:27.922050 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918896 5882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:41:27.922050 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918899 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:41:27.922050 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918903 5882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:41:27.922050 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918906 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:41:27.922050 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918909 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:41:27.922050 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918911 5882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:41:27.922050 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918914 5882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:41:27.922050 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918917 5882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:41:27.922050 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918919 5882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:41:27.922050 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918922 5882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:41:27.922050 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918925 5882 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:41:27.922050 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918927 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:41:27.922531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918930 5882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:41:27.922531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918933 5882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:41:27.922531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918936 5882 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:41:27.922531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918939 5882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:41:27.922531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918941 5882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:41:27.922531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918945 5882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:41:27.922531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918950 5882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:41:27.922531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918953 5882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:41:27.922531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918956 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:41:27.922531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918958 5882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:41:27.922531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918961 5882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:41:27.922531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918966 5882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:41:27.922531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918970 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:41:27.922531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918973 5882 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:41:27.922531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.918976 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:41:27.922531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919445 5882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:41:27.922531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919451 5882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:41:27.922531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919454 5882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:41:27.922531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919457 5882 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:41:27.923008 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919460 5882 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:41:27.923008 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919462 5882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:41:27.923008 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919466 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:41:27.923008 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919468 5882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:41:27.923008 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919471 5882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:41:27.923008 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919474 5882 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:41:27.923008 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919477 5882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:41:27.923008 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919479 5882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:41:27.923008 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919482 5882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:41:27.923008 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919485 5882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:41:27.923008 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919490 5882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:41:27.923008 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919495 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:41:27.923008 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919498 5882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:41:27.923008 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919501 5882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:41:27.923008 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919503 5882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:41:27.923008 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919506 5882 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:41:27.923008 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919508 5882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:41:27.923008 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919512 5882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:41:27.923008 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919514 5882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:41:27.923008 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919517 5882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:41:27.923531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919520 5882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:41:27.923531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919522 5882 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:41:27.923531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919525 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:41:27.923531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919528 5882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:41:27.923531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919530 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:41:27.923531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919533 5882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:41:27.923531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919536 5882 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:41:27.923531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919538 5882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:41:27.923531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919541 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:41:27.923531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919544 5882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:41:27.923531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919546 5882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:41:27.923531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919549 5882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:41:27.923531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919551 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:41:27.923531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919554 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:41:27.923531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919557 5882 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:41:27.923531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919565 5882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:41:27.923531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919568 5882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:41:27.923531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919571 5882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:41:27.923531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919575 5882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:41:27.923531 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919579 5882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:41:27.924048 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919582 5882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:41:27.924048 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919584 5882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:41:27.924048 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919588 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:41:27.924048 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919593 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:41:27.924048 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919596 5882 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:41:27.924048 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919598 5882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:41:27.924048 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919601 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:41:27.924048 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919604 5882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:41:27.924048 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919607 5882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:41:27.924048 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919609 5882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:41:27.924048 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919625 5882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:41:27.924048 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919628 5882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:41:27.924048 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919631 5882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:41:27.924048 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919634 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:41:27.924048 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919636 5882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:41:27.924048 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919639 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:41:27.924048 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919642 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:41:27.924048 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919644 5882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:41:27.924048 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919647 5882 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:41:27.924516 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919650 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:41:27.924516 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919652 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:41:27.924516 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919655 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:41:27.924516 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919659 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:41:27.924516 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919661 5882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:41:27.924516 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919664 5882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:41:27.924516 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919667 5882 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:41:27.924516 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919669 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:41:27.924516 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919672 5882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:41:27.924516 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919681 5882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:41:27.924516 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919683 5882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:41:27.924516 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919686 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:41:27.924516 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919689 5882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:41:27.924516 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919692 5882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:41:27.924516 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919695 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:41:27.924516 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919698 5882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:41:27.924516 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919702 5882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:41:27.924516 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919705 5882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:41:27.924516 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919709 5882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:41:27.924991 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919713 5882 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:41:27.924991 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919716 5882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:41:27.924991 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919719 5882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:41:27.924991 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919722 5882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:41:27.924991 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919725 5882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:41:27.924991 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919728 5882 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:41:27.924991 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919731 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:41:27.924991 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919734 5882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:41:27.924991 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919736 5882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:41:27.924991 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919739 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:41:27.924991 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919741 5882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:41:27.924991 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919744 5882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:41:27.924991 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919747 5882 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:41:27.924991 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919750 5882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:41:27.924991 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919752 5882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:41:27.924991 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919755 5882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:41:27.924991 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919757 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:41:27.924991 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919760 5882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:41:27.924991 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919764 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919766 5882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919768 5882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919771 5882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919774 5882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.919782 5882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919885 5882 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919897 5882 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919903 5882 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919907 5882 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919914 5882 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919920 5882 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919927 5882 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919933 5882 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919939 5882 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919942 5882 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919946 5882 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919950 5882 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919953 5882 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919956 5882 flags.go:64] FLAG: --cgroup-root="" Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919959 5882 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919963 5882 flags.go:64] FLAG: --client-ca-file="" Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919965 5882 flags.go:64] FLAG: --cloud-provider="external" Oct 30 02:41:27.925470 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919968 5882 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919976 5882 flags.go:64] FLAG: --cluster-domain="" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919979 5882 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919983 5882 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919986 5882 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919990 5882 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919994 5882 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.919997 5882 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920000 5882 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920004 5882 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920007 5882 flags.go:64] FLAG: --contention-profiling="false" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920010 5882 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920013 5882 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920016 5882 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920019 5882 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920030 5882 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920033 5882 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920036 5882 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920039 5882 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920044 5882 flags.go:64] FLAG: --enable-server="true" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920048 5882 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920055 5882 flags.go:64] FLAG: --event-burst="100" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920058 5882 flags.go:64] FLAG: --event-qps="50" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920061 5882 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 02:41:27.926132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920065 5882 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920069 5882 flags.go:64] FLAG: --eviction-hard="" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920073 5882 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920076 5882 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920079 5882 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920082 5882 flags.go:64] FLAG: --eviction-soft="" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920086 5882 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920089 5882 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920092 5882 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920095 5882 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920098 5882 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920101 5882 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920104 5882 flags.go:64] FLAG: --feature-gates="" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920108 5882 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920111 5882 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920115 5882 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920118 5882 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920121 5882 flags.go:64] FLAG: --healthz-port="10248" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920125 5882 flags.go:64] FLAG: --help="false" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920128 5882 flags.go:64] FLAG: --hostname-override="ip-10-0-141-200.ec2.internal" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920131 5882 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920134 5882 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920137 5882 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920141 5882 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 02:41:27.927128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920151 5882 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920155 5882 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920158 5882 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920162 5882 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920166 5882 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920169 5882 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920173 5882 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920176 5882 flags.go:64] FLAG: --kube-reserved="" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920179 5882 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920182 5882 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920185 5882 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920188 5882 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920191 5882 flags.go:64] FLAG: --lock-file="" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920194 5882 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920197 5882 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920200 5882 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920206 5882 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920209 5882 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920212 5882 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920215 5882 flags.go:64] FLAG: --logging-format="text" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920218 5882 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920222 5882 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920225 5882 flags.go:64] FLAG: --manifest-url="" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920228 5882 flags.go:64] FLAG: --manifest-url-header="" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920233 5882 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 02:41:27.927764 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920236 5882 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920240 5882 flags.go:64] FLAG: --max-pods="110" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920243 5882 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920246 5882 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920249 5882 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920252 5882 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920256 5882 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920259 5882 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920269 5882 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920277 5882 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920282 5882 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920286 5882 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920290 5882 flags.go:64] FLAG: --pod-cidr="" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920293 5882 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920296 5882 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920299 5882 flags.go:64] FLAG: --pods-per-core="0" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920302 5882 flags.go:64] FLAG: --port="10250" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920305 5882 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920308 5882 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f639581efe4a7dc6" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920312 5882 flags.go:64] FLAG: --qos-reserved="" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920315 5882 flags.go:64] FLAG: --read-only-port="10255" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920318 5882 flags.go:64] FLAG: --register-node="true" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920321 5882 flags.go:64] FLAG: --register-with-taints="" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920325 5882 flags.go:64] FLAG: --registry-burst="10" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920328 5882 flags.go:64] FLAG: --registry-qps="5" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920331 5882 flags.go:64] FLAG: --reserved-cpus="" Oct 30 02:41:27.928374 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920334 5882 flags.go:64] FLAG: --reserved-memory="" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920338 5882 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920341 5882 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920344 5882 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920347 5882 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920350 5882 flags.go:64] FLAG: --runonce="false" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920353 5882 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920356 5882 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920360 5882 flags.go:64] FLAG: --seccomp-default="false" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920363 5882 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920365 5882 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920368 5882 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920372 5882 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920375 5882 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920378 5882 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920385 5882 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920390 5882 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920395 5882 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920398 5882 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920401 5882 flags.go:64] FLAG: --system-cgroups="" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920404 5882 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920410 5882 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920413 5882 flags.go:64] FLAG: --tls-cert-file="" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920415 5882 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920420 5882 flags.go:64] FLAG: --tls-min-version="" Oct 30 02:41:27.929028 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920423 5882 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920426 5882 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920429 5882 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920432 5882 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920435 5882 flags.go:64] FLAG: --v="2" Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920440 5882 flags.go:64] FLAG: --version="false" Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920444 5882 flags.go:64] FLAG: --vmodule="" Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920448 5882 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920451 5882 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920583 5882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920587 5882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920591 5882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920593 5882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920596 5882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920599 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920601 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920604 5882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920606 5882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920609 5882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920625 5882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920628 5882 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:41:27.929644 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920631 5882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:41:27.930156 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920634 5882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:41:27.930156 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920639 5882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:41:27.930156 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920642 5882 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:41:27.930156 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920645 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:41:27.930156 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920648 5882 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:41:27.930156 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920651 5882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:41:27.930156 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920653 5882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:41:27.930156 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920656 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:41:27.930156 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920659 5882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:41:27.930156 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920661 5882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:41:27.930156 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920664 5882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:41:27.930156 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920667 5882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:41:27.930156 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920669 5882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:41:27.930156 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920672 5882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:41:27.930156 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920674 5882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:41:27.930156 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920677 5882 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:41:27.930156 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920680 5882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:41:27.930156 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920683 5882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:41:27.930156 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920685 5882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:41:27.930156 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920688 5882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:41:27.930634 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920691 5882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:41:27.930634 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920693 5882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:41:27.930634 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920696 5882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:41:27.930634 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920699 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:41:27.930634 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920702 5882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:41:27.930634 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920704 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:41:27.930634 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920707 5882 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:41:27.930634 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920709 5882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:41:27.930634 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920713 5882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:41:27.930634 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920717 5882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:41:27.930634 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920720 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:41:27.930634 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920723 5882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:41:27.930634 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920726 5882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:41:27.930634 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920729 5882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:41:27.930634 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920732 5882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:41:27.930634 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920735 5882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:41:27.930634 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920738 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:41:27.930634 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920740 5882 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:41:27.930634 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920743 5882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:41:27.931113 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920746 5882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:41:27.931113 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920748 5882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:41:27.931113 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920751 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:41:27.931113 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920754 5882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:41:27.931113 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920756 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:41:27.931113 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920759 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:41:27.931113 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920761 5882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:41:27.931113 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920764 5882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:41:27.931113 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920767 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:41:27.931113 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920769 5882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:41:27.931113 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920772 5882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:41:27.931113 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920775 5882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:41:27.931113 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920778 5882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:41:27.931113 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920780 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:41:27.931113 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920783 5882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:41:27.931113 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920786 5882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:41:27.931113 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920789 5882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:41:27.931113 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920791 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:41:27.931113 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920794 5882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:41:27.931672 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920796 5882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:41:27.931672 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920799 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:41:27.931672 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920802 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:41:27.931672 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920804 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:41:27.931672 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920807 5882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:41:27.931672 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920810 5882 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:41:27.931672 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920812 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:41:27.931672 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920818 5882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:41:27.931672 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920822 5882 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:41:27.931672 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920826 5882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:41:27.931672 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920829 5882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:41:27.931672 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920831 5882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:41:27.931672 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920834 5882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:41:27.931672 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920837 5882 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:41:27.931672 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920839 5882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:41:27.931672 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920842 5882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:41:27.931672 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920845 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:41:27.931672 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920848 5882 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:41:27.931672 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920851 5882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:41:27.931672 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920853 5882 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:41:27.932253 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920856 5882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:41:27.932253 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920858 5882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:41:27.932253 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920861 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:41:27.932253 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920864 5882 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:41:27.932253 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920866 5882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:41:27.932253 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920869 5882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:41:27.932253 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920872 5882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:41:27.932253 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920874 5882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:41:27.932253 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920877 5882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:41:27.932253 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920880 5882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:41:27.932253 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920882 5882 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:41:27.932253 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920885 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:41:27.932253 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920887 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:41:27.932253 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920890 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:41:27.932253 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920893 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:41:27.932713 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.920918 5882 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 02:41:27.932713 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920956 5882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:41:27.932713 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920964 5882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:41:27.932713 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920967 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:41:27.932713 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920970 5882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:41:27.932713 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920974 5882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:41:27.932713 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920977 5882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:41:27.932713 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920980 5882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:41:27.932713 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920983 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:41:27.932713 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920986 5882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:41:27.932713 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920989 5882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:41:27.932713 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920991 5882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:41:27.932713 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920994 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:41:27.932713 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920997 5882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:41:27.932713 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.920999 5882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:41:27.932713 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921002 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:41:27.933102 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921005 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:41:27.933102 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921008 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:41:27.933102 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921010 5882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:41:27.933102 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921013 5882 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:41:27.933102 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921016 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:41:27.933102 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921019 5882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:41:27.933102 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921024 5882 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:41:27.933102 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921027 5882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:41:27.933102 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921029 5882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:41:27.933102 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921032 5882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:41:27.933102 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921035 5882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:41:27.933102 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921037 5882 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:41:27.933102 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921040 5882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:41:27.933102 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921043 5882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:41:27.933102 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921046 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:41:27.933102 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921048 5882 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:41:27.933102 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921051 5882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:41:27.933102 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921054 5882 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:41:27.933102 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921056 5882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:41:27.933102 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921059 5882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:41:27.933587 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921062 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:41:27.933587 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921066 5882 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:41:27.933587 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921069 5882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:41:27.933587 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921072 5882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:41:27.933587 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921075 5882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:41:27.933587 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921078 5882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:41:27.933587 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921081 5882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:41:27.933587 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921084 5882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:41:27.933587 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921086 5882 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:41:27.933587 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921089 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:41:27.933587 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921092 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:41:27.933587 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921094 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:41:27.933587 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921097 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:41:27.933587 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921100 5882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:41:27.933587 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921103 5882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:41:27.933587 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921106 5882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:41:27.933587 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921108 5882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:41:27.933587 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921111 5882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:41:27.933587 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921115 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:41:27.934126 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921118 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:41:27.934126 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921121 5882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:41:27.934126 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921123 5882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:41:27.934126 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921126 5882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:41:27.934126 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921129 5882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:41:27.934126 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921131 5882 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:41:27.934126 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921134 5882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:41:27.934126 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921137 5882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:41:27.934126 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921139 5882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:41:27.934126 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921142 5882 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:41:27.934126 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921144 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:41:27.934126 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921147 5882 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:41:27.934126 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921150 5882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:41:27.934126 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921152 5882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:41:27.934126 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921155 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:41:27.934126 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921158 5882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:41:27.934126 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921161 5882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:41:27.934126 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921163 5882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:41:27.934126 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921166 5882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:41:27.934126 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921169 5882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:41:27.934832 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921171 5882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:41:27.934832 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921174 5882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:41:27.934832 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921177 5882 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:41:27.934832 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921180 5882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:41:27.934832 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921182 5882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:41:27.934832 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921185 5882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:41:27.934832 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921187 5882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:41:27.934832 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921190 5882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:41:27.934832 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921193 5882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:41:27.934832 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921195 5882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:41:27.934832 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921198 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:41:27.934832 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921202 5882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:41:27.934832 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921205 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:41:27.934832 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921207 5882 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:41:27.934832 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921210 5882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:41:27.934832 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921214 5882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:41:27.934832 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921219 5882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:41:27.934832 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921222 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:41:27.934832 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921224 5882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:41:27.935472 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921227 5882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:41:27.935472 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921229 5882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:41:27.935472 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921232 5882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:41:27.935472 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921234 5882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:41:27.935472 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921237 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:41:27.935472 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921240 5882 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:41:27.935472 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921242 5882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:41:27.935472 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921245 5882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:41:27.935472 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921247 5882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:41:27.935472 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921250 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:41:27.935472 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921252 5882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:41:27.935472 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921255 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:41:27.935472 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.921258 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:41:27.935472 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.925304 5882 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 02:41:27.935472 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.925319 5882 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 02:41:27.935472 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.925335 5882 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 02:41:27.935472 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.925340 5882 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 02:41:27.935472 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925414 5882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:41:27.935472 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925419 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:41:27.936300 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925422 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:41:27.936300 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925426 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:41:27.936300 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925429 5882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:41:27.936300 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925431 5882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:41:27.936300 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925434 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:41:27.936300 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925438 5882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:41:27.936300 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925441 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:41:27.936300 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925443 5882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:41:27.936300 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925446 5882 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:41:27.936300 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925449 5882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:41:27.936300 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925452 5882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:41:27.936300 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925455 5882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:41:27.936300 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925464 5882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:41:27.936300 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925467 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:41:27.936300 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925469 5882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:41:27.936300 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925472 5882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:41:27.936300 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925475 5882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:41:27.936300 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925477 5882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:41:27.936300 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925481 5882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:41:27.937094 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925483 5882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:41:27.937094 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925486 5882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:41:27.937094 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925489 5882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:41:27.937094 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925492 5882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:41:27.937094 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925494 5882 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:41:27.937094 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925497 5882 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:41:27.937094 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925500 5882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:41:27.937094 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925503 5882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:41:27.937094 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925506 5882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:41:27.937094 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925508 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:41:27.937094 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925511 5882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:41:27.937094 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925514 5882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:41:27.937094 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925516 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:41:27.937094 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925519 5882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:41:27.937094 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925521 5882 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:41:27.937094 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925524 5882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:41:27.937094 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925527 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:41:27.937094 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925529 5882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:41:27.937094 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925532 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:41:27.937094 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925535 5882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:41:27.938187 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925537 5882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:41:27.938187 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925540 5882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:41:27.938187 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925543 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:41:27.938187 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925545 5882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:41:27.938187 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925548 5882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:41:27.938187 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925551 5882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:41:27.938187 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925559 5882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:41:27.938187 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925561 5882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:41:27.938187 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925564 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:41:27.938187 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925567 5882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:41:27.938187 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925569 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:41:27.938187 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925572 5882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:41:27.938187 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925574 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:41:27.938187 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925577 5882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:41:27.938187 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925579 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:41:27.938187 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925582 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:41:27.938187 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925584 5882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:41:27.938187 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925587 5882 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:41:27.938187 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925591 5882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:41:27.939026 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925595 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:41:27.939026 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925598 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:41:27.939026 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925601 5882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:41:27.939026 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925603 5882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:41:27.939026 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925606 5882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:41:27.939026 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925609 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:41:27.939026 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925611 5882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:41:27.939026 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925626 5882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:41:27.939026 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925629 5882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:41:27.939026 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925631 5882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:41:27.939026 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925634 5882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:41:27.939026 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925637 5882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:41:27.939026 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925641 5882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:41:27.939026 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925645 5882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:41:27.939026 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925648 5882 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:41:27.939026 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925651 5882 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:41:27.939026 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925654 5882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:41:27.939026 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925656 5882 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:41:27.939026 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925659 5882 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:41:27.939799 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925662 5882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:41:27.939799 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925670 5882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:41:27.939799 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925673 5882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:41:27.939799 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925676 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:41:27.939799 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925679 5882 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:41:27.939799 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925682 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:41:27.939799 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925684 5882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:41:27.939799 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925687 5882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:41:27.939799 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925689 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:41:27.939799 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925692 5882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:41:27.939799 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925695 5882 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:41:27.939799 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925697 5882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:41:27.939799 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925700 5882 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:41:27.939799 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925703 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:41:27.939799 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925705 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:41:27.939799 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925708 5882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:41:27.939799 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925711 5882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:41:27.939799 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925714 5882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:41:27.939799 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925716 5882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:41:27.939799 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925719 5882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:41:27.940712 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925722 5882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:41:27.940712 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925724 5882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:41:27.940712 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925727 5882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:41:27.940712 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925730 5882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:41:27.940712 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925733 5882 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:41:27.940712 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925735 5882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:41:27.940712 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.925738 5882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:41:27.940712 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.925764 5882 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 02:41:27.940712 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926498 5882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:41:27.940712 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926529 5882 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:41:27.940712 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926534 5882 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:41:27.940712 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926538 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:41:27.940712 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926543 5882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:41:27.940712 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926548 5882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:41:27.940712 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926578 5882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:41:27.940712 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926583 5882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:41:27.941396 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926593 5882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:41:27.941396 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926598 5882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:41:27.941396 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926603 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:41:27.941396 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926608 5882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:41:27.941396 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926627 5882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:41:27.941396 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926631 5882 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:41:27.941396 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926637 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:41:27.941396 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926642 5882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:41:27.941396 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926647 5882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:41:27.941396 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926651 5882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:41:27.941396 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926656 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:41:27.941396 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926662 5882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:41:27.941396 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926674 5882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:41:27.941396 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926680 5882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:41:27.941396 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926685 5882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:41:27.941396 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926692 5882 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:41:27.941396 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926696 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:41:27.941396 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926701 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:41:27.941396 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926705 5882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:41:27.942218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926710 5882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:41:27.942218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926714 5882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:41:27.942218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926718 5882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:41:27.942218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926723 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:41:27.942218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926727 5882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:41:27.942218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926737 5882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:41:27.942218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926741 5882 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:41:27.942218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926745 5882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:41:27.942218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926749 5882 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:41:27.942218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926753 5882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:41:27.942218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926758 5882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:41:27.942218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926762 5882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:41:27.942218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926794 5882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:41:27.942218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926799 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:41:27.942218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926804 5882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:41:27.942218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926808 5882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:41:27.942218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926812 5882 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:41:27.942218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926817 5882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:41:27.942218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926826 5882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:41:27.942218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926830 5882 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:41:27.943327 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926835 5882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:41:27.943327 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926839 5882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:41:27.943327 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926843 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:41:27.943327 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926848 5882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:41:27.943327 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926852 5882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:41:27.943327 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926856 5882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:41:27.943327 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926861 5882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:41:27.943327 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926872 5882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:41:27.943327 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926880 5882 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:41:27.943327 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926885 5882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:41:27.943327 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926894 5882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:41:27.943327 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926898 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:41:27.943327 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926902 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:41:27.943327 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926907 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:41:27.943327 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926911 5882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:41:27.943327 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926915 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:41:27.943327 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926921 5882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:41:27.943327 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926928 5882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:41:27.943327 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926933 5882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:41:27.944165 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926937 5882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:41:27.944165 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926942 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:41:27.944165 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926946 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:41:27.944165 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926950 5882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:41:27.944165 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926960 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:41:27.944165 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.926964 5882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:41:27.944165 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927036 5882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:41:27.944165 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927225 5882 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:41:27.944165 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927237 5882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:41:27.944165 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927240 5882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:41:27.944165 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927244 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:41:27.944165 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927247 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:41:27.944165 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927250 5882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:41:27.944165 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927253 5882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:41:27.944165 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927256 5882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:41:27.944165 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927259 5882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:41:27.944165 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927261 5882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:41:27.944165 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927264 5882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:41:27.944165 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927267 5882 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:41:27.944165 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927270 5882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:41:27.945021 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927272 5882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:41:27.945021 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927275 5882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:41:27.945021 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927278 5882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:41:27.945021 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927281 5882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:41:27.945021 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927286 5882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:41:27.945021 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927289 5882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:41:27.945021 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927292 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:41:27.945021 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927294 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:41:27.945021 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927298 5882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:41:27.945021 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927301 5882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:41:27.945021 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927304 5882 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:41:27.945021 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927307 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:41:27.945021 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927309 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:41:27.945021 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927312 5882 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:41:27.945021 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927315 5882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:41:27.945021 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927318 5882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:41:27.945021 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927320 5882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:41:27.945021 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927324 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:41:27.945021 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927326 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:41:27.945984 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927329 5882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:41:27.945984 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.927365 5882 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 02:41:27.945984 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927420 5882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:41:27.945984 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927426 5882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:41:27.945984 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927429 5882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:41:27.945984 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927432 5882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:41:27.945984 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927435 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:41:27.945984 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927438 5882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:41:27.945984 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927441 5882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:41:27.945984 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927444 5882 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:41:27.945984 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927447 5882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:41:27.945984 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927449 5882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:41:27.945984 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927452 5882 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:41:27.945984 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927455 5882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:41:27.945984 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927458 5882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:41:27.946707 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927461 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:41:27.946707 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927464 5882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:41:27.946707 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927469 5882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:41:27.946707 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927472 5882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:41:27.946707 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927475 5882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:41:27.946707 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927477 5882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:41:27.946707 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927480 5882 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:41:27.946707 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927482 5882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:41:27.946707 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927485 5882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:41:27.946707 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927488 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:41:27.946707 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927491 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:41:27.946707 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927493 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:41:27.946707 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927496 5882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:41:27.946707 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927499 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:41:27.946707 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927503 5882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:41:27.946707 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927507 5882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:41:27.946707 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927510 5882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:41:27.946707 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927513 5882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:41:27.946707 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927516 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:41:27.947575 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927519 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:41:27.947575 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927522 5882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:41:27.947575 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927525 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:41:27.947575 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927527 5882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:41:27.947575 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927530 5882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:41:27.947575 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927532 5882 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:41:27.947575 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927535 5882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:41:27.947575 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927538 5882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:41:27.947575 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927540 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:41:27.947575 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927543 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:41:27.947575 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927546 5882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:41:27.947575 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927548 5882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:41:27.947575 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927551 5882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:41:27.947575 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927553 5882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:41:27.947575 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927556 5882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:41:27.947575 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927560 5882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:41:27.947575 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927563 5882 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:41:27.947575 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927565 5882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:41:27.947575 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927568 5882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:41:27.947575 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927571 5882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:41:27.948495 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927573 5882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:41:27.948495 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927576 5882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:41:27.948495 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927579 5882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:41:27.948495 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927581 5882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:41:27.948495 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927584 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:41:27.948495 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927587 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:41:27.948495 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927590 5882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:41:27.948495 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927592 5882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:41:27.948495 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927595 5882 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:41:27.948495 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927598 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:41:27.948495 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927601 5882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:41:27.948495 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927604 5882 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:41:27.948495 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927606 5882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:41:27.948495 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927609 5882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:41:27.948495 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927612 5882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:41:27.948495 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927635 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:41:27.948495 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927639 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:41:27.948495 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927641 5882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:41:27.948495 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927644 5882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:41:27.948495 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927647 5882 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:41:27.949411 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927650 5882 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:41:27.949411 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927652 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:41:27.949411 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927655 5882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:41:27.949411 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927658 5882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:41:27.949411 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927660 5882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:41:27.949411 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927663 5882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:41:27.949411 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927665 5882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:41:27.949411 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927670 5882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:41:27.949411 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927674 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:41:27.949411 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927677 5882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:41:27.949411 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927680 5882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:41:27.949411 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927683 5882 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:41:27.949411 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927686 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:41:27.949411 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927689 5882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:41:27.949411 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927692 5882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:41:27.949411 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927694 5882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:41:27.949411 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927697 5882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:41:27.949411 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927704 5882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:41:27.949411 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927707 5882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:41:27.950110 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927713 5882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:41:27.950110 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927715 5882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:41:27.950110 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927718 5882 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:41:27.950110 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927721 5882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:41:27.950110 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927724 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:41:27.950110 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927727 5882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:41:27.950110 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927730 5882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:41:27.950110 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927732 5882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:41:27.950110 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927735 5882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:41:27.950110 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927738 5882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:41:27.950110 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927740 5882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:41:27.950110 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927743 5882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:41:27.950110 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927746 5882 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:41:27.950110 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927748 5882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:41:27.950110 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:27.927751 5882 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:41:27.950110 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.927835 5882 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 02:41:27.950110 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.928834 5882 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 02:41:27.950110 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.928889 5882 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 02:41:27.950110 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.929061 5882 server.go:1014] "Starting client certificate rotation" Oct 30 02:41:27.950578 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.929199 5882 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 02:41:27.950578 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.929248 5882 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:35:16 +0000 UTC" deadline="2028-04-16 17:26:17.779514265 +0000 UTC" Oct 30 02:41:27.950578 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.929276 5882 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12830h44m49.85024057s" Oct 30 02:41:27.950578 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.929592 5882 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 02:41:27.950578 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.930790 5882 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 02:41:27.950578 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.931997 5882 log.go:25] "Validated CRI v1 runtime API" Oct 30 02:41:27.950578 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.933969 5882 log.go:25] "Validated CRI v1 image API" Oct 30 02:41:27.950578 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.934688 5882 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 02:41:27.950578 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.938162 5882 fs.go:135] Filesystem UUIDs: map[3732df26-ba27-476d-9822-735e2b79e213:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8093e980-7c04-4d52-829e-7815d40d5cb7:/dev/nvme0n1p4] Oct 30 02:41:27.950861 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.938194 5882 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/04c75f0ca595be3dd141be5bdb280111218a73c872d38ce9aabd85c9b0055303/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/04c75f0ca595be3dd141be5bdb280111218a73c872d38ce9aabd85c9b0055303/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06f1fc5f300539b25f978d613245bd45f3002512ea9895cbbc01030dc677a6fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06f1fc5f300539b25f978d613245bd45f3002512ea9895cbbc01030dc677a6fb/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f063e2868918936ecd1e77ae7ede121a90f16a34ba53793387629cc2b64a9dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f063e2868918936ecd1e77ae7ede121a90f16a34ba53793387629cc2b64a9dc/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1fac83ee3517ccc5d3939b215e63adc0f51a43f4ef74247f00322b61fd72cf77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1fac83ee3517ccc5d3939b215e63adc0f51a43f4ef74247f00322b61fd72cf77/userdata/shm major:0 minor:306 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3821ebc90c34eb201d35c59d28586577e4e8e1b9449e09e43abadb4790b26bfe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3821ebc90c34eb201d35c59d28586577e4e8e1b9449e09e43abadb4790b26bfe/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f2def1ae94b30dfa54bff0a45d786d2122a87be552737268dabff27b5074954/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f2def1ae94b30dfa54bff0a45d786d2122a87be552737268dabff27b5074954/userdata/shm major:0 minor:367 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4496a3998e096416a749b67d66e8be8e63555a21349198a0e13abd6e02d41f42/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4496a3998e096416a749b67d66e8be8e63555a21349198a0e13abd6e02d41f42/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a23671b801854ce8d1c37d6fd1db927e2551c9dc7ae06b31bfb9ebbece18d70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a23671b801854ce8d1c37d6fd1db927e2551c9dc7ae06b31bfb9ebbece18d70/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51dca0b80913a1012976309f7599242c791c00fc8be56b17497b1c2b320bfcfa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51dca0b80913a1012976309f7599242c791c00fc8be56b17497b1c2b320bfcfa/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/719b462c9bbf11e8fa2bba324a94d6311f2196d28e35106f50f564ce80743ec1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/719b462c9bbf11e8fa2bba324a94d6311f2196d28e35106f50f564ce80743ec1/userdata/shm major:0 minor:365 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73dcebfdcf6a6cd70831cf2f336d84c18d7ee6b04438d24fc3b27bb82e6950b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73dcebfdcf6a6cd70831cf2f336d84c18d7ee6b04438d24fc3b27bb82e6950b4/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77b23e1fbe8c6d00890c298ddb1d734a5cda96d8d22d38085a5fe134dc0cb2e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77b23e1fbe8c6d00890c298ddb1d734a5cda96d8d22d38085a5fe134dc0cb2e3/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ae40967f9880f262525f70dbad8fba75d576cec200aacf35a642824b53ad1e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ae40967f9880f262525f70dbad8fba75d576cec200aacf35a642824b53ad1e6/userdata/shm major:0 minor:492 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9372225da62eb113e6c7f903c1f76b737db097a185b64543f4398c2771a34eb3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9372225da62eb113e6c7f903c1f76b737db097a185b64543f4398c2771a34eb3/userdata/shm major:0 minor:445 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a4b4cb3bc647cf0fc8c9caf8163268b564c3c437487f98c820765b01fa125db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a4b4cb3bc647cf0fc8c9caf8163268b564c3c437487f98c820765b01fa125db/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b00353db60ca1c907bcb72884ec92d7cc53c6fbc767b74875275535caab0190d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b00353db60ca1c907bcb72884ec92d7cc53c6fbc767b74875275535caab0190d/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ccbdacb6e094ffeacd537808abbcbacea482495ffdccda28718be721036e7aad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ccbdacb6e094ffeacd537808abbcbacea482495ffdccda28718be721036e7aad/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d12148f0921e69bfc84db330ac55a37d131909162ea932bd68035e64b3960f9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d12148f0921e69bfc84db330ac55a37d131909162ea932bd68035e64b3960f9b/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1b4d081681ce4b2aad6ac3cec34e6c4985e180ce1cec7b99aa5cea2831de066/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1b4d081681ce4b2aad6ac3cec34e6c4985e180ce1cec7b99aa5cea2831de066/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d43deb655200ec4bd4fab804d41f694a5038e3837f234a33f49e5a9130df168e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d43deb655200ec4bd4fab804d41f694a5038e3837f234a33f49e5a9130df168e/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d55a5603b6cd1e4f6a135c34bc6ede909abe96f23baea1fcea1d56ec925d2ef5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d55a5603b6cd1e4f6a135c34bc6ede909abe96f23baea1fcea1d56ec925d2ef5/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e65862c0dd5dda56c009b98387b19e171a3c6cc152f0395933c41dbca985b313/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e65862c0dd5dda56c009b98387b19e171a3c6cc152f0395933c41dbca985b313/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec4b55722b73847c22ccfa652ecfe716abc269944e17f3e3338b73b01c8a0a2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec4b55722b73847c22ccfa652ecfe716abc269944e17f3e3338b73b01c8a0a2b/userdata/shm major:0 minor:488 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f34ccfce0afc3dc0ebf9c79c7878c9f45f2b22f7d6c9a9b7edb556a88938acbf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f34ccfce0afc3dc0ebf9c79c7878c9f45f2b22f7d6c9a9b7edb556a88938acbf/userdata/shm major:0 minor:491 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fbe884666880840186124d47864fe6002ab614ab816e77422b0c73277819e344/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fbe884666880840186124d47864fe6002ab614ab816e77422b0c73277819e344/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c71f4fe-cc89-4856-accb-5e6c7c9452e2/volumes/kubernetes.io~projected/kube-api-access-4vwtj:{mountpoint:/var/lib/kubelet/pods/0c71f4fe-cc89-4856-accb-5e6c7c9452e2/volumes/kubernetes.io~projected/kube-api-access-4vwtj major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c71f4fe-cc89-4856-accb-5e6c7c9452e2/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0c71f4fe-cc89-4856-accb-5e6c7c9452e2/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c71f4fe-cc89-4856-accb-5e6c7c9452e2/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/0c71f4fe-cc89-4856-accb-5e6c7c9452e2/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/107bc8a5-c01b-4b78-aa29-765aab336604/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/107bc8a5-c01b-4b78-aa29-765aab336604/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:483 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~projected/kube-api-access-p6vgd:{mountpoint:/var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~projected/kube-api-access-p6vgd major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volume-subpaths/web-config/config-reloader/2 major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volume-subpaths/web-config/prometheus/15 major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~empty-dir/config-out major:0 minor:442 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~projected/kube-api-access-pl9n4:{mountpoint:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~projected/kube-api-access-pl9n4 major:0 minor:444 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~projected/tls-assets major:0 minor:440 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/config major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:439 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:438 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:435 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/web-config major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48721522-308a-4472-b687-d08caa72978d/volumes/kubernetes.io~projected/kube-api-access-89h27:{mountpoint:/var/lib/kubelet/pods/48721522-308a-4472-b687-d08caa72978d/volumes/kubernetes.io~projected/kube-api-access-89h27 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49f725c1-f41a-403d-b729-a170fc1fe687/volumes/kubernetes.io~projected/kube-api-access-nl2vg:{mountpoint:/var/lib/kubelet/pods/49f725c1-f41a-403d-b729-a170fc1fe687/volumes/kubernetes.io~projected/kube-api-access-nl2vg major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49f725c1-f41a-403d-b729-a170fc1fe687/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/49f725c1-f41a-403d-b729-a170fc1fe687/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49f725c1-f41a-403d-b729-a170fc1fe687/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/49f725c1-f41a-403d-b729-a170fc1fe687/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5077c63e-73b2-4a23-95f0-51edd0875a47/volumes/kubernetes.io~projected/kube-api-access-jm8sz:{mountpoint:/var/lib/kubelet/pods/5077c63e-73b2-4a23-95f0-51edd0875a47/volumes/kubernetes.io~projected/kube-api-access-jm8sz major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f79e66d-ae26-455c-b111-82f211ffe507/volumes/kubernetes.io~projected/kube-api-access-xq248:{mountpoint:/var/lib/kubelet/pods/5f79e66d-ae26-455c-b111-82f211ffe507/volumes/kubernetes.io~projected/kube-api-access-xq248 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f79e66d-ae26-455c-b111-82f211ffe507/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/5f79e66d-ae26-455c-b111-82f211ffe507/volumes/kubernetes.io~secret/metrics-certs major:0 minor:487 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cecaabc-af83-4bf9-b3e5-d446b1302f12/volumes/kubernetes.io~projected/kube-api-access-9tvzr:{mountpoint:/var/lib/kubelet/pods/7cecaabc-af83-4bf9-b3e5-d446b1302f12/volumes/kubernetes.io~projected/kube-api-access-9tvzr major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cecaabc-af83-4bf9-b3e5-d446b1302f12/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/7cecaabc-af83-4bf9-b3e5-d446b1302f12/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cecaabc-af83-4bf9-b3e5-d446b1302f12/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/7cecaabc-af83-4bf9-b3e5-d446b1302f12/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e93a8c5-35f7-459b-a8f1-f282130116cb/volumes/kubernetes.io~projected/kube-api-access-tccft:{mountpoint:/var/lib/kubelet/pods/7e93a8c5-35f7-459b-a8f1-f282130116cb/volumes/kubernetes.io~projected/kube-api-access-tccft major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e93a8c5-35f7-459b-a8f1-f282130116cb/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/7e93a8c5-35f7-459b-a8f1-f282130116cb/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f749cb9-f4b4-4db4-a580-2877108e718c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7f749cb9-f4b4-4db4-a580-2877108e718c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f749cb9-f4b4-4db4-a580-2877108e718c/volumes/kubernetes.io~projected/kube-api-access-stdbm:{mountpoint:/var/lib/kubelet/pods/7f749cb9-f4b4-4db4-a580-2877108e718c/volumes/kubernetes.io~projected/kube-api-access-stdbm major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f749cb9-f4b4-4db4-a580-2877108e718c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7f749cb9-f4b4-4db4-a580-2877108e718c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b33f658-c455-4aa4-ab62-7b5e3de65be5/volumes/kubernetes.io~projected/kube-api-access-v8bsm:{mountpoint:/var/lib/kubelet/pods/8b33f658-c455-4aa4-ab62-7b5e3de65be5/volumes/kubernetes.io~projected/kube-api-access-v8bsm major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/913d7a7f-bba5-4c75-bd3d-1727790ca220/volumes/kubernetes.io~projected/kube-api-access-74x77:{mountpoint:/var/lib/kubelet/pods/913d7a7f-bba5-4c75-bd3d-1727790ca220/volumes/kubernetes.io~projected/kube-api-access-74x77 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volume-subpaths/web-config/alertmanager/9 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volume-subpaths/web-config/config-reloader/6 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~empty-dir/config-out major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~projected/kube-api-access-kndrd:{mountpoint:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~projected/kube-api-access-kndrd major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~projected/tls-assets major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/config-volume major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/web-config major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d2896c8-e75c-4565-b5fe-3320f26fdd8a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/9d2896c8-e75c-4565-b5fe-3320f26fdd8a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d2896c8-e75c-4565-b5fe-3320f26fdd8a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/9d2896c8-e75c-4565-b5fe-3320f26fdd8a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d2896c8-e75c-4565-b5fe-3320f26fdd8a/volumes/kubernetes.io~projected/kube-api-access-6bflr:{mountpoint:/var/lib/kubelet/pods/9d2896c8-e75c-4565-b5fe-3320f26fdd8a/volumes/kubernetes.io~projected/kube-api-access-6bflr major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2f64d21-ae78-4fdd-b2e5-83c7629e563a/volumes/kubernetes.io~projected/kube-api-access-84khn:{mountpoint:/var/lib/kubelet/pods/b2f64d21-ae78-4fdd-b2e5-83c7629e563a/volumes/kubernetes.io~projected/kube-api-access-84khn major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2f64d21-ae78-4fdd-b2e5-83c7629e563a/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/b2f64d21-ae78-4fdd-b2e5-83c7629e563a/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be9a9474-03c1-489e-ab01-0e8e83d79daf/volumes/kubernetes.io~projected/kube-api-access-77cp8:{mountpoint:/var/lib/kubelet/pods/be9a9474-03c1-489e-ab01-0e8e83d79daf/volumes/kubernetes.io~projected/kube-api-access-77cp8 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be9a9474-03c1-489e-ab01-0e8e83d79daf/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/be9a9474-03c1-489e-ab01-0e8e83d79daf/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c42eb03a-a0d4-4509-8d16-ae82e1ac3c26/volumes/kubernetes.io~projected/kube-api-access-mwcpz:{mountpoint:/var/lib/kubelet/pods/c42eb03a-a0d4-4509-8d16-ae82e1ac3c26/volumes/kubernetes.io~projected/kube-api-access-mwcpz major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c42eb03a-a0d4-4509-8d16-ae82e1ac3c26/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/c42eb03a-a0d4-4509-8d16-ae82e1ac3c26/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e06d9d5b-be5f-46aa-a633-8ce9c9c37589/volumes/kubernetes.io~projected/kube-api-access-php67:{mountpoint:/var/lib/kubelet/pods/e06d9d5b-be5f-46aa-a633-8ce9c9c37589/volumes/kubernetes.io~projected/kube-api-access-php67 major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e06d9d5b-be5f-46aa-a633-8ce9c9c37589/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e06d9d5b-be5f-46aa-a633-8ce9c9c37589/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e06d9d5b-be5f-46aa-a633-8ce9c9c37589/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/e06d9d5b-be5f-46aa-a633-8ce9c9c37589/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1/volumes/kubernetes.io~projected/kube-api-access-stmvr:{mountpoint:/var/lib/kubelet/pods/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1/volumes/kubernetes.io~projected/kube-api-access-stmvr major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb1a01bf-dc4f-430b-8b6d-d5569f85c981/volumes/kubernetes.io~projected/kube-api-access-7d98p:{mountpoint:/var/lib/kubelet/pods/fb1a01bf-dc4f-430b-8b6d-d5569f85c981/volumes/kubernetes.io~projected/kube-api-access-7d98p major:0 minor:490 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb7cc1ad-e964-4602-9ce5-c838212826d2/volumes/kubernetes.io~projected/kube-api-access-g8x7w:{mountpoint:/var/lib/kubelet/pods/fb7cc1ad-e964-4602-9ce5-c838212826d2/volumes/kubernetes.io~projected/kube-api-access-g8x7w major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd320963-1dae-4be2-9352-15a810104d69/volumes/kubernetes.io~projected/kube-api-access-fqcx7:{mountpoint:/var/lib/kubelet/pods/fd320963-1dae-4be2-9352-15a810104d69/volumes/kubernetes.io~projected/kube-api-access-fqcx7 major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd320963-1dae-4be2-9352-15a810104d69/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/fd320963-1dae-4be2-9352-15a810104d69/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd320963-1dae-4be2-9352-15a810104d69/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/fd320963-1dae-4be2-9352-15a810104d69/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd320963-1dae-4be2-9352-15a810104d69/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/fd320963-1dae-4be2-9352-15a810104d69/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:338 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/1fe9b4b77adeb257f3c625593ad67318d1c7b7e4d31e11f93991d424911a4adb/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c1bcd5183638e563556837220e385a25b8fbc0ce370ae5acfc83d272200493b0/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/cf11492b027d73c4251bdf1893c988edbaacdcf10f0c07fb2227b786622ad9df/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/16a0c3337775d9247dcaafdc2eca31b49b8c43145641b03f559530e5d0bcd910/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/ae8fb102ce334541cfe4724a68976e928feac45a0ca527c15f303c426adb11ca/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/4d54983869b7cda976d2402f7dc3ac8d4e38bcd5cc4140860bb726a16489a836/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/cc0d7f8ced49ec5f0124802cdfb2b5808ee5fb7bdfc707cb49b3852f247bb5cd/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3ad0e81170534bab038848d02593d58f1875ff763f1b62b4a6d9c5d93a05d0fe/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/25c41212d015eb283e1857a5ce1f2c53d0d6e358ff63259d08d0eee3dcef579e/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/d86d145f4f86f8e698650ec3ec321108240207db6ae12c354861b7f5a96867c2/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/36977dde2c7dc80a87eb0deede988b4cb7168f2e387c8885fc4c22cf71451c4a/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/1e6986d14f1c944fd15e9b10c603b5985bfd53e6d8aaf9fd63c1e27e8d881030/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/d78eeff9857766fd612c6671e17618d0cc74d842afc75d83f599204c5c5e5623/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/65d0ce8d04128cf4eab118896495b4db877b1f9c3063023e40049d8c665c1781/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/cfa6edf6fba38ba4bfead24dea30edaee8f2ea8aab099c5435cce6c3b041e634/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/063e499c9a0628acc8e4826a1ea5da78871c44a32faccc6715f890e1c2429d49/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/953a63c08e762901a9d939ea9772d8ae7e1c45de6fd733b8d2761a2e56506dc0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/ca3c354524ef418ff5509ef6b4ac09508010c1f961d88b8fcf181a058be33b73/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/f4026366518ba870dd65850005efff51a4609015f9265ee53eff378c55e9a811/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/490ba969903a87b8a48ca2f503c996deee8aa4b8e6875d707225d4c4272208ee/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/91d6c541405e9971511d62af1fcba0b0cb82ecb76edaf76f1ab9f39390201989/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/f3242a8af8b18deb362fea4edbce9f9f4d26f43d35ae42f8077cd7621336dedc/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/587ef34bbdcefe21656e88b92dab9cf983fe95c0a8d0f3f4d2a4c233adb735fe/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/6609fb62aebc7ed171a7089e0b896ae1bfecfe579c7cc9bb005c9bb13d7cfe96/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/3de5b56291204d4f805c45f07521992a7efaeecbb4f62d9d4100ac65b18e3c52/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/9633657539af401216d8ba087c29fe8521065aad48b683b0c4c8e962c81f5d52/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/004d848bae8418702838e96b3acd0cfc02b095c01ed30b1421cf5b328915d7d6/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/e7812b12e54eee296f83e859f9941882714860def1ec050c03224a0907a4443b/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/f42a220cad2b3c3b92fe9e4cff67bdb2744ffc0a9ce3c6f988ee423813692ab1/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/1c295fafa7cc387fa37f34c32f258a7774d18a66011346070cab4c233e9d4b98/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/5a3e9c0efdb656c14085887f02bd0020df198622173efc40397a27869602ea29/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/57461f2dd95f1f3632e5134de23220430c216998589254463a499d4f8c60ebbc/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/7c90a1f4267f267ded037f314558e750915417f4b2adfeea6c89818603787a70/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/86c3e8cfe4ae1cbb6d7ed541bce7250a133ff74eb55d80cc1f7f3bdefe0b9991/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/11b027521cccd913d5b830854a7461f5958b68861dec2434d55a4e8f71c5652d/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/75b5e17810436aef96523cac10b30ad9732cfbdc57bfdf9a94774f6dbd3d4013/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/3abfa9a11286658281e15f0ac7d11c388d36d0b055c0c2fdebc7bf1714aacd12/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/0e7595794aed2f1c79b9a485ee884e0c927a15adeb937da3cc0ac3248cd9f163/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/4b88bc3e777ef7a7792652eb4ae064c02d3fb0b602cf8e083cd268ffa6db10a9/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/fdbfdd0e72e480499500c80643433376b906a2bce7ea9c0e7455689be5ba0e08/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/a661180a22f055f66fb1349428ad417efbb92c07f8f4423b682c9735bed15817/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/d0d04aeea2780a098f828668e7691f1936f5b95adf4dec021a110d15d37b159b/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/b6c819515b29b76bdccd48deb350b9d48bf6a08f7a65ac3428046b0318437b9f/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/35c1eeef155b7842f1ea9e0ace003ceb51cc8e1c3b0d1c1cf7767352a653eeb9/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/8be1d82e66db81b313b554bd6e749f4f482ef8af00db0581e882a84071ae1fe8/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/d63f5a68573b493f8b640f57260d4897bd559bfe3ef38b8072a480804c60dcfb/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/f42a00a661b9e544347e407c491d5d7b525c95e973066433e17fdce4f3b2c692/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-371:{mountpoint:/var/lib/containers/storage/overlay/af0f51c888c15d0fd711f1847038e8f002a60ba0f977116e3b3d2a7537957076/merged major:0 minor:371 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/595085a5f1eb88ac753414c2d3f0d4cd3238cc2e16126e323982f9ba0555bc52/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/c885f6d3fff410849253af3ed287d6b6f012050c58d53653632972c24c001a06/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/bb841d370e9b7691c44da23a409d962e7946c6f1ab3e4206e0819fcee2d95b19/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/3dd826e5607af28fe16d4072889ebbe197bceaab16d732642c8fe65e513ba1d5/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/07f68e27d2b70b7f107a935378b025a5fc7e1f3cba55b000b5a468e808ac8cbe/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/2a57acebde64d927fa2f6244bb163c3f9c359f5f1adf805783364277413cf184/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/eae53f78bb118c848e987db3b642aa0710b581d9d95ad5d7a5efeed00534acca/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/0a065553df4083b0afe1bf565a3c85b892ac66b4de0a3dee6b37ea3daff08eb0/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/028d625bd7eb2eedfa9578ff3857de9472d9c93b2ce25c0214e404bd272288c1/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/e62c62690ef7514decf360ea5473dda91b193018d1b7f2fd765d35fbd40c3dd2/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/8e4d1927e9b577c092d655da990fd744a63b90df6cc6c9a2bec09b847e2ee4d7/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/0d222b4a93a4d979f37af5af760324cc256c285548636e63fc5cf5738fda8aea/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/9ad47ef2daadf52112c20b8540d27cad71562e56d50a7a04e9d9f63f1b61c011/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-449:{mountpoint:/var/lib/containers/storage/overlay/c0a918ee1f202be72e1c917f1635e40ea13474cadd7fc40173874da4fdb96c64/merged major:0 minor:449 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/4dc2dac240c42575fefd723ee6827bc44844a33d192006e06b82a6f5fbf9360b/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-457:{mountpoint:/var/lib/containers/storage/overlay/7f55a51eca4da52b8e6b1cd8cb8b7c9a47abfee027420faf125d87ccf982b6ba/merged major:0 minor:457 fsType:overlay blockSize:0} overlay_0-463:{mountpoint:/var/lib/containers/storage/overlay/cb8627663a31c3345e8921961dc42319db92b2d684f29086f954878bbb919d91/merged major:0 minor:463 fsType:overlay blockSize:0} overlay_0-468:{mountpoint:/var/lib/containers/storage/overlay/930ad6c2fd0ff6538cd4edd8531371a6dbfc476f532e41c695c699cf0465a608/merged major:0 minor:468 fsType:overlay blockSize:0} overlay_0-473:{mountpoint:/var/lib/containers/storage/overlay/9c459dc611bb5438b400331093571a1b3e864153d1abe0e7f965a32aee73b415/merged major:0 minor:473 fsType:overlay blockSize:0} overlay_0-478:{mountpoint:/var/lib/containers/storage/overlay/1836a4c67eb003e5115f9d52da3c19f3dc31da5ed564dd61518d4d052c620852/merged major:0 minor:478 fsType:overlay blockSize:0} overlay_0-495:{mountpoint:/var/lib/containers/storage/overlay/61c91c688c19c209d862127a21b0b0de90277aafd078ee2cf64ddcf054b156b6/merged major:0 minor:495 fsType:overlay blockSize:0} overlay_0-497:{mountpoint:/var/lib/containers/storage/overlay/54ea615394fe72e2b9756e444aeae044a85687a87f707f8d5e04a2f1797759e1/merged major:0 minor:497 fsType:overlay blockSize:0} overlay_0-499:{mountpoint:/var/lib/containers/storage/overlay/4c143bd8a8e36088ab0f27d77dfafad59df3e8d0db264f9889fffce1a7f42eb3/merged major:0 minor:499 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/ccfa1780e991b18614f46a49076b0919d21428c2261c1a36279b6b937cdf4546/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-503:{mountpoint:/var/lib/containers/storage/overlay/393b3531452201dc7ff5591df39d191df9ec1c06b5b193ab596135332f98e6c7/merged major:0 minor:503 fsType:overlay blockSize:0} overlay_0-509:{mountpoint:/var/lib/containers/storage/overlay/abb0bb664ec94d4fa644b30258a2a6e3daad9bcd2ad9ca9b9db7db015bf6c522/merged major:0 minor:509 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/7397007758c2c6a2491539f28e7a551df5844f887a41b11a7e5be543d1d4b711/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-514:{mountpoint:/var/lib/containers/storage/overlay/e1894575294f733630215e5b9bc2a5dfa20ec9f2c3168fa2c202388812443e26/merged major:0 minor:514 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/d4b343d772a1c57d4de0fbc9a361eccd45abbdaff459423c7f344e030721f450/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/789d8c9403c256ffeccb51073fb4b766d9752865709876428cc4a37db6be144d/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/f0cbca4a48eafea5f1dfcfc0c526269c99ac0e6d1efd8c449bbad277ca3f3ec7/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/41069a8dde31d74eeb0a0f7189e9d9ecd25b8a6297bc1abefab0498347440e0d/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/be3c89352337383bc5112da6f30af40c9866b38e6bbfae1a6cda59fa0af0946e/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/95723d2989b7918cd33adddd3b27f6cbceb8843c39c185eeeb0b77a481aaf244/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/ae5fd50f3458fc92e5f34a14beaab529f8cd1bf12a913de37b2e96a951a5d020/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a93c925e57fc0b085ba338b63429ea709c13f06c6cef1ad72f47305d900bff61/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/5e91c1ac28dd754b8c4850ce5b149cdeaa8b805e6805d0c3363f8c54b55f2c04/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 02:41:27.952724 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.952236 5882 manager.go:217] Machine: {Timestamp:2026-10-30 02:41:27.951743791 +0000 UTC m=+0.066058755 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098068 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f2b591c7a88874aaa82913ca59e2e SystemUUID:ec2f2b59-1c7a-8887-4aaa-82913ca59e2e BootID:1a15ae50-fc30-480e-88dd-9d244d9262e9 Filesystems:[{Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-503 DeviceMajor:0 DeviceMinor:503 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f79e66d-ae26-455c-b111-82f211ffe507/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:487 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fb1a01bf-dc4f-430b-8b6d-d5569f85c981/volumes/kubernetes.io~projected/kube-api-access-7d98p DeviceMajor:0 DeviceMinor:490 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ae40967f9880f262525f70dbad8fba75d576cec200aacf35a642824b53ad1e6/userdata/shm DeviceMajor:0 DeviceMinor:492 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-495 DeviceMajor:0 DeviceMinor:495 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b33f658-c455-4aa4-ab62-7b5e3de65be5/volumes/kubernetes.io~projected/kube-api-access-v8bsm DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d12148f0921e69bfc84db330ac55a37d131909162ea932bd68035e64b3960f9b/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7e93a8c5-35f7-459b-a8f1-f282130116cb/volumes/kubernetes.io~projected/kube-api-access-tccft DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1fac83ee3517ccc5d3939b215e63adc0f51a43f4ef74247f00322b61fd72cf77/userdata/shm DeviceMajor:0 DeviceMinor:306 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f063e2868918936ecd1e77ae7ede121a90f16a34ba53793387629cc2b64a9dc/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/49f725c1-f41a-403d-b729-a170fc1fe687/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:358 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/49f725c1-f41a-403d-b729-a170fc1fe687/volumes/kubernetes.io~projected/kube-api-access-nl2vg DeviceMajor:0 DeviceMinor:363 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:429 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:439 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-468 DeviceMajor:0 DeviceMinor:468 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d2896c8-e75c-4565-b5fe-3320f26fdd8a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd320963-1dae-4be2-9352-15a810104d69/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec4b55722b73847c22ccfa652ecfe716abc269944e17f3e3338b73b01c8a0a2b/userdata/shm DeviceMajor:0 DeviceMinor:488 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-499 DeviceMajor:0 DeviceMinor:499 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4496a3998e096416a749b67d66e8be8e63555a21349198a0e13abd6e02d41f42/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e06d9d5b-be5f-46aa-a633-8ce9c9c37589/volumes/kubernetes.io~projected/kube-api-access-php67 DeviceMajor:0 DeviceMinor:249 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:441 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/b2f64d21-ae78-4fdd-b2e5-83c7629e563a/volumes/kubernetes.io~projected/kube-api-access-84khn DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-463 DeviceMajor:0 DeviceMinor:463 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be9a9474-03c1-489e-ab01-0e8e83d79daf/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0c71f4fe-cc89-4856-accb-5e6c7c9452e2/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:245 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e93a8c5-35f7-459b-a8f1-f282130116cb/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:329 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51dca0b80913a1012976309f7599242c791c00fc8be56b17497b1c2b320bfcfa/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-371 DeviceMajor:0 DeviceMinor:371 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ccbdacb6e094ffeacd537808abbcbacea482495ffdccda28718be721036e7aad/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/be9a9474-03c1-489e-ab01-0e8e83d79daf/volumes/kubernetes.io~projected/kube-api-access-77cp8 DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-473 DeviceMajor:0 DeviceMinor:473 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb7cc1ad-e964-4602-9ce5-c838212826d2/volumes/kubernetes.io~projected/kube-api-access-g8x7w DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f749cb9-f4b4-4db4-a580-2877108e718c/volumes/kubernetes.io~projected/kube-api-access-stdbm DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~projected/kube-api-access-kndrd DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-457 DeviceMajor:0 DeviceMinor:457 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/913d7a7f-bba5-4c75-bd3d-1727790ca220/volumes/kubernetes.io~projected/kube-api-access-74x77 DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fbe884666880840186124d47864fe6002ab614ab816e77422b0c73277819e344/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7cecaabc-af83-4bf9-b3e5-d446b1302f12/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:256 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:437 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e65862c0dd5dda56c009b98387b19e171a3c6cc152f0395933c41dbca985b313/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd320963-1dae-4be2-9352-15a810104d69/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-509 DeviceMajor:0 DeviceMinor:509 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-514 DeviceMajor:0 DeviceMinor:514 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7cecaabc-af83-4bf9-b3e5-d446b1302f12/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:254 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:434 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:440 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:443 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5077c63e-73b2-4a23-95f0-51edd0875a47/volumes/kubernetes.io~projected/kube-api-access-jm8sz DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3821ebc90c34eb201d35c59d28586577e4e8e1b9449e09e43abadb4790b26bfe/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:328 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:441 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-497 DeviceMajor:0 DeviceMinor:497 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c71f4fe-cc89-4856-accb-5e6c7c9452e2/volumes/kubernetes.io~projected/kube-api-access-4vwtj DeviceMajor:0 DeviceMinor:246 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~projected/kube-api-access-p6vgd DeviceMajor:0 DeviceMinor:333 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-478 DeviceMajor:0 DeviceMinor:478 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f79e66d-ae26-455c-b111-82f211ffe507/volumes/kubernetes.io~projected/kube-api-access-xq248 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e06d9d5b-be5f-46aa-a633-8ce9c9c37589/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:255 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c42eb03a-a0d4-4509-8d16-ae82e1ac3c26/volumes/kubernetes.io~projected/kube-api-access-mwcpz DeviceMajor:0 DeviceMinor:364 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:442 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:441 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/04c75f0ca595be3dd141be5bdb280111218a73c872d38ce9aabd85c9b0055303/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b2f64d21-ae78-4fdd-b2e5-83c7629e563a/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:330 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/49f725c1-f41a-403d-b729-a170fc1fe687/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:359 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c42eb03a-a0d4-4509-8d16-ae82e1ac3c26/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:357 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:438 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b00353db60ca1c907bcb72884ec92d7cc53c6fbc767b74875275535caab0190d/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d55a5603b6cd1e4f6a135c34bc6ede909abe96f23baea1fcea1d56ec925d2ef5/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fd320963-1dae-4be2-9352-15a810104d69/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:338 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/719b462c9bbf11e8fa2bba324a94d6311f2196d28e35106f50f564ce80743ec1/userdata/shm DeviceMajor:0 DeviceMinor:365 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9372225da62eb113e6c7f903c1f76b737db097a185b64543f4398c2771a34eb3/userdata/shm DeviceMajor:0 DeviceMinor:445 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-449 DeviceMajor:0 DeviceMinor:449 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:435 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/107bc8a5-c01b-4b78-aa29-765aab336604/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:483 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7f749cb9-f4b4-4db4-a580-2877108e718c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/48721522-308a-4472-b687-d08caa72978d/volumes/kubernetes.io~projected/kube-api-access-89h27 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d2896c8-e75c-4565-b5fe-3320f26fdd8a/volumes/kubernetes.io~projected/kube-api-access-6bflr DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a4b4cb3bc647cf0fc8c9caf8163268b564c3c437487f98c820765b01fa125db/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06f1fc5f300539b25f978d613245bd45f3002512ea9895cbbc01030dc677a6fb/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d43deb655200ec4bd4fab804d41f694a5038e3837f234a33f49e5a9130df168e/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:285 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:436 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f2def1ae94b30dfa54bff0a45d786d2122a87be552737268dabff27b5074954/userdata/shm DeviceMajor:0 DeviceMinor:367 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes/kubernetes.io~projected/kube-api-access-pl9n4 DeviceMajor:0 DeviceMinor:444 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c71f4fe-cc89-4856-accb-5e6c7c9452e2/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:241 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1/volumes/kubernetes.io~projected/kube-api-access-stmvr DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e06d9d5b-be5f-46aa-a633-8ce9c9c37589/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:257 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:441 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77b23e1fbe8c6d00890c298ddb1d734a5cda96d8d22d38085a5fe134dc0cb2e3/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1b4d081681ce4b2aad6ac3cec34e6c4985e180ce1cec7b99aa5cea2831de066/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f34ccfce0afc3dc0ebf9c79c7878c9f45f2b22f7d6c9a9b7edb556a88938acbf/userdata/shm DeviceMajor:0 DeviceMinor:491 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d2896c8-e75c-4565-b5fe-3320f26fdd8a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a23671b801854ce8d1c37d6fd1db927e2551c9dc7ae06b31bfb9ebbece18d70/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73dcebfdcf6a6cd70831cf2f336d84c18d7ee6b04438d24fc3b27bb82e6950b4/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7cecaabc-af83-4bf9-b3e5-d446b1302f12/volumes/kubernetes.io~projected/kube-api-access-9tvzr DeviceMajor:0 DeviceMinor:258 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:332 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd320963-1dae-4be2-9352-15a810104d69/volumes/kubernetes.io~projected/kube-api-access-fqcx7 DeviceMajor:0 DeviceMinor:343 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f749cb9-f4b4-4db4-a580-2877108e718c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:327 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:331 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0f063e286891893 MacAddress:26:a7:cb:23:65:29 Speed:10000 Mtu:8901} {Name:1fac83ee3517ccc MacAddress:aa:a1:d8:7e:02:66 Speed:10000 Mtu:8901} {Name:3821ebc90c34eb2 MacAddress:06:33:97:ce:b2:13 Speed:10000 Mtu:8901} {Name:3f2def1ae94b30d MacAddress:1e:eb:76:6b:99:f0 Speed:10000 Mtu:8901} {Name:51dca0b80913a10 MacAddress:36:53:e2:2d:d9:e6 Speed:10000 Mtu:8901} {Name:719b462c9bbf11e MacAddress:7e:f8:ca:32:14:99 Speed:10000 Mtu:8901} {Name:73dcebfdcf6a6cd MacAddress:82:c5:49:78:65:67 Speed:10000 Mtu:8901} {Name:8ae40967f9880f2 MacAddress:b6:fe:d7:2d:06:26 Speed:10000 Mtu:8901} {Name:9372225da62eb11 MacAddress:ea:b0:d9:c3:61:e2 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cb:40:0f:2d Speed:0 Mtu:9001} {Name:br-int MacAddress:76:ae:85:dc:92:20 Speed:0 Mtu:8901} {Name:d12148f0921e69b MacAddress:66:c9:6c:29:e8:2d Speed:10000 Mtu:8901} {Name:d43deb655200ec4 MacAddress:4a:cc:7e:ce:7d:2b Speed:10000 Mtu:8901} {Name:ec4b55722b73847 MacAddress:be:51:0d:9f:59:b5 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:cb:40:0f:2d Speed:0 Mtu:9001} {Name:f34ccfce0afc3dc MacAddress:ee:6a:aa:90:67:19 Speed:10000 Mtu:8901} {Name:fbe884666880840 MacAddress:52:a8:f6:d8:9a:f5 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ba:ac:54:9c:9d:de Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:62:a5:dd:54:8b:d0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 02:41:27.952980 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.952729 5882 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 02:41:27.952980 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.952782 5882 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 02:41:27.953190 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.953162 5882 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 02:41:27.953337 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.953192 5882 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-200.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 02:41:27.953381 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.953340 5882 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 02:41:27.953381 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.953349 5882 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 02:41:27.953381 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.953359 5882 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 02:41:27.953460 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.953384 5882 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 02:41:27.953498 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.953490 5882 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 02:41:27.953709 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.953699 5882 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 02:41:27.953823 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.953814 5882 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 02:41:27.953856 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.953852 5882 kubelet.go:493] "Attempting to sync node with API server" Oct 30 02:41:27.953886 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.953872 5882 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 02:41:27.953886 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.953882 5882 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 02:41:27.953950 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.953891 5882 kubelet.go:405] "Adding apiserver pod source" Oct 30 02:41:27.953950 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.953901 5882 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 02:41:27.954984 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.954970 5882 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 02:41:27.954984 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.954983 5882 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 02:41:27.955546 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.955528 5882 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 02:41:27.958568 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.958553 5882 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 02:41:27.958633 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.958569 5882 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 02:41:27.958784 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.958772 5882 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 02:41:27.958984 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.958972 5882 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 02:41:27.959017 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.959002 5882 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 02:41:27.959105 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.959095 5882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 02:41:27.959149 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.959110 5882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 02:41:27.959149 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.959116 5882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 02:41:27.959149 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.959122 5882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 02:41:27.959149 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.959127 5882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 02:41:27.959149 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.959133 5882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 02:41:27.959149 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.959139 5882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 02:41:27.959149 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.959145 5882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 02:41:27.959149 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.959152 5882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 02:41:27.959515 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.959158 5882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 02:41:27.959515 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.959167 5882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 02:41:27.959515 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.959176 5882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 02:41:27.959515 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.959202 5882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 02:41:27.959515 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.959209 5882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 02:41:27.962691 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.962676 5882 server.go:1263] "Started kubelet" Oct 30 02:41:27.963152 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.962762 5882 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 02:41:27.963152 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.962820 5882 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 02:41:27.963152 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.962941 5882 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 02:41:27.963341 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.963298 5882 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 02:41:27.963339 ip-10-0-141-200 systemd[1]: Started Kubernetes Kubelet. Oct 30 02:41:27.967810 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.965438 5882 server.go:317] "Adding debug handlers to kubelet server" Oct 30 02:41:27.969859 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.969641 5882 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 02:41:27.969859 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.969668 5882 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 02:41:27.969859 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.969709 5882 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:35:18 +0000 UTC" deadline="2028-04-29 12:51:37.753791304 +0000 UTC" Oct 30 02:41:27.969859 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.969727 5882 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13138h10m9.784067093s" Oct 30 02:41:27.969859 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:27.969805 5882 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-200.ec2.internal\" not found" Oct 30 02:41:27.969859 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.969859 5882 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 02:41:27.969859 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.969867 5882 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 02:41:27.970235 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.969930 5882 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 02:41:27.971535 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.971515 5882 factory.go:55] Registering systemd factory Oct 30 02:41:27.971665 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.971651 5882 factory.go:223] Registration of the systemd container factory successfully Oct 30 02:41:27.972211 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.972191 5882 factory.go:153] Registering CRI-O factory Oct 30 02:41:27.972211 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.972213 5882 factory.go:223] Registration of the crio container factory successfully Oct 30 02:41:27.972309 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.972269 5882 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 02:41:27.972309 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.972288 5882 factory.go:103] Registering Raw factory Oct 30 02:41:27.972309 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.972302 5882 manager.go:1199] Started watching for new ooms in manager Oct 30 02:41:27.972701 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.972675 5882 manager.go:319] Starting recovery of all containers Oct 30 02:41:27.972894 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.972846 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ccba25-b0ea-47c8-b664-105feea298a3" volumeName="kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 02:41:27.972977 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.972891 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ccba25-b0ea-47c8-b664-105feea298a3" volumeName="kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 02:41:27.972977 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.972910 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f725c1-f41a-403d-b729-a170fc1fe687" volumeName="kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert" seLinuxMountContext="" Oct 30 02:41:27.972977 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.972930 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e93a8c5-35f7-459b-a8f1-f282130116cb" volumeName="kubernetes.io/configmap/7e93a8c5-35f7-459b-a8f1-f282130116cb-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 02:41:27.972977 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.972947 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f749cb9-f4b4-4db4-a580-2877108e718c" volumeName="kubernetes.io/projected/7f749cb9-f4b4-4db4-a580-2877108e718c-kube-api-access-stdbm" seLinuxMountContext="" Oct 30 02:41:27.972977 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.972965 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992d8669-c914-486f-83ea-688dc69ca5a0" volumeName="kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-config-volume" seLinuxMountContext="" Oct 30 02:41:27.973214 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.972985 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992d8669-c914-486f-83ea-688dc69ca5a0" volumeName="kubernetes.io/projected/992d8669-c914-486f-83ea-688dc69ca5a0-tls-assets" seLinuxMountContext="" Oct 30 02:41:27.973214 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973004 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f749cb9-f4b4-4db4-a580-2877108e718c" volumeName="kubernetes.io/configmap/7f749cb9-f4b4-4db4-a580-2877108e718c-ovnkube-script-lib" seLinuxMountContext="" Oct 30 02:41:27.973214 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973023 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0" volumeName="kubernetes.io/projected/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-kube-api-access-p6vgd" seLinuxMountContext="" Oct 30 02:41:27.973214 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973064 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ccba25-b0ea-47c8-b664-105feea298a3" volumeName="kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 02:41:27.973214 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973081 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ccba25-b0ea-47c8-b664-105feea298a3" volumeName="kubernetes.io/projected/43ccba25-b0ea-47c8-b664-105feea298a3-kube-api-access-pl9n4" seLinuxMountContext="" Oct 30 02:41:27.973214 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973098 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ccba25-b0ea-47c8-b664-105feea298a3" volumeName="kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 02:41:27.973214 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973112 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5077c63e-73b2-4a23-95f0-51edd0875a47" volumeName="kubernetes.io/projected/5077c63e-73b2-4a23-95f0-51edd0875a47-kube-api-access-jm8sz" seLinuxMountContext="" Oct 30 02:41:27.973214 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973126 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cecaabc-af83-4bf9-b3e5-d446b1302f12" volumeName="kubernetes.io/secret/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 02:41:27.973214 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973140 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d2896c8-e75c-4565-b5fe-3320f26fdd8a" volumeName="kubernetes.io/projected/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-kube-api-access-6bflr" seLinuxMountContext="" Oct 30 02:41:27.973214 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973153 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ccba25-b0ea-47c8-b664-105feea298a3" volumeName="kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 02:41:27.973214 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973166 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0" volumeName="kubernetes.io/configmap/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-metrics-client-ca" seLinuxMountContext="" Oct 30 02:41:27.973214 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973180 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0" volumeName="kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-grpc-tls" seLinuxMountContext="" Oct 30 02:41:27.973214 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973191 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ccba25-b0ea-47c8-b664-105feea298a3" volumeName="kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 02:41:27.973214 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973203 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f725c1-f41a-403d-b729-a170fc1fe687" volumeName="kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert" seLinuxMountContext="" Oct 30 02:41:27.973214 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973217 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f79e66d-ae26-455c-b111-82f211ffe507" volumeName="kubernetes.io/projected/5f79e66d-ae26-455c-b111-82f211ffe507-kube-api-access-xq248" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973230 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0" volumeName="kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973243 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ccba25-b0ea-47c8-b664-105feea298a3" volumeName="kubernetes.io/empty-dir/43ccba25-b0ea-47c8-b664-105feea298a3-config-out" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973256 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f79e66d-ae26-455c-b111-82f211ffe507" volumeName="kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973268 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992d8669-c914-486f-83ea-688dc69ca5a0" volumeName="kubernetes.io/projected/992d8669-c914-486f-83ea-688dc69ca5a0-kube-api-access-kndrd" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973282 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c71f4fe-cc89-4856-accb-5e6c7c9452e2" volumeName="kubernetes.io/secret/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973294 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f725c1-f41a-403d-b729-a170fc1fe687" volumeName="kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973307 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ccba25-b0ea-47c8-b664-105feea298a3" volumeName="kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973338 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ccba25-b0ea-47c8-b664-105feea298a3" volumeName="kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973354 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992d8669-c914-486f-83ea-688dc69ca5a0" volumeName="kubernetes.io/empty-dir/992d8669-c914-486f-83ea-688dc69ca5a0-config-out" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973366 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992d8669-c914-486f-83ea-688dc69ca5a0" volumeName="kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973401 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e754f0c3-6ac7-4e3e-ab0e-c343b673abc1" volumeName="kubernetes.io/projected/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1-kube-api-access-stmvr" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973415 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd320963-1dae-4be2-9352-15a810104d69" volumeName="kubernetes.io/configmap/fd320963-1dae-4be2-9352-15a810104d69-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973459 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ccba25-b0ea-47c8-b664-105feea298a3" volumeName="kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973472 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48721522-308a-4472-b687-d08caa72978d" volumeName="kubernetes.io/configmap/48721522-308a-4472-b687-d08caa72978d-multus-daemon-config" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973509 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992d8669-c914-486f-83ea-688dc69ca5a0" volumeName="kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-web-config" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973522 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992d8669-c914-486f-83ea-688dc69ca5a0" volumeName="kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-cluster-tls-config" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973544 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be9a9474-03c1-489e-ab01-0e8e83d79daf" volumeName="kubernetes.io/projected/be9a9474-03c1-489e-ab01-0e8e83d79daf-kube-api-access-77cp8" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973593 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0" volumeName="kubernetes.io/empty-dir/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-tmp" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973610 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0" volumeName="kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973637 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992d8669-c914-486f-83ea-688dc69ca5a0" volumeName="kubernetes.io/configmap/992d8669-c914-486f-83ea-688dc69ca5a0-metrics-client-ca" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973658 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e06d9d5b-be5f-46aa-a633-8ce9c9c37589" volumeName="kubernetes.io/configmap/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-metrics-client-ca" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973671 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0" volumeName="kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973684 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f749cb9-f4b4-4db4-a580-2877108e718c" volumeName="kubernetes.io/configmap/7f749cb9-f4b4-4db4-a580-2877108e718c-ovnkube-config" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973695 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e754f0c3-6ac7-4e3e-ab0e-c343b673abc1" volumeName="kubernetes.io/configmap/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1-serviceca" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973708 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd320963-1dae-4be2-9352-15a810104d69" volumeName="kubernetes.io/secret/fd320963-1dae-4be2-9352-15a810104d69-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973720 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ccba25-b0ea-47c8-b664-105feea298a3" volumeName="kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973736 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f749cb9-f4b4-4db4-a580-2877108e718c" volumeName="kubernetes.io/secret/7f749cb9-f4b4-4db4-a580-2877108e718c-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973751 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992d8669-c914-486f-83ea-688dc69ca5a0" volumeName="kubernetes.io/empty-dir/992d8669-c914-486f-83ea-688dc69ca5a0-alertmanager-main-db" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973765 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f725c1-f41a-403d-b729-a170fc1fe687" volumeName="kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973777 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d2896c8-e75c-4565-b5fe-3320f26fdd8a" volumeName="kubernetes.io/empty-dir/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-tmp" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973792 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ccba25-b0ea-47c8-b664-105feea298a3" volumeName="kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-config" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973804 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f725c1-f41a-403d-b729-a170fc1fe687" volumeName="kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973848 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b33f658-c455-4aa4-ab62-7b5e3de65be5" volumeName="kubernetes.io/empty-dir/8b33f658-c455-4aa4-ab62-7b5e3de65be5-tmp-dir" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973862 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c71f4fe-cc89-4856-accb-5e6c7c9452e2" volumeName="kubernetes.io/projected/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-kube-api-access-4vwtj" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973875 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f725c1-f41a-403d-b729-a170fc1fe687" volumeName="kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973907 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e06d9d5b-be5f-46aa-a633-8ce9c9c37589" volumeName="kubernetes.io/configmap/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973918 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b33f658-c455-4aa4-ab62-7b5e3de65be5" volumeName="kubernetes.io/projected/8b33f658-c455-4aa4-ab62-7b5e3de65be5-kube-api-access-v8bsm" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973931 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="913d7a7f-bba5-4c75-bd3d-1727790ca220" volumeName="kubernetes.io/projected/913d7a7f-bba5-4c75-bd3d-1727790ca220-kube-api-access-74x77" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973948 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e06d9d5b-be5f-46aa-a633-8ce9c9c37589" volumeName="kubernetes.io/secret/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 02:41:27.973938 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973961 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c71f4fe-cc89-4856-accb-5e6c7c9452e2" volumeName="kubernetes.io/configmap/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-metrics-client-ca" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973973 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0" volumeName="kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973986 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ccba25-b0ea-47c8-b664-105feea298a3" volumeName="kubernetes.io/projected/43ccba25-b0ea-47c8-b664-105feea298a3-tls-assets" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.973999 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb" volumeName="kubernetes.io/secret/6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb-agent-certs" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974011 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992d8669-c914-486f-83ea-688dc69ca5a0" volumeName="kubernetes.io/configmap/992d8669-c914-486f-83ea-688dc69ca5a0-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974025 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be9a9474-03c1-489e-ab01-0e8e83d79daf" volumeName="kubernetes.io/secret/be9a9474-03c1-489e-ab01-0e8e83d79daf-cert" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974042 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e93a8c5-35f7-459b-a8f1-f282130116cb" volumeName="kubernetes.io/secret/7e93a8c5-35f7-459b-a8f1-f282130116cb-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974054 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992d8669-c914-486f-83ea-688dc69ca5a0" volumeName="kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974066 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f725c1-f41a-403d-b729-a170fc1fe687" volumeName="kubernetes.io/empty-dir/49f725c1-f41a-403d-b729-a170fc1fe687-tmp" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974078 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="913d7a7f-bba5-4c75-bd3d-1727790ca220" volumeName="kubernetes.io/configmap/913d7a7f-bba5-4c75-bd3d-1727790ca220-cni-binary-copy" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974089 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e06d9d5b-be5f-46aa-a633-8ce9c9c37589" volumeName="kubernetes.io/projected/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-api-access-php67" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974102 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd320963-1dae-4be2-9352-15a810104d69" volumeName="kubernetes.io/configmap/fd320963-1dae-4be2-9352-15a810104d69-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974114 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e93a8c5-35f7-459b-a8f1-f282130116cb" volumeName="kubernetes.io/projected/7e93a8c5-35f7-459b-a8f1-f282130116cb-kube-api-access-tccft" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974127 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f749cb9-f4b4-4db4-a580-2877108e718c" volumeName="kubernetes.io/configmap/7f749cb9-f4b4-4db4-a580-2877108e718c-env-overrides" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974139 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e06d9d5b-be5f-46aa-a633-8ce9c9c37589" volumeName="kubernetes.io/empty-dir/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-volume-directive-shadow" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974150 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd320963-1dae-4be2-9352-15a810104d69" volumeName="kubernetes.io/empty-dir/fd320963-1dae-4be2-9352-15a810104d69-audit-log" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974164 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb7cc1ad-e964-4602-9ce5-c838212826d2" volumeName="kubernetes.io/projected/fb7cc1ad-e964-4602-9ce5-c838212826d2-kube-api-access-g8x7w" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974180 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cecaabc-af83-4bf9-b3e5-d446b1302f12" volumeName="kubernetes.io/projected/7cecaabc-af83-4bf9-b3e5-d446b1302f12-kube-api-access-9tvzr" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974193 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cecaabc-af83-4bf9-b3e5-d446b1302f12" volumeName="kubernetes.io/secret/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-tls" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974206 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="107bc8a5-c01b-4b78-aa29-765aab336604" volumeName="kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974218 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48721522-308a-4472-b687-d08caa72978d" volumeName="kubernetes.io/projected/48721522-308a-4472-b687-d08caa72978d-kube-api-access-89h27" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974233 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cecaabc-af83-4bf9-b3e5-d446b1302f12" volumeName="kubernetes.io/configmap/7cecaabc-af83-4bf9-b3e5-d446b1302f12-metrics-client-ca" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974249 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="913d7a7f-bba5-4c75-bd3d-1727790ca220" volumeName="kubernetes.io/configmap/913d7a7f-bba5-4c75-bd3d-1727790ca220-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974263 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c42eb03a-a0d4-4509-8d16-ae82e1ac3c26" volumeName="kubernetes.io/secret/c42eb03a-a0d4-4509-8d16-ae82e1ac3c26-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974277 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f725c1-f41a-403d-b729-a170fc1fe687" volumeName="kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974290 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cecaabc-af83-4bf9-b3e5-d446b1302f12" volumeName="kubernetes.io/configmap/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974303 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e93a8c5-35f7-459b-a8f1-f282130116cb" volumeName="kubernetes.io/empty-dir/7e93a8c5-35f7-459b-a8f1-f282130116cb-data-volume" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974316 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2f64d21-ae78-4fdd-b2e5-83c7629e563a" volumeName="kubernetes.io/configmap/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-config-volume" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974327 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd320963-1dae-4be2-9352-15a810104d69" volumeName="kubernetes.io/secret/fd320963-1dae-4be2-9352-15a810104d69-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974342 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c71f4fe-cc89-4856-accb-5e6c7c9452e2" volumeName="kubernetes.io/secret/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974354 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ccba25-b0ea-47c8-b664-105feea298a3" volumeName="kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974366 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ccba25-b0ea-47c8-b664-105feea298a3" volumeName="kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-web-config" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974380 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2f64d21-ae78-4fdd-b2e5-83c7629e563a" volumeName="kubernetes.io/projected/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-kube-api-access-84khn" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974393 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2f64d21-ae78-4fdd-b2e5-83c7629e563a" volumeName="kubernetes.io/secret/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-metrics-tls" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974404 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ccba25-b0ea-47c8-b664-105feea298a3" volumeName="kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974419 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="913d7a7f-bba5-4c75-bd3d-1727790ca220" volumeName="kubernetes.io/configmap/913d7a7f-bba5-4c75-bd3d-1727790ca220-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974433 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c42eb03a-a0d4-4509-8d16-ae82e1ac3c26" volumeName="kubernetes.io/projected/c42eb03a-a0d4-4509-8d16-ae82e1ac3c26-kube-api-access-mwcpz" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974447 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb1a01bf-dc4f-430b-8b6d-d5569f85c981" volumeName="kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974461 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb7cc1ad-e964-4602-9ce5-c838212826d2" volumeName="kubernetes.io/configmap/fb7cc1ad-e964-4602-9ce5-c838212826d2-iptables-alerter-script" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974473 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0" volumeName="kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974485 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ccba25-b0ea-47c8-b664-105feea298a3" volumeName="kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-grpc-tls" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974498 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cecaabc-af83-4bf9-b3e5-d446b1302f12" volumeName="kubernetes.io/empty-dir/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-textfile" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974512 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ccba25-b0ea-47c8-b664-105feea298a3" volumeName="kubernetes.io/empty-dir/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-db" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974525 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992d8669-c914-486f-83ea-688dc69ca5a0" volumeName="kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974539 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb" volumeName="kubernetes.io/configmap/6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb-konnectivity-ca" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974552 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992d8669-c914-486f-83ea-688dc69ca5a0" volumeName="kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974565 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ccba25-b0ea-47c8-b664-105feea298a3" volumeName="kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974579 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48721522-308a-4472-b687-d08caa72978d" volumeName="kubernetes.io/configmap/48721522-308a-4472-b687-d08caa72978d-cni-binary-copy" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974590 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d2896c8-e75c-4565-b5fe-3320f26fdd8a" volumeName="kubernetes.io/empty-dir/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-tuned" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974602 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2f64d21-ae78-4fdd-b2e5-83c7629e563a" volumeName="kubernetes.io/empty-dir/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-tmp-dir" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974629 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd320963-1dae-4be2-9352-15a810104d69" volumeName="kubernetes.io/projected/fd320963-1dae-4be2-9352-15a810104d69-kube-api-access-fqcx7" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974641 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e06d9d5b-be5f-46aa-a633-8ce9c9c37589" volumeName="kubernetes.io/secret/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974671 5882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd320963-1dae-4be2-9352-15a810104d69" volumeName="kubernetes.io/secret/fd320963-1dae-4be2-9352-15a810104d69-client-ca-bundle" seLinuxMountContext="" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974684 5882 reconstruct.go:97] "Volume reconstruction finished" Oct 30 02:41:27.975442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.974691 5882 reconciler.go:29] "Reconciler: start to sync state" Oct 30 02:41:27.997954 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:27.997919 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 02:41:28.001130 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.001099 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 02:41:28.001939 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.001909 5882 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 02:41:28.003332 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.003312 5882 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 02:41:28.003453 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.003347 5882 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 02:41:28.003453 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.003370 5882 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 02:41:28.003453 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:28.003411 5882 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 02:41:28.017435 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.017404 5882 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:41:28.017596 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.017577 5882 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 02:41:28.021148 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.021119 5882 generic.go:356] "Generic (PLEG): container finished" podID="43ccba25-b0ea-47c8-b664-105feea298a3" containerID="53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb" exitCode=0 Oct 30 02:41:28.028108 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.028084 5882 generic.go:356] "Generic (PLEG): container finished" podID="992d8669-c914-486f-83ea-688dc69ca5a0" containerID="8171d08cc27a8278b5dfe6c057eb59c5a9992947a7555c85c48f4780cf193488" exitCode=0 Oct 30 02:41:28.029504 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.029484 5882 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:41:28.041487 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.041462 5882 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:41:28.041598 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.041511 5882 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:41:28.053392 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.053370 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 02:41:28.054759 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.054735 5882 generic.go:356] "Generic (PLEG): container finished" podID="7f749cb9-f4b4-4db4-a580-2877108e718c" containerID="5a2ae849a76af523e6d52f9d57c5b9ec835538192fd42df0b6ae7fe9d16cedc8" exitCode=1 Oct 30 02:41:28.064466 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.064440 5882 generic.go:356] "Generic (PLEG): container finished" podID="913d7a7f-bba5-4c75-bd3d-1727790ca220" containerID="d4b1947d35f3f81b39f13f8975784b7523dfb1bf2a4f4ee8ddb35f734cd17409" exitCode=0 Oct 30 02:41:28.064466 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.064466 5882 generic.go:356] "Generic (PLEG): container finished" podID="913d7a7f-bba5-4c75-bd3d-1727790ca220" containerID="b37278b8239e3875c96e11b030f206b58b509b5f29a246aa5ef64b3a5e93386b" exitCode=0 Oct 30 02:41:28.064649 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.064476 5882 generic.go:356] "Generic (PLEG): container finished" podID="913d7a7f-bba5-4c75-bd3d-1727790ca220" containerID="abd59f82eb8c5ba752fb7ac5a968802829ab0c3e341593d05d2df4a4d628fe64" exitCode=0 Oct 30 02:41:28.064649 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.064487 5882 generic.go:356] "Generic (PLEG): container finished" podID="913d7a7f-bba5-4c75-bd3d-1727790ca220" containerID="e62d03f5c935645288cc58e84e92bc4c4f8f8cc54f1c2f3535c11e67d8bd2ea8" exitCode=0 Oct 30 02:41:28.064649 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.064496 5882 generic.go:356] "Generic (PLEG): container finished" podID="913d7a7f-bba5-4c75-bd3d-1727790ca220" containerID="8f24fc7ec6ab978325ed17f02a396648cb53e63d0430eb73ce3c97e15df3db44" exitCode=0 Oct 30 02:41:28.064649 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.064504 5882 generic.go:356] "Generic (PLEG): container finished" podID="913d7a7f-bba5-4c75-bd3d-1727790ca220" containerID="c9d85b1f0972e6497edfa002179a68141edbda4bf07cbe91884ac3111fb36f11" exitCode=0 Oct 30 02:41:28.067933 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.067908 5882 generic.go:356] "Generic (PLEG): container finished" podID="6496276491b4aff226133cbf4e8c9798" containerID="4972329a555a16e7f846265b3e466e660d22139f947c63691d4259b3b469346c" exitCode=0 Oct 30 02:41:28.073699 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.073676 5882 generic.go:356] "Generic (PLEG): container finished" podID="7cecaabc-af83-4bf9-b3e5-d446b1302f12" containerID="03dba5fd4f38b874d3c102b40a79af1a1c01bfb20d57608b4a1e64a4ce225891" exitCode=0 Oct 30 02:41:28.096706 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.096652 5882 manager.go:324] Recovery completed Oct 30 02:41:28.103476 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:28.103456 5882 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 02:41:28.122412 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.122383 5882 cpu_manager.go:226] "Starting" policy="none" Oct 30 02:41:28.122412 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.122408 5882 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 02:41:28.122685 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.122435 5882 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 02:41:28.122858 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.122823 5882 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 02:41:28.123075 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.123042 5882 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 02:41:28.123106 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.123077 5882 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 02:41:28.123106 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.123084 5882 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 02:41:28.123106 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.123090 5882 policy_none.go:50] "Start" Oct 30 02:41:28.123106 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.123100 5882 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 02:41:28.123215 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.123115 5882 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 02:41:28.124354 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.124341 5882 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 02:41:28.124387 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.124355 5882 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 02:41:28.124387 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.124362 5882 policy_none.go:44] "Start" Oct 30 02:41:28.129190 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.129141 5882 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 02:41:28.131871 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.131857 5882 manager.go:341] "Starting Device Plugin manager" Oct 30 02:41:28.132448 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:28.131885 5882 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 02:41:28.132513 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.132451 5882 server.go:87] "Starting device plugin registration server" Oct 30 02:41:28.132751 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.132739 5882 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 02:41:28.132788 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.132752 5882 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 02:41:28.132873 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.132853 5882 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 02:41:28.132998 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.132962 5882 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 02:41:28.133067 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.133009 5882 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 02:41:28.133067 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.133016 5882 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 02:41:28.133153 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.133087 5882 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:41:28.132996672Z","UUID":"70d8e687-a49b-46ef-8e37-de05bc23555d","Handler":null,"Name":"","Endpoint":""} Oct 30 02:41:28.134504 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.134484 5882 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 02:41:28.134591 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.134513 5882 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 02:41:28.134879 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:28.134846 5882 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 02:41:28.233833 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.233799 5882 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:41:28.235675 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.235647 5882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:41:28.235748 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.235689 5882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:41:28.235748 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.235700 5882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:41:28.235748 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.235748 5882 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-200.ec2.internal" Oct 30 02:41:28.274224 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.274187 5882 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-200.ec2.internal" Oct 30 02:41:28.274380 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.274271 5882 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-200.ec2.internal" Oct 30 02:41:28.303612 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.303547 5882 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-200.ec2.internal"] Oct 30 02:41:28.304293 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.304277 5882 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" Oct 30 02:41:28.304359 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.304306 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-200.ec2.internal" event={"ID":"09b727cb32d7ac800f6bb9bd152bad1f","Type":"ContainerStarted","Data":"25357029410b453a1058c5e40df1945c3e08fed76b8509c54c929332a079da9a"} Oct 30 02:41:28.304359 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.304354 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-200.ec2.internal" event={"ID":"09b727cb32d7ac800f6bb9bd152bad1f","Type":"ContainerStarted","Data":"d55a5603b6cd1e4f6a135c34bc6ede909abe96f23baea1fcea1d56ec925d2ef5"} Oct 30 02:41:28.304490 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.304478 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" event={"ID":"6496276491b4aff226133cbf4e8c9798","Type":"ContainerStarted","Data":"48d489eb7b5c2cafc71f1bed529d66e769a95d60df16f4b548c1743f77c59de9"} Oct 30 02:41:28.304522 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.304502 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" event={"ID":"6496276491b4aff226133cbf4e8c9798","Type":"ContainerDied","Data":"4972329a555a16e7f846265b3e466e660d22139f947c63691d4259b3b469346c"} Oct 30 02:41:28.304522 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.304516 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" event={"ID":"6496276491b4aff226133cbf4e8c9798","Type":"ContainerStarted","Data":"4a23671b801854ce8d1c37d6fd1db927e2551c9dc7ae06b31bfb9ebbece18d70"} Oct 30 02:41:28.304661 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.304649 5882 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-200.ec2.internal" Oct 30 02:41:28.329872 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.329847 5882 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 02:41:28.329970 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:28.329892 5882 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-200.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-200.ec2.internal" Oct 30 02:41:28.330346 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.330333 5882 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 02:41:28.330392 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:28.330365 5882 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" Oct 30 02:41:28.420762 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.420687 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09b727cb32d7ac800f6bb9bd152bad1f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-200.ec2.internal\" (UID: \"09b727cb32d7ac800f6bb9bd152bad1f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-200.ec2.internal" Oct 30 02:41:28.420762 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.420721 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6496276491b4aff226133cbf4e8c9798-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal\" (UID: \"6496276491b4aff226133cbf4e8c9798\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" Oct 30 02:41:28.420762 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.420743 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6496276491b4aff226133cbf4e8c9798-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal\" (UID: \"6496276491b4aff226133cbf4e8c9798\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" Oct 30 02:41:28.521278 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.521244 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09b727cb32d7ac800f6bb9bd152bad1f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-200.ec2.internal\" (UID: \"09b727cb32d7ac800f6bb9bd152bad1f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-200.ec2.internal" Oct 30 02:41:28.521278 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.521264 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09b727cb32d7ac800f6bb9bd152bad1f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-200.ec2.internal\" (UID: \"09b727cb32d7ac800f6bb9bd152bad1f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-200.ec2.internal" Oct 30 02:41:28.521457 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.521306 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6496276491b4aff226133cbf4e8c9798-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal\" (UID: \"6496276491b4aff226133cbf4e8c9798\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" Oct 30 02:41:28.521457 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.521325 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6496276491b4aff226133cbf4e8c9798-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal\" (UID: \"6496276491b4aff226133cbf4e8c9798\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" Oct 30 02:41:28.521457 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.521353 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6496276491b4aff226133cbf4e8c9798-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal\" (UID: \"6496276491b4aff226133cbf4e8c9798\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" Oct 30 02:41:28.521457 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.521399 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6496276491b4aff226133cbf4e8c9798-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal\" (UID: \"6496276491b4aff226133cbf4e8c9798\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" Oct 30 02:41:28.954763 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:28.954727 5882 apiserver.go:52] "Watching apiserver" Oct 30 02:41:29.006496 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.006467 5882 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 02:41:29.007190 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.007165 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck","openshift-network-diagnostics/network-check-target-t765c","kube-system/global-pull-secret-syncer-cd6rj","openshift-dns/dns-default-2xmlx","openshift-network-operator/iptables-alerter-llk8x","openshift-dns/node-resolver-86gjv","openshift-monitoring/node-exporter-c5f6v","openshift-monitoring/metrics-server-dc789c958-vff7h","openshift-multus/multus-additional-cni-plugins-ttd2z","openshift-multus/network-metrics-daemon-xg5ng","openshift-image-registry/node-ca-s5lq6","openshift-monitoring/thanos-querier-67fdc77d96-6slgz","openshift-console/console-647f7b9c85-ldfjx","openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh","kube-system/kube-apiserver-proxy-ip-10-0-141-200.ec2.internal","openshift-ingress-canary/ingress-canary-q9hsv","openshift-monitoring/alertmanager-main-0","openshift-ovn-kubernetes/ovnkube-node-j9v4r","openshift-cluster-node-tuning-operator/tuned-v4kfc","openshift-monitoring/prometheus-k8s-0","openshift-insights/insights-runtime-extractor-tfqlk","openshift-monitoring/monitoring-plugin-77f4f76594-shbmn","kube-system/konnectivity-agent-xkkvm","openshift-multus/multus-d7vl2"] Oct 30 02:41:29.027179 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.027151 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 02:41:29.027179 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.027152 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 02:41:29.027350 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.027231 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 02:41:29.027442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.027420 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 02:41:29.027571 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.027517 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 02:41:29.027571 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.027559 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2qbtl\"" Oct 30 02:41:29.027692 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.027568 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dfk55\"" Oct 30 02:41:29.027692 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.027585 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7tzfh\"" Oct 30 02:41:29.027692 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.027646 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 02:41:29.027692 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.027649 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 02:41:29.072646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.072610 5882 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 02:41:29.077742 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.077719 5882 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" Oct 30 02:41:29.102520 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.102498 5882 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 02:41:29.102595 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.102541 5882 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-200.ec2.internal" Oct 30 02:41:29.122786 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.122761 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:29.122873 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.122792 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/913d7a7f-bba5-4c75-bd3d-1727790ca220-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:41:29.122873 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.122810 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-textfile\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.122873 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.122827 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm8sz\" (UniqueName: \"kubernetes.io/projected/5077c63e-73b2-4a23-95f0-51edd0875a47-kube-api-access-jm8sz\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:41:29.122873 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.122843 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:29.122873 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.122861 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1-host\") pod \"node-ca-s5lq6\" (UID: \"e754f0c3-6ac7-4e3e-ab0e-c343b673abc1\") " pod="openshift-image-registry/node-ca-s5lq6" Oct 30 02:41:29.123127 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.122886 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-tmp\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:29.123127 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.122913 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-multus-conf-dir\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.123127 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123008 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:29.123127 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123043 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-textfile\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.123127 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123052 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.123127 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123101 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.123304 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123130 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-tmp\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:29.123304 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123133 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-systemd-units\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.123304 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123177 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-var-lib-openvswitch\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.123304 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123210 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fd320963-1dae-4be2-9352-15a810104d69-secret-metrics-server-client-certs\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:29.123304 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123248 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fd320963-1dae-4be2-9352-15a810104d69-client-ca-bundle\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:29.123304 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123279 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-run-multus-certs\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.123304 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123297 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-run-systemd\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.123491 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123329 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f749cb9-f4b4-4db4-a580-2877108e718c-ovnkube-config\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.123491 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123362 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-sysconfig\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.123491 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123473 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-run\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.123604 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123515 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-metrics-client-ca\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:29.123604 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123550 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f749cb9-f4b4-4db4-a580-2877108e718c-ovnkube-config\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.123604 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123582 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-var-lib-kubelet\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.123784 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123675 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/913d7a7f-bba5-4c75-bd3d-1727790ca220-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:41:29.123784 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123725 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:29.123784 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123753 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.123916 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123902 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.123957 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123927 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:41:29.123957 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123943 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-run-ovn\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.124044 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.123963 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.124094 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124026 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fd320963-1dae-4be2-9352-15a810104d69-secret-metrics-server-tls\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:29.124094 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124064 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:29.124094 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124082 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:29.124222 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124097 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-run-netns\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.124222 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124135 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tccft\" (UniqueName: \"kubernetes.io/projected/7e93a8c5-35f7-459b-a8f1-f282130116cb-kube-api-access-tccft\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:29.124222 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.124166 5882 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 02:41:29.124222 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124178 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-sys-fs\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:41:29.124222 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.124182 5882 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 02:41:29.124442 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.124313 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:29.624286258 +0000 UTC m=+1.738601234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"console-serving-cert" not registered Oct 30 02:41:29.124442 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.124329 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:29.624322266 +0000 UTC m=+1.738637218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"service-ca" not registered Oct 30 02:41:29.124442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124375 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwcpz\" (UniqueName: \"kubernetes.io/projected/c42eb03a-a0d4-4509-8d16-ae82e1ac3c26-kube-api-access-mwcpz\") pod \"monitoring-plugin-77f4f76594-shbmn\" (UID: \"c42eb03a-a0d4-4509-8d16-ae82e1ac3c26\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-shbmn" Oct 30 02:41:29.124442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124385 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/107bc8a5-c01b-4b78-aa29-765aab336604-original-pull-secret\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:41:29.124442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124401 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs\") pod \"network-metrics-daemon-xg5ng\" (UID: \"5f79e66d-ae26-455c-b111-82f211ffe507\") " pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:41:29.124442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124420 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vwtj\" (UniqueName: \"kubernetes.io/projected/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-kube-api-access-4vwtj\") pod \"openshift-state-metrics-7bd7489957-z5vhl\" (UID: \"0c71f4fe-cc89-4856-accb-5e6c7c9452e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:29.124442 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124437 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.124753 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124461 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f749cb9-f4b4-4db4-a580-2877108e718c-ovnkube-script-lib\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.124753 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124485 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-tmp\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.124753 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124506 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-cnibin\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.124753 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124524 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89h27\" (UniqueName: \"kubernetes.io/projected/48721522-308a-4472-b687-d08caa72978d-kube-api-access-89h27\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.124753 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124549 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/992d8669-c914-486f-83ea-688dc69ca5a0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.124753 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124578 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.124753 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124583 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-tmp\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.124753 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124608 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.124753 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124654 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49f725c1-f41a-403d-b729-a170fc1fe687-tmp\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:29.124753 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124689 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:29.124753 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124717 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.124753 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124750 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49f725c1-f41a-403d-b729-a170fc1fe687-tmp\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:29.125123 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124766 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-slash\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.125123 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124785 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-node-log\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.125123 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.124753 5882 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 02:41:29.125123 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.124852 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:29.624837384 +0000 UTC m=+1.739152350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 02:41:29.125123 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124846 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-host\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.125123 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124902 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-registration-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:41:29.125123 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124929 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-var-lib-cni-bin\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.125123 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124972 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:29.125123 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.124998 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-tls\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.125123 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125016 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6vgd\" (UniqueName: \"kubernetes.io/projected/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-kube-api-access-p6vgd\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:29.125123 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125038 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48721522-308a-4472-b687-d08caa72978d-cni-binary-copy\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.125123 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.125053 5882 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 02:41:29.125123 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125063 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/913d7a7f-bba5-4c75-bd3d-1727790ca220-system-cni-dir\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:41:29.125123 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125094 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/913d7a7f-bba5-4c75-bd3d-1727790ca220-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:41:29.125123 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.125100 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:29.625086218 +0000 UTC m=+1.739401175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"console-config" not registered Oct 30 02:41:29.125650 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125161 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74x77\" (UniqueName: \"kubernetes.io/projected/913d7a7f-bba5-4c75-bd3d-1727790ca220-kube-api-access-74x77\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:41:29.125650 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125182 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.125650 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125198 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl9n4\" (UniqueName: \"kubernetes.io/projected/43ccba25-b0ea-47c8-b664-105feea298a3-kube-api-access-pl9n4\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.125650 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125294 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb7cc1ad-e964-4602-9ce5-c838212826d2-host-slash\") pod \"iptables-alerter-llk8x\" (UID: \"fb7cc1ad-e964-4602-9ce5-c838212826d2\") " pod="openshift-network-operator/iptables-alerter-llk8x" Oct 30 02:41:29.125650 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125325 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.125650 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125407 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-socket-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:41:29.125650 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125430 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:29.125650 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125449 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c42eb03a-a0d4-4509-8d16-ae82e1ac3c26-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-shbmn\" (UID: \"c42eb03a-a0d4-4509-8d16-ae82e1ac3c26\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-shbmn" Oct 30 02:41:29.125650 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125472 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fd320963-1dae-4be2-9352-15a810104d69-metrics-server-audit-profiles\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:29.125650 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125506 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/913d7a7f-bba5-4c75-bd3d-1727790ca220-cnibin\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:41:29.125650 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125577 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:29.125650 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125608 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/992d8669-c914-486f-83ea-688dc69ca5a0-config-out\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.125650 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.125641 5882 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.125677 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:29.625667702 +0000 UTC m=+1.739982655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"console-oauth-config" not registered Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125674 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/107bc8a5-c01b-4b78-aa29-765aab336604-kubelet-config\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125682 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/992d8669-c914-486f-83ea-688dc69ca5a0-config-out\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125711 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e93a8c5-35f7-459b-a8f1-f282130116cb-data-volume\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125731 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb-agent-certs\") pod \"konnectivity-agent-xkkvm\" (UID: \"6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb\") " pod="kube-system/konnectivity-agent-xkkvm" Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125748 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-z5vhl\" (UID: \"0c71f4fe-cc89-4856-accb-5e6c7c9452e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125765 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-z5vhl\" (UID: \"0c71f4fe-cc89-4856-accb-5e6c7c9452e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125772 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e93a8c5-35f7-459b-a8f1-f282130116cb-data-volume\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125789 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-run-ovn-kubernetes\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125805 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-systemd\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125930 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-sys\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125975 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-tmp-dir\") pod \"dns-default-2xmlx\" (UID: \"b2f64d21-ae78-4fdd-b2e5-83c7629e563a\") " pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.125993 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cecaabc-af83-4bf9-b3e5-d446b1302f12-sys\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126009 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-var-lib-kubelet\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126023 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7cecaabc-af83-4bf9-b3e5-d446b1302f12-root\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126040 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cecaabc-af83-4bf9-b3e5-d446b1302f12-metrics-client-ca\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126057 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b33f658-c455-4aa4-ab62-7b5e3de65be5-hosts-file\") pod \"node-resolver-86gjv\" (UID: \"8b33f658-c455-4aa4-ab62-7b5e3de65be5\") " pod="openshift-dns/node-resolver-86gjv" Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126033 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-tmp-dir\") pod \"dns-default-2xmlx\" (UID: \"b2f64d21-ae78-4fdd-b2e5-83c7629e563a\") " pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126101 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b33f658-c455-4aa4-ab62-7b5e3de65be5-tmp-dir\") pod \"node-resolver-86gjv\" (UID: \"8b33f658-c455-4aa4-ab62-7b5e3de65be5\") " pod="openshift-dns/node-resolver-86gjv" Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126123 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:29.126159 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126169 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b33f658-c455-4aa4-ab62-7b5e3de65be5-tmp-dir\") pod \"node-resolver-86gjv\" (UID: \"8b33f658-c455-4aa4-ab62-7b5e3de65be5\") " pod="openshift-dns/node-resolver-86gjv" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126207 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd320963-1dae-4be2-9352-15a810104d69-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126233 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fd320963-1dae-4be2-9352-15a810104d69-audit-log\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126302 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stmvr\" (UniqueName: \"kubernetes.io/projected/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1-kube-api-access-stmvr\") pod \"node-ca-s5lq6\" (UID: \"e754f0c3-6ac7-4e3e-ab0e-c343b673abc1\") " pod="openshift-image-registry/node-ca-s5lq6" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126308 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fd320963-1dae-4be2-9352-15a810104d69-audit-log\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126332 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb-konnectivity-ca\") pod \"konnectivity-agent-xkkvm\" (UID: \"6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb\") " pod="kube-system/konnectivity-agent-xkkvm" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126363 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77cp8\" (UniqueName: \"kubernetes.io/projected/be9a9474-03c1-489e-ab01-0e8e83d79daf-kube-api-access-77cp8\") pod \"ingress-canary-q9hsv\" (UID: \"be9a9474-03c1-489e-ab01-0e8e83d79daf\") " pod="openshift-ingress-canary/ingress-canary-q9hsv" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126434 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-etc-kubernetes\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126455 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/913d7a7f-bba5-4c75-bd3d-1727790ca220-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126495 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1-serviceca\") pod \"node-ca-s5lq6\" (UID: \"e754f0c3-6ac7-4e3e-ab0e-c343b673abc1\") " pod="openshift-image-registry/node-ca-s5lq6" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126533 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43ccba25-b0ea-47c8-b664-105feea298a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126553 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43ccba25-b0ea-47c8-b664-105feea298a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126600 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126642 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f749cb9-f4b4-4db4-a580-2877108e718c-ovn-node-metrics-cert\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126673 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8x7w\" (UniqueName: \"kubernetes.io/projected/fb7cc1ad-e964-4602-9ce5-c838212826d2-kube-api-access-g8x7w\") pod \"iptables-alerter-llk8x\" (UID: \"fb7cc1ad-e964-4602-9ce5-c838212826d2\") " pod="openshift-network-operator/iptables-alerter-llk8x" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126710 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43ccba25-b0ea-47c8-b664-105feea298a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126726 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-sysctl-d\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126756 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e93a8c5-35f7-459b-a8f1-f282130116cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126785 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-grpc-tls\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126875 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-php67\" (UniqueName: \"kubernetes.io/projected/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-api-access-php67\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126939 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/992d8669-c914-486f-83ea-688dc69ca5a0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.126964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126960 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-cni-bin\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.127844 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.126981 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-cni-netd\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.127844 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127068 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:41:29.127844 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127145 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.127844 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127166 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1-serviceca\") pod \"node-ca-s5lq6\" (UID: \"e754f0c3-6ac7-4e3e-ab0e-c343b673abc1\") " pod="openshift-image-registry/node-ca-s5lq6" Oct 30 02:41:29.127844 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127204 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.127844 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127245 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/992d8669-c914-486f-83ea-688dc69ca5a0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.127844 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127278 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-run-openvswitch\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.127844 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127305 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-modprobe-d\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.127844 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127322 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/992d8669-c914-486f-83ea-688dc69ca5a0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.127844 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127329 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e93a8c5-35f7-459b-a8f1-f282130116cb-crio-socket\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:29.127844 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127388 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-wtmp\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.127844 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127409 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-multus-socket-dir-parent\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.127844 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127427 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-z5vhl\" (UID: \"0c71f4fe-cc89-4856-accb-5e6c7c9452e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:29.127844 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127480 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.127844 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127544 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.127844 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127638 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f749cb9-f4b4-4db4-a580-2877108e718c-env-overrides\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.127844 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127660 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stdbm\" (UniqueName: \"kubernetes.io/projected/7f749cb9-f4b4-4db4-a580-2877108e718c-kube-api-access-stdbm\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.127844 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127684 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-hostroot\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.127844 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127798 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/48721522-308a-4472-b687-d08caa72978d-multus-daemon-config\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.127844 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127840 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq248\" (UniqueName: \"kubernetes.io/projected/5f79e66d-ae26-455c-b111-82f211ffe507-kube-api-access-xq248\") pod \"network-metrics-daemon-xg5ng\" (UID: \"5f79e66d-ae26-455c-b111-82f211ffe507\") " pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127856 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/992d8669-c914-486f-83ea-688dc69ca5a0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127872 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127901 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-etc-openvswitch\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127921 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-tls\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127938 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2vg\" (UniqueName: \"kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.127980 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128048 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128073 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-log-socket\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128089 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-kubernetes\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128112 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-sysctl-conf\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128156 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-etc-selinux\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128162 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128217 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128245 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-accelerators-collector-config\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128263 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-multus-cni-dir\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128282 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kndrd\" (UniqueName: \"kubernetes.io/projected/992d8669-c914-486f-83ea-688dc69ca5a0-kube-api-access-kndrd\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128297 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-config\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128319 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128361 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-kubelet\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128378 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bflr\" (UniqueName: \"kubernetes.io/projected/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-kube-api-access-6bflr\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128399 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tvzr\" (UniqueName: \"kubernetes.io/projected/7cecaabc-af83-4bf9-b3e5-d446b1302f12-kube-api-access-9tvzr\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128441 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-run-k8s-cni-cncf-io\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128497 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-run-netns\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128538 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.128606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128583 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb7cc1ad-e964-4602-9ce5-c838212826d2-iptables-alerter-script\") pod \"iptables-alerter-llk8x\" (UID: \"fb7cc1ad-e964-4602-9ce5-c838212826d2\") " pod="openshift-network-operator/iptables-alerter-llk8x" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128686 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128713 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-tuned\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128734 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-config-volume\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128750 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-web-config\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128767 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128793 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e93a8c5-35f7-459b-a8f1-f282130116cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128825 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-device-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128844 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-tuned\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128853 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128897 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/107bc8a5-c01b-4b78-aa29-765aab336604-dbus\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128932 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be9a9474-03c1-489e-ab01-0e8e83d79daf-cert\") pod \"ingress-canary-q9hsv\" (UID: \"be9a9474-03c1-489e-ab01-0e8e83d79daf\") " pod="openshift-ingress-canary/ingress-canary-q9hsv" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.128989 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-metrics-tls\") pod \"dns-default-2xmlx\" (UID: \"b2f64d21-ae78-4fdd-b2e5-83c7629e563a\") " pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.129019 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqcx7\" (UniqueName: \"kubernetes.io/projected/fd320963-1dae-4be2-9352-15a810104d69-kube-api-access-fqcx7\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.129038 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-system-cni-dir\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.129059 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.129171 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d98p\" (UniqueName: \"kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p\") pod \"network-check-target-t765c\" (UID: \"fb1a01bf-dc4f-430b-8b6d-d5569f85c981\") " pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.129210 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.129240 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-lib-modules\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.129267 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-var-lib-cni-multus\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.129299 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.129315 5882 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.129337 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-config-volume\") pod \"dns-default-2xmlx\" (UID: \"b2f64d21-ae78-4fdd-b2e5-83c7629e563a\") " pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.129390 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84khn\" (UniqueName: \"kubernetes.io/projected/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-kube-api-access-84khn\") pod \"dns-default-2xmlx\" (UID: \"b2f64d21-ae78-4fdd-b2e5-83c7629e563a\") " pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.129400 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:29.629386401 +0000 UTC m=+1.743701374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 02:41:29.129523 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.129454 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8bsm\" (UniqueName: \"kubernetes.io/projected/8b33f658-c455-4aa4-ab62-7b5e3de65be5-kube-api-access-v8bsm\") pod \"node-resolver-86gjv\" (UID: \"8b33f658-c455-4aa4-ab62-7b5e3de65be5\") " pod="openshift-dns/node-resolver-86gjv" Oct 30 02:41:29.132141 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.132027 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 02:41:29.132141 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.132114 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 02:41:29.132274 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.132170 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 02:41:29.132274 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.132268 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 02:41:29.132377 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.132325 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 02:41:29.132444 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.132429 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 02:41:29.132444 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.132433 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 02:41:29.132509 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.132492 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.133036 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.133295 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.133354 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.134051 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.134672 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.134702 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.134837 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.134879 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.135052 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.135111 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.135238 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.135524 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.135628 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.135686 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.135735 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.135742 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.136039 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.136049 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.136184 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.136300 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.136322 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.136331 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fd320963-1dae-4be2-9352-15a810104d69-metrics-server-audit-profiles\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.136055 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.136472 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.136539 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.136600 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.136187 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.136691 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb-konnectivity-ca\") pod \"konnectivity-agent-xkkvm\" (UID: \"6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb\") " pod="kube-system/konnectivity-agent-xkkvm" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.136798 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.136540 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.136830 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43ccba25-b0ea-47c8-b664-105feea298a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.136845 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.136910 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.136978 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.137059 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.137100 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.137123 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.137155 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.137154 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e93a8c5-35f7-459b-a8f1-f282130116cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.137224 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.137358 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd320963-1dae-4be2-9352-15a810104d69-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.137414 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.137416 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.137427 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.137489 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.137417 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.139355 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.137563 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.139355 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.137783 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.139355 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.137979 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f749cb9-f4b4-4db4-a580-2877108e718c-env-overrides\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.139355 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.138169 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/48721522-308a-4472-b687-d08caa72978d-multus-daemon-config\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.139355 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.138338 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.139355 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.138476 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/992d8669-c914-486f-83ea-688dc69ca5a0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.139355 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.139318 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-web-config\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.139728 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.139501 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:29.139728 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.139707 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-config-volume\") pod \"dns-default-2xmlx\" (UID: \"b2f64d21-ae78-4fdd-b2e5-83c7629e563a\") " pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:29.143469 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.143443 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:29.143469 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.143456 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.143629 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.143450 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/913d7a7f-bba5-4c75-bd3d-1727790ca220-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:41:29.143629 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.143475 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fd320963-1dae-4be2-9352-15a810104d69-secret-metrics-server-client-certs\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:29.145200 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.145174 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.145386 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.145367 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 02:41:29.145444 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.145378 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 02:41:29.145492 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.145470 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 02:41:29.145552 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.145534 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.145606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.145588 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pjcnc\"" Oct 30 02:41:29.145751 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.145734 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 02:41:29.145820 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.145761 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:29.145820 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.145768 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-19g27kfjqkpbt\"" Oct 30 02:41:29.145902 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.145840 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cb5f2\"" Oct 30 02:41:29.145946 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.145926 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 02:41:29.146014 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146000 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-q4z4m\"" Oct 30 02:41:29.146116 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146100 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 02:41:29.146175 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146139 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 02:41:29.146175 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146146 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 02:41:29.146175 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146146 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cjkxs\"" Oct 30 02:41:29.146175 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146168 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-96il72fi4oust\"" Oct 30 02:41:29.146175 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146106 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 02:41:29.146364 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146283 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 02:41:29.146364 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146312 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2trkp\"" Oct 30 02:41:29.146364 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146334 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nd26x\"" Oct 30 02:41:29.146364 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146338 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:41:29.146364 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146354 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 02:41:29.146364 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146358 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-227fv\"" Oct 30 02:41:29.146643 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146402 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 02:41:29.146643 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146410 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8td9f\"" Oct 30 02:41:29.146643 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146404 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-74jzq\"" Oct 30 02:41:29.146643 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146487 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 02:41:29.146643 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146495 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 02:41:29.146643 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146505 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nmhn4\"" Oct 30 02:41:29.146643 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146510 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p7m4t\"" Oct 30 02:41:29.146643 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146519 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 02:41:29.146643 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146515 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:29.146643 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146598 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mmm8g\"" Oct 30 02:41:29.146643 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146593 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 02:41:29.146643 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146602 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 02:41:29.147151 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146668 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ksjrs\"" Oct 30 02:41:29.147151 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146669 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xkr2x\"" Oct 30 02:41:29.147151 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146691 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bdpn101jv6nnb\"" Oct 30 02:41:29.147151 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146698 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 02:41:29.147151 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146760 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 02:41:29.147151 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146785 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 02:41:29.147151 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146799 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lz8p4\"" Oct 30 02:41:29.147151 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146811 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 02:41:29.147151 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146816 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jvpkf\"" Oct 30 02:41:29.147151 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146852 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 02:41:29.147151 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146930 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9rd8b\"" Oct 30 02:41:29.147151 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.146947 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 02:41:29.147151 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.147029 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f749cb9-f4b4-4db4-a580-2877108e718c-ovn-node-metrics-cert\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.147151 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.147109 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q9zqs\"" Oct 30 02:41:29.147727 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.147179 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 02:41:29.147727 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.147189 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-grpc-tls\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:29.147727 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.147251 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8cdxd\"" Oct 30 02:41:29.147727 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.147309 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 02:41:29.147727 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.147387 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 02:41:29.147727 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.147411 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 02:41:29.147727 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.147518 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 02:41:29.147727 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.147576 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.147727 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.147639 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 02:41:29.147727 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.147642 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rw9z8\"" Oct 30 02:41:29.147727 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.147691 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 02:41:29.148042 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.147751 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-z5vhl\" (UID: \"0c71f4fe-cc89-4856-accb-5e6c7c9452e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:29.148042 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.147784 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.148498 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.148476 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-secret-thanos-querier-tls\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:29.148589 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.148565 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:29.148715 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.148692 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-accelerators-collector-config\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.148818 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.148802 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.148902 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.148881 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:29.148969 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.148952 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-config\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.149078 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.149056 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e93a8c5-35f7-459b-a8f1-f282130116cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:29.149221 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.149121 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.149300 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.149229 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be9a9474-03c1-489e-ab01-0e8e83d79daf-cert\") pod \"ingress-canary-q9hsv\" (UID: \"be9a9474-03c1-489e-ab01-0e8e83d79daf\") " pod="openshift-ingress-canary/ingress-canary-q9hsv" Oct 30 02:41:29.149300 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.149275 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb7cc1ad-e964-4602-9ce5-c838212826d2-iptables-alerter-script\") pod \"iptables-alerter-llk8x\" (UID: \"fb7cc1ad-e964-4602-9ce5-c838212826d2\") " pod="openshift-network-operator/iptables-alerter-llk8x" Oct 30 02:41:29.149420 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.149359 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-config-volume\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.149470 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.149423 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-metrics-tls\") pod \"dns-default-2xmlx\" (UID: \"b2f64d21-ae78-4fdd-b2e5-83c7629e563a\") " pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:29.149573 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.149548 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:29.149573 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.149548 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.151961 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.151941 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 02:41:29.154007 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.153754 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-metrics-client-ca\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:29.154007 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.153768 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fd320963-1dae-4be2-9352-15a810104d69-client-ca-bundle\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.154861 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.154964 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.154973 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fd320963-1dae-4be2-9352-15a810104d69-secret-metrics-server-tls\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.155003 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/992d8669-c914-486f-83ea-688dc69ca5a0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.155008 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f749cb9-f4b4-4db4-a580-2877108e718c-ovnkube-script-lib\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.154979 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f79e66d-ae26-455c-b111-82f211ffe507-metrics-certs\") pod \"network-metrics-daemon-xg5ng\" (UID: \"5f79e66d-ae26-455c-b111-82f211ffe507\") " pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.155686 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48721522-308a-4472-b687-d08caa72978d-cni-binary-copy\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.155721 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.155774 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/913d7a7f-bba5-4c75-bd3d-1727790ca220-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.155853 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-tls\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.155968 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c42eb03a-a0d4-4509-8d16-ae82e1ac3c26-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-shbmn\" (UID: \"c42eb03a-a0d4-4509-8d16-ae82e1ac3c26\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-shbmn" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.156098 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-z5vhl\" (UID: \"0c71f4fe-cc89-4856-accb-5e6c7c9452e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.156133 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb-agent-certs\") pod \"konnectivity-agent-xkkvm\" (UID: \"6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb\") " pod="kube-system/konnectivity-agent-xkkvm" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.156210 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-z5vhl\" (UID: \"0c71f4fe-cc89-4856-accb-5e6c7c9452e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.156337 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cecaabc-af83-4bf9-b3e5-d446b1302f12-metrics-client-ca\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.156703 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/913d7a7f-bba5-4c75-bd3d-1727790ca220-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.157252 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/992d8669-c914-486f-83ea-688dc69ca5a0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.157285 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.157390 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.158851 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm8sz\" (UniqueName: \"kubernetes.io/projected/5077c63e-73b2-4a23-95f0-51edd0875a47-kube-api-access-jm8sz\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.159012 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.159043 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vwtj\" (UniqueName: \"kubernetes.io/projected/0c71f4fe-cc89-4856-accb-5e6c7c9452e2-kube-api-access-4vwtj\") pod \"openshift-state-metrics-7bd7489957-z5vhl\" (UID: \"0c71f4fe-cc89-4856-accb-5e6c7c9452e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z5vhl" Oct 30 02:41:29.159646 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.159508 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.163530 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.163510 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tccft\" (UniqueName: \"kubernetes.io/projected/7e93a8c5-35f7-459b-a8f1-f282130116cb-kube-api-access-tccft\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:29.163828 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.163807 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.164292 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.164277 5882 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 02:41:29.164348 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.164294 5882 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 02:41:29.164348 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.164304 5882 projected.go:196] Error preparing data for projected volume kube-api-access-nl2vg for pod openshift-console/console-647f7b9c85-ldfjx: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 02:41:29.164445 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.164367 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:29.664353825 +0000 UTC m=+1.778668791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nl2vg" (UniqueName: "kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 02:41:29.164561 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.164539 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89h27\" (UniqueName: \"kubernetes.io/projected/48721522-308a-4472-b687-d08caa72978d-kube-api-access-89h27\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.164743 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.164677 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tvzr\" (UniqueName: \"kubernetes.io/projected/7cecaabc-af83-4bf9-b3e5-d446b1302f12-kube-api-access-9tvzr\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.165992 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.165971 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8bsm\" (UniqueName: \"kubernetes.io/projected/8b33f658-c455-4aa4-ab62-7b5e3de65be5-kube-api-access-v8bsm\") pod \"node-resolver-86gjv\" (UID: \"8b33f658-c455-4aa4-ab62-7b5e3de65be5\") " pod="openshift-dns/node-resolver-86gjv" Oct 30 02:41:29.166362 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.166341 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74x77\" (UniqueName: \"kubernetes.io/projected/913d7a7f-bba5-4c75-bd3d-1727790ca220-kube-api-access-74x77\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:41:29.166545 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.166525 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d98p\" (UniqueName: \"kubernetes.io/projected/fb1a01bf-dc4f-430b-8b6d-d5569f85c981-kube-api-access-7d98p\") pod \"network-check-target-t765c\" (UID: \"fb1a01bf-dc4f-430b-8b6d-d5569f85c981\") " pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:41:29.166840 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.166821 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bflr\" (UniqueName: \"kubernetes.io/projected/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-kube-api-access-6bflr\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.166966 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.166923 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6vgd\" (UniqueName: \"kubernetes.io/projected/2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0-kube-api-access-p6vgd\") pod \"thanos-querier-67fdc77d96-6slgz\" (UID: \"2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0\") " pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:29.169401 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.169382 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqcx7\" (UniqueName: \"kubernetes.io/projected/fd320963-1dae-4be2-9352-15a810104d69-kube-api-access-fqcx7\") pod \"metrics-server-dc789c958-vff7h\" (UID: \"fd320963-1dae-4be2-9352-15a810104d69\") " pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:29.170132 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.170112 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kndrd\" (UniqueName: \"kubernetes.io/projected/992d8669-c914-486f-83ea-688dc69ca5a0-kube-api-access-kndrd\") pod \"alertmanager-main-0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:29.170720 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.170701 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stmvr\" (UniqueName: \"kubernetes.io/projected/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1-kube-api-access-stmvr\") pod \"node-ca-s5lq6\" (UID: \"e754f0c3-6ac7-4e3e-ab0e-c343b673abc1\") " pod="openshift-image-registry/node-ca-s5lq6" Oct 30 02:41:29.171024 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.171009 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8x7w\" (UniqueName: \"kubernetes.io/projected/fb7cc1ad-e964-4602-9ce5-c838212826d2-kube-api-access-g8x7w\") pod \"iptables-alerter-llk8x\" (UID: \"fb7cc1ad-e964-4602-9ce5-c838212826d2\") " pod="openshift-network-operator/iptables-alerter-llk8x" Oct 30 02:41:29.174392 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.174371 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq248\" (UniqueName: \"kubernetes.io/projected/5f79e66d-ae26-455c-b111-82f211ffe507-kube-api-access-xq248\") pod \"network-metrics-daemon-xg5ng\" (UID: \"5f79e66d-ae26-455c-b111-82f211ffe507\") " pod="openshift-multus/network-metrics-daemon-xg5ng" Oct 30 02:41:29.174985 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.174964 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84khn\" (UniqueName: \"kubernetes.io/projected/b2f64d21-ae78-4fdd-b2e5-83c7629e563a-kube-api-access-84khn\") pod \"dns-default-2xmlx\" (UID: \"b2f64d21-ae78-4fdd-b2e5-83c7629e563a\") " pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:29.175110 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.175091 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stdbm\" (UniqueName: \"kubernetes.io/projected/7f749cb9-f4b4-4db4-a580-2877108e718c-kube-api-access-stdbm\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.175152 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.175139 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77cp8\" (UniqueName: \"kubernetes.io/projected/be9a9474-03c1-489e-ab01-0e8e83d79daf-kube-api-access-77cp8\") pod \"ingress-canary-q9hsv\" (UID: \"be9a9474-03c1-489e-ab01-0e8e83d79daf\") " pod="openshift-ingress-canary/ingress-canary-q9hsv" Oct 30 02:41:29.175339 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.175318 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-php67\" (UniqueName: \"kubernetes.io/projected/e06d9d5b-be5f-46aa-a633-8ce9c9c37589-kube-api-access-php67\") pod \"kube-state-metrics-5dc8d7d89b-s6gck\" (UID: \"e06d9d5b-be5f-46aa-a633-8ce9c9c37589\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s6gck" Oct 30 02:41:29.175453 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.175439 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwcpz\" (UniqueName: \"kubernetes.io/projected/c42eb03a-a0d4-4509-8d16-ae82e1ac3c26-kube-api-access-mwcpz\") pod \"monitoring-plugin-77f4f76594-shbmn\" (UID: \"c42eb03a-a0d4-4509-8d16-ae82e1ac3c26\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-shbmn" Oct 30 02:41:29.175930 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.175914 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl9n4\" (UniqueName: \"kubernetes.io/projected/43ccba25-b0ea-47c8-b664-105feea298a3-kube-api-access-pl9n4\") pod \"prometheus-k8s-0\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:29.230035 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.229946 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-sys-fs\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:41:29.230035 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.229989 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-cnibin\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.230035 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230009 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.230035 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230036 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-slash\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230056 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-node-log\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230071 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-host\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230068 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-sys-fs\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230085 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-registration-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230089 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-cnibin\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230101 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-var-lib-cni-bin\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230135 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230148 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/913d7a7f-bba5-4c75-bd3d-1727790ca220-system-cni-dir\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230136 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-var-lib-cni-bin\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230135 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-slash\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230180 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb7cc1ad-e964-4602-9ce5-c838212826d2-host-slash\") pod \"iptables-alerter-llk8x\" (UID: \"fb7cc1ad-e964-4602-9ce5-c838212826d2\") " pod="openshift-network-operator/iptables-alerter-llk8x" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230182 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-node-log\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230173 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-host\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230201 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-registration-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230218 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb7cc1ad-e964-4602-9ce5-c838212826d2-host-slash\") pod \"iptables-alerter-llk8x\" (UID: \"fb7cc1ad-e964-4602-9ce5-c838212826d2\") " pod="openshift-network-operator/iptables-alerter-llk8x" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230224 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-socket-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230180 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/913d7a7f-bba5-4c75-bd3d-1727790ca220-system-cni-dir\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230257 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/913d7a7f-bba5-4c75-bd3d-1727790ca220-cnibin\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230274 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-socket-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230282 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/913d7a7f-bba5-4c75-bd3d-1727790ca220-cnibin\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230309 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/107bc8a5-c01b-4b78-aa29-765aab336604-kubelet-config\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:41:29.230343 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230339 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-run-ovn-kubernetes\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230362 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-systemd\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230386 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-sys\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230402 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-run-ovn-kubernetes\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230401 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/107bc8a5-c01b-4b78-aa29-765aab336604-kubelet-config\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230410 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cecaabc-af83-4bf9-b3e5-d446b1302f12-sys\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230442 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-var-lib-kubelet\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230446 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-systemd\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230458 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-sys\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230466 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7cecaabc-af83-4bf9-b3e5-d446b1302f12-root\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230494 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b33f658-c455-4aa4-ab62-7b5e3de65be5-hosts-file\") pod \"node-resolver-86gjv\" (UID: \"8b33f658-c455-4aa4-ab62-7b5e3de65be5\") " pod="openshift-dns/node-resolver-86gjv" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230496 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cecaabc-af83-4bf9-b3e5-d446b1302f12-sys\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230507 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-var-lib-kubelet\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230515 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7cecaabc-af83-4bf9-b3e5-d446b1302f12-root\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230536 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-etc-kubernetes\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230554 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b33f658-c455-4aa4-ab62-7b5e3de65be5-hosts-file\") pod \"node-resolver-86gjv\" (UID: \"8b33f658-c455-4aa4-ab62-7b5e3de65be5\") " pod="openshift-dns/node-resolver-86gjv" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230558 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-sysctl-d\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230595 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-cni-bin\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230632 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-etc-kubernetes\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230643 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-cni-netd\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230660 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-cni-bin\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230670 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230676 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-sysctl-d\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230673 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-cni-netd\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230705 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230698 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-run-openvswitch\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230738 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-modprobe-d\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230744 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-run-openvswitch\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230754 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e93a8c5-35f7-459b-a8f1-f282130116cb-crio-socket\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230771 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-wtmp\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230786 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-multus-socket-dir-parent\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230804 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-hostroot\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230804 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e93a8c5-35f7-459b-a8f1-f282130116cb-crio-socket\") pod \"insights-runtime-extractor-tfqlk\" (UID: \"7e93a8c5-35f7-459b-a8f1-f282130116cb\") " pod="openshift-insights/insights-runtime-extractor-tfqlk" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230822 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-etc-openvswitch\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230841 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-modprobe-d\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230846 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7cecaabc-af83-4bf9-b3e5-d446b1302f12-node-exporter-wtmp\") pod \"node-exporter-c5f6v\" (UID: \"7cecaabc-af83-4bf9-b3e5-d446b1302f12\") " pod="openshift-monitoring/node-exporter-c5f6v" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230854 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-hostroot\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230862 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-multus-socket-dir-parent\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230866 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-etc-openvswitch\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230880 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-log-socket\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230892 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-log-socket\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230907 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-kubernetes\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230930 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-sysctl-conf\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230944 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-etc-selinux\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230960 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-multus-cni-dir\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230977 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-kubelet\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230981 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-kubernetes\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.230995 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-run-k8s-cni-cncf-io\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231009 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-run-netns\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231021 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-sysctl-conf\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231022 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-etc-selinux\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231038 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-device-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231044 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-multus-cni-dir\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231057 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/107bc8a5-c01b-4b78-aa29-765aab336604-dbus\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231057 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-kubelet\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231061 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-run-k8s-cni-cncf-io\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231066 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-run-netns\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231080 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5077c63e-73b2-4a23-95f0-51edd0875a47-device-dir\") pod \"aws-ebs-csi-driver-node-fpgzh\" (UID: \"5077c63e-73b2-4a23-95f0-51edd0875a47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpgzh" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231089 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-system-cni-dir\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231100 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/107bc8a5-c01b-4b78-aa29-765aab336604-dbus\") pod \"global-pull-secret-syncer-cd6rj\" (UID: \"107bc8a5-c01b-4b78-aa29-765aab336604\") " pod="kube-system/global-pull-secret-syncer-cd6rj" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231131 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-lib-modules\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231155 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-var-lib-cni-multus\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231181 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-system-cni-dir\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231188 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1-host\") pod \"node-ca-s5lq6\" (UID: \"e754f0c3-6ac7-4e3e-ab0e-c343b673abc1\") " pod="openshift-image-registry/node-ca-s5lq6" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231183 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-var-lib-cni-multus\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231216 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e754f0c3-6ac7-4e3e-ab0e-c343b673abc1-host\") pod \"node-ca-s5lq6\" (UID: \"e754f0c3-6ac7-4e3e-ab0e-c343b673abc1\") " pod="openshift-image-registry/node-ca-s5lq6" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231216 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-multus-conf-dir\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231224 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-lib-modules\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231253 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-systemd-units\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231256 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-multus-conf-dir\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231283 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-var-lib-openvswitch\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231279 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-systemd-units\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231315 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-run-multus-certs\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231330 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-var-lib-openvswitch\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231339 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-run-systemd\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231359 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-run-multus-certs\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231362 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-sysconfig\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231384 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-run-systemd\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.231334 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231385 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-run\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.233450 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231411 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-var-lib-kubelet\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.233450 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231425 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-run\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.233450 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231427 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/913d7a7f-bba5-4c75-bd3d-1727790ca220-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:41:29.233450 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231458 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/913d7a7f-bba5-4c75-bd3d-1727790ca220-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttd2z\" (UID: \"913d7a7f-bba5-4c75-bd3d-1727790ca220\") " pod="openshift-multus/multus-additional-cni-plugins-ttd2z" Oct 30 02:41:29.233450 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231464 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-run-ovn\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.233450 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231474 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48721522-308a-4472-b687-d08caa72978d-host-var-lib-kubelet\") pod \"multus-d7vl2\" (UID: \"48721522-308a-4472-b687-d08caa72978d\") " pod="openshift-multus/multus-d7vl2" Oct 30 02:41:29.233450 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231514 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-run-netns\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.233450 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231477 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d2896c8-e75c-4565-b5fe-3320f26fdd8a-etc-sysconfig\") pod \"tuned-v4kfc\" (UID: \"9d2896c8-e75c-4565-b5fe-3320f26fdd8a\") " pod="openshift-cluster-node-tuning-operator/tuned-v4kfc" Oct 30 02:41:29.233450 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231511 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-run-ovn\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.233450 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.231572 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f749cb9-f4b4-4db4-a580-2877108e718c-host-run-netns\") pod \"ovnkube-node-j9v4r\" (UID: \"7f749cb9-f4b4-4db4-a580-2877108e718c\") " pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:29.520427 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.520393 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-67fdc77d96-6slgz" Oct 30 02:41:29.636366 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.636330 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:29.636366 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.636368 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:29.636566 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.636392 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:29.636566 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.636443 5882 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 02:41:29.636566 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.636458 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:29.636566 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.636499 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:29.636566 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.636501 5882 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 02:41:29.636566 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.636509 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:30.636496207 +0000 UTC m=+2.750811163 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 02:41:29.636566 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.636524 5882 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 02:41:29.636566 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.636536 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:29.636566 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.636563 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:30.636541275 +0000 UTC m=+2.750856245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"console-oauth-config" not registered Oct 30 02:41:29.636566 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.636448 5882 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 02:41:29.636930 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.636642 5882 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 02:41:29.636930 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.636647 5882 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 02:41:29.636930 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.636650 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:30.63663839 +0000 UTC m=+2.750953385 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 02:41:29.636930 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.636711 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:30.636696177 +0000 UTC m=+2.751011131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"console-config" not registered Oct 30 02:41:29.636930 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.636729 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:30.636719077 +0000 UTC m=+2.751034040 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"service-ca" not registered Oct 30 02:41:29.636930 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.636752 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:30.636741881 +0000 UTC m=+2.751056833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"console-serving-cert" not registered Oct 30 02:41:29.737263 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:29.737238 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2vg\" (UniqueName: \"kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:29.737406 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.737350 5882 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 02:41:29.737406 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.737364 5882 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 02:41:29.737406 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.737373 5882 projected.go:196] Error preparing data for projected volume kube-api-access-nl2vg for pod openshift-console/console-647f7b9c85-ldfjx: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 02:41:29.737521 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:29.737411 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:30.737400802 +0000 UTC m=+2.851715768 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nl2vg" (UniqueName: "kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 02:41:30.646849 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:30.646808 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:30.647355 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:30.646911 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:30.647355 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:30.646947 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:30.647355 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:30.646959 5882 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 02:41:30.647355 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:30.647016 5882 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 02:41:30.647355 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:30.647023 5882 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 02:41:30.647355 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:30.647038 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:32.647018607 +0000 UTC m=+4.761333577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"console-oauth-config" not registered Oct 30 02:41:30.647355 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:30.646969 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:30.647355 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:30.647073 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:32.647061238 +0000 UTC m=+4.761376214 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"service-ca" not registered Oct 30 02:41:30.647355 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:30.647079 5882 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 02:41:30.647355 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:30.647092 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:32.647083122 +0000 UTC m=+4.761398098 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 02:41:30.647355 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:30.647122 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:30.647355 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:30.647145 5882 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 02:41:30.647355 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:30.647144 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:30.647355 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:30.647167 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:32.647153218 +0000 UTC m=+4.761468177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"console-serving-cert" not registered Oct 30 02:41:30.647355 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:30.647173 5882 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 02:41:30.647355 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:30.647189 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:32.647180305 +0000 UTC m=+4.761495263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 02:41:30.647355 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:30.647206 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:32.647197929 +0000 UTC m=+4.761512882 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"console-config" not registered Oct 30 02:41:30.748245 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:30.748207 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2vg\" (UniqueName: \"kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:30.748412 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:30.748355 5882 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 02:41:30.748412 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:30.748374 5882 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 02:41:30.748412 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:30.748385 5882 projected.go:196] Error preparing data for projected volume kube-api-access-nl2vg for pod openshift-console/console-647f7b9c85-ldfjx: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 02:41:30.748519 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:30.748429 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:32.748416731 +0000 UTC m=+4.862731684 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nl2vg" (UniqueName: "kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 02:41:31.186253 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:31.186221 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2xmlx" Oct 30 02:41:31.349710 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:31.349682 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:31.566753 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:31.566725 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:31.779229 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:31.779192 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-shbmn" Oct 30 02:41:32.088259 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:32.088229 5882 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 02:41:32.668180 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:32.668143 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:32.668353 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:32.668200 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:32.668353 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:32.668221 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:32.668353 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:32.668263 5882 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 02:41:32.668353 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:32.668315 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:32.668353 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:32.668344 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:36.668326482 +0000 UTC m=+8.782641449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 02:41:32.668353 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:32.668353 5882 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 02:41:32.668561 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:32.668367 5882 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 02:41:32.668561 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:32.668387 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:32.668561 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:32.668393 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:36.668381542 +0000 UTC m=+8.782696508 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"console-serving-cert" not registered Oct 30 02:41:32.668561 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:32.668275 5882 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 02:41:32.668561 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:32.668416 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:36.668404711 +0000 UTC m=+8.782719682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 02:41:32.668561 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:32.668434 5882 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 02:41:32.668561 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:32.668442 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:36.668431799 +0000 UTC m=+8.782746756 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"service-ca" not registered Oct 30 02:41:32.668561 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:32.668460 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:36.668453714 +0000 UTC m=+8.782768702 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"console-config" not registered Oct 30 02:41:32.668561 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:32.668480 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:32.668561 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:32.668543 5882 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 02:41:32.668894 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:32.668595 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:36.668582412 +0000 UTC m=+8.782897381 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"console-oauth-config" not registered Oct 30 02:41:32.769330 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:32.769298 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2vg\" (UniqueName: \"kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:32.769475 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:32.769462 5882 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 02:41:32.769527 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:32.769486 5882 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 02:41:32.769527 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:32.769498 5882 projected.go:196] Error preparing data for projected volume kube-api-access-nl2vg for pod openshift-console/console-647f7b9c85-ldfjx: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 02:41:32.769600 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:32.769544 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:36.769530964 +0000 UTC m=+8.883845939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nl2vg" (UniqueName: "kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 02:41:33.386081 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:33.386046 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:33.404914 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:33.404889 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:35.809318 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:35.809285 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j9v4r" Oct 30 02:41:36.240032 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:36.239964 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-dc789c958-vff7h" Oct 30 02:41:36.710675 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:36.710639 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:36.710870 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:36.710715 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:36.710870 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:36.710740 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:36.710870 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:36.710769 5882 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 02:41:36.710870 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:36.710832 5882 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 02:41:36.710870 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:36.710854 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:44.710834689 +0000 UTC m=+16.825149642 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 02:41:36.710870 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:36.710858 5882 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 02:41:36.711190 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:36.710900 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:44.710881468 +0000 UTC m=+16.825196449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 02:41:36.711190 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:36.710773 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:36.711190 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:36.710924 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:44.710908822 +0000 UTC m=+16.825223792 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"console-serving-cert" not registered Oct 30 02:41:36.711190 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:36.710945 5882 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 02:41:36.711190 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:36.710971 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:36.711190 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:36.711001 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:44.7109887 +0000 UTC m=+16.825303673 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"service-ca" not registered Oct 30 02:41:36.711190 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:36.711025 5882 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 02:41:36.711190 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:36.711047 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:36.711190 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:36.711062 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:44.711049993 +0000 UTC m=+16.825364961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"console-config" not registered Oct 30 02:41:36.711190 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:36.711115 5882 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 02:41:36.711190 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:36.711157 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:44.711145429 +0000 UTC m=+16.825460410 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"console-oauth-config" not registered Oct 30 02:41:36.812134 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:36.812099 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2vg\" (UniqueName: \"kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:36.812509 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:36.812252 5882 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 02:41:36.812509 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:36.812271 5882 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 02:41:36.812509 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:36.812281 5882 projected.go:196] Error preparing data for projected volume kube-api-access-nl2vg for pod openshift-console/console-647f7b9c85-ldfjx: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 02:41:36.812509 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:36.812332 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:44.812319813 +0000 UTC m=+16.926634778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nl2vg" (UniqueName: "kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 02:41:36.968077 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:36.968006 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xkkvm" Oct 30 02:41:37.171653 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:37.171599 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:41:41.880466 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:41.880385 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 02:41:41.880930 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:41.880726 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="alertmanager" containerID="cri-o://b275acc06ff965bb21d8a827ff90709765488dc1693af7ff8796e7f71839284a" gracePeriod=120 Oct 30 02:41:41.880930 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:41.880779 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="kube-rbac-proxy-metric" containerID="cri-o://d8544c73aa52ed57a312182780a274c1313b7d1433b37282644dc43e40c38000" gracePeriod=120 Oct 30 02:41:41.880930 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:41.880803 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="kube-rbac-proxy-web" containerID="cri-o://dc5b32e7376c2369d0e322c9d544b1a3d954a5ff21be5b69c11ed00dc3ea1ddd" gracePeriod=120 Oct 30 02:41:41.880930 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:41.880826 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="prom-label-proxy" containerID="cri-o://cfdba29c45f9d63d76dc9654306b899c4d640075a5dc791c02b01392b7d16914" gracePeriod=120 Oct 30 02:41:41.880930 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:41.880878 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="config-reloader" containerID="cri-o://9da09420916edc7f979b2d38c165556fbff94ac53a75c645c315df70ab9edc2d" gracePeriod=120 Oct 30 02:41:41.880930 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:41.880833 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="kube-rbac-proxy" containerID="cri-o://7277f4a31564fee3139b0db16e3efe98870b42fff7421f6d1f7db3046282f39c" gracePeriod=120 Oct 30 02:41:42.129051 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.129017 5882 generic.go:356] "Generic (PLEG): container finished" podID="992d8669-c914-486f-83ea-688dc69ca5a0" containerID="cfdba29c45f9d63d76dc9654306b899c4d640075a5dc791c02b01392b7d16914" exitCode=0 Oct 30 02:41:42.129051 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.129048 5882 generic.go:356] "Generic (PLEG): container finished" podID="992d8669-c914-486f-83ea-688dc69ca5a0" containerID="d8544c73aa52ed57a312182780a274c1313b7d1433b37282644dc43e40c38000" exitCode=0 Oct 30 02:41:42.129244 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.129059 5882 generic.go:356] "Generic (PLEG): container finished" podID="992d8669-c914-486f-83ea-688dc69ca5a0" containerID="7277f4a31564fee3139b0db16e3efe98870b42fff7421f6d1f7db3046282f39c" exitCode=0 Oct 30 02:41:42.129244 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.129067 5882 generic.go:356] "Generic (PLEG): container finished" podID="992d8669-c914-486f-83ea-688dc69ca5a0" containerID="dc5b32e7376c2369d0e322c9d544b1a3d954a5ff21be5b69c11ed00dc3ea1ddd" exitCode=0 Oct 30 02:41:42.129244 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.129076 5882 generic.go:356] "Generic (PLEG): container finished" podID="992d8669-c914-486f-83ea-688dc69ca5a0" containerID="9da09420916edc7f979b2d38c165556fbff94ac53a75c645c315df70ab9edc2d" exitCode=0 Oct 30 02:41:42.129244 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.129084 5882 generic.go:356] "Generic (PLEG): container finished" podID="992d8669-c914-486f-83ea-688dc69ca5a0" containerID="b275acc06ff965bb21d8a827ff90709765488dc1693af7ff8796e7f71839284a" exitCode=0 Oct 30 02:41:42.129244 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.129084 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"992d8669-c914-486f-83ea-688dc69ca5a0","Type":"ContainerDied","Data":"cfdba29c45f9d63d76dc9654306b899c4d640075a5dc791c02b01392b7d16914"} Oct 30 02:41:42.129244 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.129171 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"992d8669-c914-486f-83ea-688dc69ca5a0","Type":"ContainerDied","Data":"d8544c73aa52ed57a312182780a274c1313b7d1433b37282644dc43e40c38000"} Oct 30 02:41:42.129244 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.129215 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"992d8669-c914-486f-83ea-688dc69ca5a0","Type":"ContainerDied","Data":"7277f4a31564fee3139b0db16e3efe98870b42fff7421f6d1f7db3046282f39c"} Oct 30 02:41:42.129518 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.129257 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"992d8669-c914-486f-83ea-688dc69ca5a0","Type":"ContainerDied","Data":"dc5b32e7376c2369d0e322c9d544b1a3d954a5ff21be5b69c11ed00dc3ea1ddd"} Oct 30 02:41:42.129518 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.129317 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"992d8669-c914-486f-83ea-688dc69ca5a0","Type":"ContainerDied","Data":"9da09420916edc7f979b2d38c165556fbff94ac53a75c645c315df70ab9edc2d"} Oct 30 02:41:42.129518 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.129370 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"992d8669-c914-486f-83ea-688dc69ca5a0","Type":"ContainerDied","Data":"b275acc06ff965bb21d8a827ff90709765488dc1693af7ff8796e7f71839284a"} Oct 30 02:41:42.160818 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.160761 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:42.259577 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.259541 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-config-volume\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-config-volume\") pod \"992d8669-c914-486f-83ea-688dc69ca5a0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " Oct 30 02:41:42.259744 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.259605 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-web-config\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-web-config\") pod \"992d8669-c914-486f-83ea-688dc69ca5a0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " Oct 30 02:41:42.259744 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.259654 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/992d8669-c914-486f-83ea-688dc69ca5a0-tls-assets\" (UniqueName: \"kubernetes.io/projected/992d8669-c914-486f-83ea-688dc69ca5a0-tls-assets\") pod \"992d8669-c914-486f-83ea-688dc69ca5a0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " Oct 30 02:41:42.259744 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.259672 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy\") pod \"992d8669-c914-486f-83ea-688dc69ca5a0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " Oct 30 02:41:42.259744 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.259696 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"992d8669-c914-486f-83ea-688dc69ca5a0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " Oct 30 02:41:42.259744 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.259727 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/992d8669-c914-486f-83ea-688dc69ca5a0-config-out\" (UniqueName: \"kubernetes.io/empty-dir/992d8669-c914-486f-83ea-688dc69ca5a0-config-out\") pod \"992d8669-c914-486f-83ea-688dc69ca5a0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " Oct 30 02:41:42.260030 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.259763 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-main-tls\") pod \"992d8669-c914-486f-83ea-688dc69ca5a0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " Oct 30 02:41:42.260030 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.259813 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/992d8669-c914-486f-83ea-688dc69ca5a0-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/992d8669-c914-486f-83ea-688dc69ca5a0-alertmanager-main-db\") pod \"992d8669-c914-486f-83ea-688dc69ca5a0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " Oct 30 02:41:42.260030 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.259855 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"992d8669-c914-486f-83ea-688dc69ca5a0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " Oct 30 02:41:42.260030 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.259902 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/992d8669-c914-486f-83ea-688dc69ca5a0-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/992d8669-c914-486f-83ea-688dc69ca5a0-metrics-client-ca\") pod \"992d8669-c914-486f-83ea-688dc69ca5a0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " Oct 30 02:41:42.260030 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.259973 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/992d8669-c914-486f-83ea-688dc69ca5a0-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/992d8669-c914-486f-83ea-688dc69ca5a0-alertmanager-trusted-ca-bundle\") pod \"992d8669-c914-486f-83ea-688dc69ca5a0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " Oct 30 02:41:42.260030 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.260009 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/992d8669-c914-486f-83ea-688dc69ca5a0-kube-api-access-kndrd\" (UniqueName: \"kubernetes.io/projected/992d8669-c914-486f-83ea-688dc69ca5a0-kube-api-access-kndrd\") pod \"992d8669-c914-486f-83ea-688dc69ca5a0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " Oct 30 02:41:42.260347 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.260048 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-cluster-tls-config\") pod \"992d8669-c914-486f-83ea-688dc69ca5a0\" (UID: \"992d8669-c914-486f-83ea-688dc69ca5a0\") " Oct 30 02:41:42.261273 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.260969 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/992d8669-c914-486f-83ea-688dc69ca5a0-alertmanager-trusted-ca-bundle" pod "992d8669-c914-486f-83ea-688dc69ca5a0" (UID: "992d8669-c914-486f-83ea-688dc69ca5a0"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:42.262487 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.262455 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/992d8669-c914-486f-83ea-688dc69ca5a0-metrics-client-ca" pod "992d8669-c914-486f-83ea-688dc69ca5a0" (UID: "992d8669-c914-486f-83ea-688dc69ca5a0"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:42.262585 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.262526 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992d8669-c914-486f-83ea-688dc69ca5a0-alertmanager-main-db" pod "992d8669-c914-486f-83ea-688dc69ca5a0" (UID: "992d8669-c914-486f-83ea-688dc69ca5a0"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:42.262946 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.262918 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-config-volume" pod "992d8669-c914-486f-83ea-688dc69ca5a0" (UID: "992d8669-c914-486f-83ea-688dc69ca5a0"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:42.263954 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.263920 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy-web" pod "992d8669-c914-486f-83ea-688dc69ca5a0" (UID: "992d8669-c914-486f-83ea-688dc69ca5a0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:42.264053 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.264023 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy" pod "992d8669-c914-486f-83ea-688dc69ca5a0" (UID: "992d8669-c914-486f-83ea-688dc69ca5a0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:42.264166 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.263910 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992d8669-c914-486f-83ea-688dc69ca5a0-tls-assets" pod "992d8669-c914-486f-83ea-688dc69ca5a0" (UID: "992d8669-c914-486f-83ea-688dc69ca5a0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:42.264166 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.264157 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy-metric" pod "992d8669-c914-486f-83ea-688dc69ca5a0" (UID: "992d8669-c914-486f-83ea-688dc69ca5a0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:42.264310 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.264291 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-main-tls" pod "992d8669-c914-486f-83ea-688dc69ca5a0" (UID: "992d8669-c914-486f-83ea-688dc69ca5a0"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:42.264571 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.264554 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992d8669-c914-486f-83ea-688dc69ca5a0-kube-api-access-kndrd" pod "992d8669-c914-486f-83ea-688dc69ca5a0" (UID: "992d8669-c914-486f-83ea-688dc69ca5a0"). InnerVolumeSpecName "kube-api-access-kndrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:42.265737 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.265711 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992d8669-c914-486f-83ea-688dc69ca5a0-config-out" pod "992d8669-c914-486f-83ea-688dc69ca5a0" (UID: "992d8669-c914-486f-83ea-688dc69ca5a0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:42.267242 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.267223 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-cluster-tls-config" pod "992d8669-c914-486f-83ea-688dc69ca5a0" (UID: "992d8669-c914-486f-83ea-688dc69ca5a0"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:42.274917 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.274888 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-web-config" pod "992d8669-c914-486f-83ea-688dc69ca5a0" (UID: "992d8669-c914-486f-83ea-688dc69ca5a0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:42.361585 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.361554 5882 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-config-volume\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:42.361585 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.361582 5882 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-web-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:42.361585 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.361591 5882 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/992d8669-c914-486f-83ea-688dc69ca5a0-tls-assets\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:42.361804 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.361599 5882 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:42.361804 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.361608 5882 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:42.361804 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.361638 5882 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/992d8669-c914-486f-83ea-688dc69ca5a0-config-out\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:42.361804 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.361647 5882 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-main-tls\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:42.361804 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.361657 5882 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/992d8669-c914-486f-83ea-688dc69ca5a0-alertmanager-main-db\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:42.361804 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.361666 5882 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:42.361804 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.361674 5882 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/992d8669-c914-486f-83ea-688dc69ca5a0-metrics-client-ca\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:42.361804 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.361683 5882 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/992d8669-c914-486f-83ea-688dc69ca5a0-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:42.361804 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.361691 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kndrd\" (UniqueName: \"kubernetes.io/projected/992d8669-c914-486f-83ea-688dc69ca5a0-kube-api-access-kndrd\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:42.361804 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:42.361699 5882 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/992d8669-c914-486f-83ea-688dc69ca5a0-cluster-tls-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.138980 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.138939 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"992d8669-c914-486f-83ea-688dc69ca5a0","Type":"ContainerDied","Data":"1fac83ee3517ccc5d3939b215e63adc0f51a43f4ef74247f00322b61fd72cf77"} Oct 30 02:41:43.139368 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.138986 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.139368 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.139064 5882 scope.go:122] "RemoveContainer" containerID="cfdba29c45f9d63d76dc9654306b899c4d640075a5dc791c02b01392b7d16914" Oct 30 02:41:43.157465 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.157446 5882 scope.go:122] "RemoveContainer" containerID="d8544c73aa52ed57a312182780a274c1313b7d1433b37282644dc43e40c38000" Oct 30 02:41:43.164299 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.164280 5882 scope.go:122] "RemoveContainer" containerID="7277f4a31564fee3139b0db16e3efe98870b42fff7421f6d1f7db3046282f39c" Oct 30 02:41:43.170867 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.170849 5882 scope.go:122] "RemoveContainer" containerID="dc5b32e7376c2369d0e322c9d544b1a3d954a5ff21be5b69c11ed00dc3ea1ddd" Oct 30 02:41:43.177111 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.177094 5882 scope.go:122] "RemoveContainer" containerID="9da09420916edc7f979b2d38c165556fbff94ac53a75c645c315df70ab9edc2d" Oct 30 02:41:43.183947 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.183927 5882 scope.go:122] "RemoveContainer" containerID="b275acc06ff965bb21d8a827ff90709765488dc1693af7ff8796e7f71839284a" Oct 30 02:41:43.190726 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.190705 5882 scope.go:122] "RemoveContainer" containerID="8171d08cc27a8278b5dfe6c057eb59c5a9992947a7555c85c48f4780cf193488" Oct 30 02:41:43.203413 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.203390 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 02:41:43.215303 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.215280 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 02:41:43.240370 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240340 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 02:41:43.240659 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240646 5882 cpu_manager.go:414] "containerMap: removing container" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="config-reloader" Oct 30 02:41:43.240698 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240660 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="config-reloader" Oct 30 02:41:43.240698 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240676 5882 cpu_manager.go:414] "containerMap: removing container" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="kube-rbac-proxy-metric" Oct 30 02:41:43.240698 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240682 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="kube-rbac-proxy-metric" Oct 30 02:41:43.240698 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240689 5882 cpu_manager.go:414] "containerMap: removing container" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="alertmanager" Oct 30 02:41:43.240698 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240694 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="alertmanager" Oct 30 02:41:43.240934 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240705 5882 cpu_manager.go:414] "containerMap: removing container" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="kube-rbac-proxy-web" Oct 30 02:41:43.240934 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240710 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="kube-rbac-proxy-web" Oct 30 02:41:43.240934 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240721 5882 cpu_manager.go:414] "containerMap: removing container" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="prom-label-proxy" Oct 30 02:41:43.240934 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240725 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="prom-label-proxy" Oct 30 02:41:43.240934 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240733 5882 cpu_manager.go:414] "containerMap: removing container" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="kube-rbac-proxy" Oct 30 02:41:43.240934 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240738 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="kube-rbac-proxy" Oct 30 02:41:43.240934 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240746 5882 cpu_manager.go:414] "containerMap: removing container" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="init-config-reloader" Oct 30 02:41:43.240934 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240750 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="init-config-reloader" Oct 30 02:41:43.240934 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240795 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="config-reloader" Oct 30 02:41:43.240934 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240804 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="kube-rbac-proxy-metric" Oct 30 02:41:43.240934 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240810 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="alertmanager" Oct 30 02:41:43.240934 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240817 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="kube-rbac-proxy-web" Oct 30 02:41:43.240934 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240824 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="prom-label-proxy" Oct 30 02:41:43.240934 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240830 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="kube-rbac-proxy" Oct 30 02:41:43.240934 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.240836 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" containerName="init-config-reloader" Oct 30 02:41:43.267767 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.267744 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.270110 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.270081 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 02:41:43.283253 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.283232 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 02:41:43.283348 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.283272 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 02:41:43.283801 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.283782 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 02:41:43.283881 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.283857 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 02:41:43.283922 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.283884 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9rd8b\"" Oct 30 02:41:43.283962 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.283916 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 02:41:43.283997 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.283992 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 02:41:43.284046 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.284027 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 02:41:43.284046 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.284033 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 02:41:43.295909 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.295892 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 02:41:43.371237 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.371199 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7vxk\" (UniqueName: \"kubernetes.io/projected/7171f659-3a2a-4a26-bd1d-2047f058d00c-kube-api-access-h7vxk\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.371237 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.371239 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7171f659-3a2a-4a26-bd1d-2047f058d00c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.371415 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.371269 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7171f659-3a2a-4a26-bd1d-2047f058d00c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.371415 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.371403 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.371485 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.371436 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7171f659-3a2a-4a26-bd1d-2047f058d00c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.371485 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.371457 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7171f659-3a2a-4a26-bd1d-2047f058d00c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.371567 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.371487 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7171f659-3a2a-4a26-bd1d-2047f058d00c-config-out\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.371641 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.371604 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.371683 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.371653 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-config-volume\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.371724 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.371695 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.371724 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.371714 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-web-config\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.371811 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.371749 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.371859 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.371843 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.473075 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.472996 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.473075 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.473028 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-config-volume\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.473075 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.473047 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.473332 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.473094 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-web-config\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.473332 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.473147 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.473332 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.473176 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.473332 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.473221 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7vxk\" (UniqueName: \"kubernetes.io/projected/7171f659-3a2a-4a26-bd1d-2047f058d00c-kube-api-access-h7vxk\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.473332 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.473238 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7171f659-3a2a-4a26-bd1d-2047f058d00c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.473332 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.473271 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7171f659-3a2a-4a26-bd1d-2047f058d00c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.473332 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.473315 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.473704 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.473339 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7171f659-3a2a-4a26-bd1d-2047f058d00c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.473704 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.473380 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7171f659-3a2a-4a26-bd1d-2047f058d00c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.473704 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.473395 5882 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 02:41:43.473704 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.473441 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7171f659-3a2a-4a26-bd1d-2047f058d00c-config-out\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.473916 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.473728 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7171f659-3a2a-4a26-bd1d-2047f058d00c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.475073 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.474162 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7171f659-3a2a-4a26-bd1d-2047f058d00c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.475073 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.475027 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7171f659-3a2a-4a26-bd1d-2047f058d00c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.476111 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.476062 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7171f659-3a2a-4a26-bd1d-2047f058d00c-config-out\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.476223 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.476141 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.476731 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.476705 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.476952 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.476934 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-web-config\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.477027 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.476962 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-config-volume\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.477027 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.477007 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.477272 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.477254 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7171f659-3a2a-4a26-bd1d-2047f058d00c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.477327 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.477262 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.477887 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.477869 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7171f659-3a2a-4a26-bd1d-2047f058d00c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.496795 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.496774 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7vxk\" (UniqueName: \"kubernetes.io/projected/7171f659-3a2a-4a26-bd1d-2047f058d00c-kube-api-access-h7vxk\") pod \"alertmanager-main-0\" (UID: \"7171f659-3a2a-4a26-bd1d-2047f058d00c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:43.665915 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:43.665863 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:41:44.008573 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:44.008545 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="992d8669-c914-486f-83ea-688dc69ca5a0" path="/var/lib/kubelet/pods/992d8669-c914-486f-83ea-688dc69ca5a0/volumes" Oct 30 02:41:44.015952 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:44.015846 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 02:41:44.018662 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:44.018634 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7171f659_3a2a_4a26_bd1d_2047f058d00c.slice/crio-e076bbce606f29b14708abfb7896ebc1342549f579dd57048ec40a229c280006 WatchSource:0}: Error finding container e076bbce606f29b14708abfb7896ebc1342549f579dd57048ec40a229c280006: Status 404 returned error can't find the container with id e076bbce606f29b14708abfb7896ebc1342549f579dd57048ec40a229c280006 Oct 30 02:41:44.144492 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:44.144462 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7171f659-3a2a-4a26-bd1d-2047f058d00c","Type":"ContainerStarted","Data":"cc9266cd72ea765df0c606a4606bf0c374ab7ffe2c3cb6a74d0526f70b254535"} Oct 30 02:41:44.144901 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:44.144550 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7171f659-3a2a-4a26-bd1d-2047f058d00c","Type":"ContainerStarted","Data":"e076bbce606f29b14708abfb7896ebc1342549f579dd57048ec40a229c280006"} Oct 30 02:41:44.787915 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:44.787841 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:44.788038 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:44.787927 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:44.788038 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:44.787972 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:44.788038 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:44.787988 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:44.788038 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:44.787999 5882 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 02:41:44.788038 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:44.788019 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:44.788270 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:44.788047 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:44.788270 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:44.788047 5882 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 02:41:44.788270 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:44.788065 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:00.788047211 +0000 UTC m=+32.902362177 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"console-oauth-config" not registered Oct 30 02:41:44.788270 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:44.788082 5882 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 02:41:44.788270 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:44.788100 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:00.788086778 +0000 UTC m=+32.902401750 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 02:41:44.788270 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:44.788065 5882 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 02:41:44.788270 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:44.788083 5882 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 02:41:44.788270 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:44.788123 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:00.788112554 +0000 UTC m=+32.902427530 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 02:41:44.788270 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:44.788139 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:00.788129706 +0000 UTC m=+32.902444661 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"console-serving-cert" not registered Oct 30 02:41:44.788270 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:44.788148 5882 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 02:41:44.788270 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:44.788159 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:00.78815098 +0000 UTC m=+32.902465942 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"service-ca" not registered Oct 30 02:41:44.788270 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:44.788234 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:00.788216151 +0000 UTC m=+32.902531112 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : object "openshift-console"/"console-config" not registered Oct 30 02:41:44.888968 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:44.888929 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2vg\" (UniqueName: \"kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg\") pod \"console-647f7b9c85-ldfjx\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:44.889137 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:44.889105 5882 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 02:41:44.889137 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:44.889127 5882 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 02:41:44.889137 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:44.889136 5882 projected.go:196] Error preparing data for projected volume kube-api-access-nl2vg for pod openshift-console/console-647f7b9c85-ldfjx: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 02:41:44.889235 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:44.889191 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg podName:49f725c1-f41a-403d-b729-a170fc1fe687 nodeName:}" failed. No retries permitted until 2026-10-30 02:42:00.889175412 +0000 UTC m=+33.003490366 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nl2vg" (UniqueName: "kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg") pod "console-647f7b9c85-ldfjx" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 02:41:45.152762 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.152676 5882 generic.go:356] "Generic (PLEG): container finished" podID="7171f659-3a2a-4a26-bd1d-2047f058d00c" containerID="cc9266cd72ea765df0c606a4606bf0c374ab7ffe2c3cb6a74d0526f70b254535" exitCode=0 Oct 30 02:41:45.152762 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.152725 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7171f659-3a2a-4a26-bd1d-2047f058d00c","Type":"ContainerDied","Data":"cc9266cd72ea765df0c606a4606bf0c374ab7ffe2c3cb6a74d0526f70b254535"} Oct 30 02:41:45.153256 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.152800 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7171f659-3a2a-4a26-bd1d-2047f058d00c","Type":"ContainerStarted","Data":"956774cea763da0bd9be0a78c742d1e825400ce95b21ae4e445137eb627be2a1"} Oct 30 02:41:45.153256 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.152827 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7171f659-3a2a-4a26-bd1d-2047f058d00c","Type":"ContainerStarted","Data":"317f2788c745d9e438b2a04f650f750c63f6e77cce3777a188401d7d2d95b004"} Oct 30 02:41:45.153256 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.152862 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7171f659-3a2a-4a26-bd1d-2047f058d00c","Type":"ContainerStarted","Data":"c466ed42ab1aea22393ac57b84918e5b5c887993c6ad7aa889bca3f4e5a48430"} Oct 30 02:41:45.153256 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.152887 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7171f659-3a2a-4a26-bd1d-2047f058d00c","Type":"ContainerStarted","Data":"6acbc9157f049777f7ea6a3d53e44023eba5bf0f85a24c74ea33e402e8775324"} Oct 30 02:41:45.153256 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.152911 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7171f659-3a2a-4a26-bd1d-2047f058d00c","Type":"ContainerStarted","Data":"ed7a82dd4e8c33f7090e72ef766cb9b83a46359065d33c7359f79fa0e7fd3841"} Oct 30 02:41:45.153256 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.152932 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7171f659-3a2a-4a26-bd1d-2047f058d00c","Type":"ContainerStarted","Data":"d3b03c6cd5c134bb966603a045ee96acb3b331047908054e12963f8becf90ce0"} Oct 30 02:41:45.219377 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.219320 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.205496767 podStartE2EDuration="2.219308003s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.811236ms" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:45.218178949 +0000 UTC m=+17.332493924" watchObservedRunningTime="2026-10-30 02:41:45.219308003 +0000 UTC m=+17.333623023" Oct 30 02:41:45.847949 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.847912 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf"] Oct 30 02:41:45.870256 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.870224 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:45.886577 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.886553 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 02:41:45.887105 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.887063 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 02:41:45.887220 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.887125 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 02:41:45.887220 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.887128 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wvdtf\"" Oct 30 02:41:45.887220 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.887186 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 02:41:45.899217 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.899191 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf"] Oct 30 02:41:45.939096 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.939061 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 02:41:45.951932 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.951909 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 02:41:45.999782 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.999746 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c426339e-5bc3-45ec-a994-7b205c2cdc44-serving-certs-ca-bundle\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:45.999899 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.999810 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c426339e-5bc3-45ec-a994-7b205c2cdc44-federate-client-tls\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:45.999948 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.999923 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c426339e-5bc3-45ec-a994-7b205c2cdc44-secret-telemeter-client\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:45.999999 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:45.999987 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c426339e-5bc3-45ec-a994-7b205c2cdc44-telemeter-client-tls\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.000075 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.000052 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c426339e-5bc3-45ec-a994-7b205c2cdc44-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.000109 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.000095 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c426339e-5bc3-45ec-a994-7b205c2cdc44-metrics-client-ca\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.000145 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.000114 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqbrx\" (UniqueName: \"kubernetes.io/projected/c426339e-5bc3-45ec-a994-7b205c2cdc44-kube-api-access-hqbrx\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.000209 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.000195 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c426339e-5bc3-45ec-a994-7b205c2cdc44-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.100959 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.100870 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c426339e-5bc3-45ec-a994-7b205c2cdc44-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.100959 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.100934 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c426339e-5bc3-45ec-a994-7b205c2cdc44-serving-certs-ca-bundle\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.101185 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.100984 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c426339e-5bc3-45ec-a994-7b205c2cdc44-federate-client-tls\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.101185 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.101039 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c426339e-5bc3-45ec-a994-7b205c2cdc44-secret-telemeter-client\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.101185 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.101102 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c426339e-5bc3-45ec-a994-7b205c2cdc44-telemeter-client-tls\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.101185 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.101130 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c426339e-5bc3-45ec-a994-7b205c2cdc44-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.101185 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.101160 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c426339e-5bc3-45ec-a994-7b205c2cdc44-metrics-client-ca\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.101422 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.101193 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqbrx\" (UniqueName: \"kubernetes.io/projected/c426339e-5bc3-45ec-a994-7b205c2cdc44-kube-api-access-hqbrx\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.101852 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.101821 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c426339e-5bc3-45ec-a994-7b205c2cdc44-serving-certs-ca-bundle\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.102056 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.102018 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c426339e-5bc3-45ec-a994-7b205c2cdc44-metrics-client-ca\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.102140 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.102100 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c426339e-5bc3-45ec-a994-7b205c2cdc44-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.103546 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.103516 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c426339e-5bc3-45ec-a994-7b205c2cdc44-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.103695 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.103680 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c426339e-5bc3-45ec-a994-7b205c2cdc44-federate-client-tls\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.103794 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.103773 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c426339e-5bc3-45ec-a994-7b205c2cdc44-telemeter-client-tls\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.103859 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.103825 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c426339e-5bc3-45ec-a994-7b205c2cdc44-secret-telemeter-client\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.124805 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.124782 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqbrx\" (UniqueName: \"kubernetes.io/projected/c426339e-5bc3-45ec-a994-7b205c2cdc44-kube-api-access-hqbrx\") pod \"telemeter-client-84cd9b5f86-vtnlf\" (UID: \"c426339e-5bc3-45ec-a994-7b205c2cdc44\") " pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.229369 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.229336 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" Oct 30 02:41:46.560913 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.560882 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf"] Oct 30 02:41:46.564935 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:46.564910 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc426339e_5bc3_45ec_a994_7b205c2cdc44.slice/crio-d8dbc6419fbe1bb8c9b90aefb8983a98944e1eb0add0ae913f6d50e95a9afc6d WatchSource:0}: Error finding container d8dbc6419fbe1bb8c9b90aefb8983a98944e1eb0add0ae913f6d50e95a9afc6d: Status 404 returned error can't find the container with id d8dbc6419fbe1bb8c9b90aefb8983a98944e1eb0add0ae913f6d50e95a9afc6d Oct 30 02:41:46.566928 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.566906 5882 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:41:46.824342 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.824263 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 02:41:46.824951 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.824666 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="prometheus" containerID="cri-o://b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757" gracePeriod=600 Oct 30 02:41:46.824951 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.824688 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="kube-rbac-proxy" containerID="cri-o://d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8" gracePeriod=600 Oct 30 02:41:46.824951 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.824767 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="thanos-sidecar" containerID="cri-o://87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e" gracePeriod=600 Oct 30 02:41:46.824951 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.824855 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="config-reloader" containerID="cri-o://a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd" gracePeriod=600 Oct 30 02:41:46.824951 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.824922 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="kube-rbac-proxy-web" containerID="cri-o://871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae" gracePeriod=600 Oct 30 02:41:46.825286 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:46.825235 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="kube-rbac-proxy-thanos" containerID="cri-o://cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c" gracePeriod=600 Oct 30 02:41:47.120603 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.120575 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.164128 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.164085 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" event={"ID":"c426339e-5bc3-45ec-a994-7b205c2cdc44","Type":"ContainerStarted","Data":"d8dbc6419fbe1bb8c9b90aefb8983a98944e1eb0add0ae913f6d50e95a9afc6d"} Oct 30 02:41:47.167326 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.167302 5882 generic.go:356] "Generic (PLEG): container finished" podID="43ccba25-b0ea-47c8-b664-105feea298a3" containerID="cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c" exitCode=0 Oct 30 02:41:47.167326 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.167323 5882 generic.go:356] "Generic (PLEG): container finished" podID="43ccba25-b0ea-47c8-b664-105feea298a3" containerID="d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8" exitCode=0 Oct 30 02:41:47.167483 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.167336 5882 generic.go:356] "Generic (PLEG): container finished" podID="43ccba25-b0ea-47c8-b664-105feea298a3" containerID="871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae" exitCode=0 Oct 30 02:41:47.167483 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.167347 5882 generic.go:356] "Generic (PLEG): container finished" podID="43ccba25-b0ea-47c8-b664-105feea298a3" containerID="87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e" exitCode=0 Oct 30 02:41:47.167483 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.167353 5882 generic.go:356] "Generic (PLEG): container finished" podID="43ccba25-b0ea-47c8-b664-105feea298a3" containerID="a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd" exitCode=0 Oct 30 02:41:47.167483 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.167360 5882 generic.go:356] "Generic (PLEG): container finished" podID="43ccba25-b0ea-47c8-b664-105feea298a3" containerID="b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757" exitCode=0 Oct 30 02:41:47.167483 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.167383 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.167483 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.167383 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43ccba25-b0ea-47c8-b664-105feea298a3","Type":"ContainerDied","Data":"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c"} Oct 30 02:41:47.167688 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.167509 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43ccba25-b0ea-47c8-b664-105feea298a3","Type":"ContainerDied","Data":"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8"} Oct 30 02:41:47.167688 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.167537 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43ccba25-b0ea-47c8-b664-105feea298a3","Type":"ContainerDied","Data":"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae"} Oct 30 02:41:47.167688 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.167562 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43ccba25-b0ea-47c8-b664-105feea298a3","Type":"ContainerDied","Data":"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e"} Oct 30 02:41:47.167688 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.167577 5882 scope.go:122] "RemoveContainer" containerID="cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c" Oct 30 02:41:47.167688 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.167590 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43ccba25-b0ea-47c8-b664-105feea298a3","Type":"ContainerDied","Data":"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd"} Oct 30 02:41:47.167688 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.167634 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43ccba25-b0ea-47c8-b664-105feea298a3","Type":"ContainerDied","Data":"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757"} Oct 30 02:41:47.167688 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.167659 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"43ccba25-b0ea-47c8-b664-105feea298a3","Type":"ContainerDied","Data":"9372225da62eb113e6c7f903c1f76b737db097a185b64543f4398c2771a34eb3"} Oct 30 02:41:47.174953 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.174935 5882 scope.go:122] "RemoveContainer" containerID="d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8" Oct 30 02:41:47.184450 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.183955 5882 scope.go:122] "RemoveContainer" containerID="871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae" Oct 30 02:41:47.195264 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.195243 5882 scope.go:122] "RemoveContainer" containerID="87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e" Oct 30 02:41:47.202841 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.202819 5882 scope.go:122] "RemoveContainer" containerID="a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd" Oct 30 02:41:47.209534 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.209516 5882 scope.go:122] "RemoveContainer" containerID="b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757" Oct 30 02:41:47.213997 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.213977 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ccba25-b0ea-47c8-b664-105feea298a3-config-out\" (UniqueName: \"kubernetes.io/empty-dir/43ccba25-b0ea-47c8-b664-105feea298a3-config-out\") pod \"43ccba25-b0ea-47c8-b664-105feea298a3\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " Oct 30 02:41:47.214096 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.214016 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-db\") pod \"43ccba25-b0ea-47c8-b664-105feea298a3\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " Oct 30 02:41:47.214096 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.214053 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-config\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-config\") pod \"43ccba25-b0ea-47c8-b664-105feea298a3\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " Oct 30 02:41:47.214096 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.214091 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-web-config\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-web-config\") pod \"43ccba25-b0ea-47c8-b664-105feea298a3\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " Oct 30 02:41:47.214257 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.214214 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-kubelet-serving-ca-bundle\") pod \"43ccba25-b0ea-47c8-b664-105feea298a3\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " Oct 30 02:41:47.214330 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.214272 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-grpc-tls\") pod \"43ccba25-b0ea-47c8-b664-105feea298a3\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " Oct 30 02:41:47.214330 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.214303 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-kube-rbac-proxy\") pod \"43ccba25-b0ea-47c8-b664-105feea298a3\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " Oct 30 02:41:47.214482 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.214336 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-tls\") pod \"43ccba25-b0ea-47c8-b664-105feea298a3\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " Oct 30 02:41:47.214482 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.214376 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"43ccba25-b0ea-47c8-b664-105feea298a3\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " Oct 30 02:41:47.214482 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.214414 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-thanos-prometheus-http-client-file\") pod \"43ccba25-b0ea-47c8-b664-105feea298a3\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " Oct 30 02:41:47.214482 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.214451 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-trusted-ca-bundle\") pod \"43ccba25-b0ea-47c8-b664-105feea298a3\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " Oct 30 02:41:47.214705 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.214514 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ccba25-b0ea-47c8-b664-105feea298a3-kube-api-access-pl9n4\" (UniqueName: \"kubernetes.io/projected/43ccba25-b0ea-47c8-b664-105feea298a3-kube-api-access-pl9n4\") pod \"43ccba25-b0ea-47c8-b664-105feea298a3\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " Oct 30 02:41:47.214705 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.214560 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-2\") pod \"43ccba25-b0ea-47c8-b664-105feea298a3\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " Oct 30 02:41:47.214705 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.214564 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-kubelet-serving-ca-bundle" pod "43ccba25-b0ea-47c8-b664-105feea298a3" (UID: "43ccba25-b0ea-47c8-b664-105feea298a3"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:47.214705 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.214591 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-0\") pod \"43ccba25-b0ea-47c8-b664-105feea298a3\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " Oct 30 02:41:47.214705 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.214695 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-db" pod "43ccba25-b0ea-47c8-b664-105feea298a3" (UID: "43ccba25-b0ea-47c8-b664-105feea298a3"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:47.214976 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.214700 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ccba25-b0ea-47c8-b664-105feea298a3-tls-assets\" (UniqueName: \"kubernetes.io/projected/43ccba25-b0ea-47c8-b664-105feea298a3-tls-assets\") pod \"43ccba25-b0ea-47c8-b664-105feea298a3\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " Oct 30 02:41:47.214976 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.214795 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-metrics-client-certs\") pod \"43ccba25-b0ea-47c8-b664-105feea298a3\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " Oct 30 02:41:47.214976 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.214838 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"43ccba25-b0ea-47c8-b664-105feea298a3\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " Oct 30 02:41:47.214976 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.214889 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-serving-certs-ca-bundle\") pod \"43ccba25-b0ea-47c8-b664-105feea298a3\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " Oct 30 02:41:47.214976 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.214960 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-metrics-client-ca\") pod \"43ccba25-b0ea-47c8-b664-105feea298a3\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " Oct 30 02:41:47.215228 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.215006 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-1\") pod \"43ccba25-b0ea-47c8-b664-105feea298a3\" (UID: \"43ccba25-b0ea-47c8-b664-105feea298a3\") " Oct 30 02:41:47.215430 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.215396 5882 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.215430 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.215422 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-db\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.216602 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.216088 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-serving-certs-ca-bundle" pod "43ccba25-b0ea-47c8-b664-105feea298a3" (UID: "43ccba25-b0ea-47c8-b664-105feea298a3"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:47.217153 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.217127 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-2" pod "43ccba25-b0ea-47c8-b664-105feea298a3" (UID: "43ccba25-b0ea-47c8-b664-105feea298a3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:47.217718 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.217690 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-trusted-ca-bundle" pod "43ccba25-b0ea-47c8-b664-105feea298a3" (UID: "43ccba25-b0ea-47c8-b664-105feea298a3"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:47.217953 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.217918 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-config" pod "43ccba25-b0ea-47c8-b664-105feea298a3" (UID: "43ccba25-b0ea-47c8-b664-105feea298a3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:47.218079 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.218053 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ccba25-b0ea-47c8-b664-105feea298a3-tls-assets" pod "43ccba25-b0ea-47c8-b664-105feea298a3" (UID: "43ccba25-b0ea-47c8-b664-105feea298a3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:47.223086 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.219804 5882 scope.go:122] "RemoveContainer" containerID="53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb" Oct 30 02:41:47.223086 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.219871 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-metrics-client-ca" pod "43ccba25-b0ea-47c8-b664-105feea298a3" (UID: "43ccba25-b0ea-47c8-b664-105feea298a3"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:47.223086 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.220106 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-kube-rbac-proxy" pod "43ccba25-b0ea-47c8-b664-105feea298a3" (UID: "43ccba25-b0ea-47c8-b664-105feea298a3"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:47.223086 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.220903 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ccba25-b0ea-47c8-b664-105feea298a3-kube-api-access-pl9n4" pod "43ccba25-b0ea-47c8-b664-105feea298a3" (UID: "43ccba25-b0ea-47c8-b664-105feea298a3"). InnerVolumeSpecName "kube-api-access-pl9n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:47.223086 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.220921 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ccba25-b0ea-47c8-b664-105feea298a3-config-out" pod "43ccba25-b0ea-47c8-b664-105feea298a3" (UID: "43ccba25-b0ea-47c8-b664-105feea298a3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:47.223086 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.220942 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-thanos-prometheus-http-client-file" pod "43ccba25-b0ea-47c8-b664-105feea298a3" (UID: "43ccba25-b0ea-47c8-b664-105feea298a3"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:47.223086 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.221053 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-tls" pod "43ccba25-b0ea-47c8-b664-105feea298a3" (UID: "43ccba25-b0ea-47c8-b664-105feea298a3"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:47.223086 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.221082 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-kube-rbac-proxy-web" pod "43ccba25-b0ea-47c8-b664-105feea298a3" (UID: "43ccba25-b0ea-47c8-b664-105feea298a3"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:47.223086 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.221369 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-thanos-sidecar-tls" pod "43ccba25-b0ea-47c8-b664-105feea298a3" (UID: "43ccba25-b0ea-47c8-b664-105feea298a3"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:47.223086 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.221787 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-1" pod "43ccba25-b0ea-47c8-b664-105feea298a3" (UID: "43ccba25-b0ea-47c8-b664-105feea298a3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:47.223086 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.222001 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-grpc-tls" pod "43ccba25-b0ea-47c8-b664-105feea298a3" (UID: "43ccba25-b0ea-47c8-b664-105feea298a3"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:47.223086 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.223030 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-0" pod "43ccba25-b0ea-47c8-b664-105feea298a3" (UID: "43ccba25-b0ea-47c8-b664-105feea298a3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:47.223584 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.223446 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-metrics-client-certs" pod "43ccba25-b0ea-47c8-b664-105feea298a3" (UID: "43ccba25-b0ea-47c8-b664-105feea298a3"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:47.232941 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.232918 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-web-config" pod "43ccba25-b0ea-47c8-b664-105feea298a3" (UID: "43ccba25-b0ea-47c8-b664-105feea298a3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:47.251958 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.251909 5882 scope.go:122] "RemoveContainer" containerID="cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c" Oct 30 02:41:47.252320 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:47.252298 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c\": container with ID starting with cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c not found: ID does not exist" containerID="cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c" Oct 30 02:41:47.252414 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.252333 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c"} err="failed to get container status \"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c\": rpc error: code = NotFound desc = could not find container \"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c\": container with ID starting with cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c not found: ID does not exist" Oct 30 02:41:47.252414 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.252380 5882 scope.go:122] "RemoveContainer" containerID="d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8" Oct 30 02:41:47.252780 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:47.252750 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8\": container with ID starting with d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8 not found: ID does not exist" containerID="d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8" Oct 30 02:41:47.252871 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.252787 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8"} err="failed to get container status \"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8\": rpc error: code = NotFound desc = could not find container \"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8\": container with ID starting with d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8 not found: ID does not exist" Oct 30 02:41:47.252871 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.252810 5882 scope.go:122] "RemoveContainer" containerID="871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae" Oct 30 02:41:47.253183 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:47.253155 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae\": container with ID starting with 871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae not found: ID does not exist" containerID="871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae" Oct 30 02:41:47.253269 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.253189 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae"} err="failed to get container status \"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae\": rpc error: code = NotFound desc = could not find container \"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae\": container with ID starting with 871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae not found: ID does not exist" Oct 30 02:41:47.253269 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.253205 5882 scope.go:122] "RemoveContainer" containerID="87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e" Oct 30 02:41:47.254092 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:47.254069 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e\": container with ID starting with 87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e not found: ID does not exist" containerID="87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e" Oct 30 02:41:47.254185 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.254100 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e"} err="failed to get container status \"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e\": rpc error: code = NotFound desc = could not find container \"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e\": container with ID starting with 87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e not found: ID does not exist" Oct 30 02:41:47.254185 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.254118 5882 scope.go:122] "RemoveContainer" containerID="a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd" Oct 30 02:41:47.254400 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:47.254380 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd\": container with ID starting with a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd not found: ID does not exist" containerID="a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd" Oct 30 02:41:47.254457 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.254406 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd"} err="failed to get container status \"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd\": rpc error: code = NotFound desc = could not find container \"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd\": container with ID starting with a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd not found: ID does not exist" Oct 30 02:41:47.254457 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.254426 5882 scope.go:122] "RemoveContainer" containerID="b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757" Oct 30 02:41:47.254738 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:47.254668 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757\": container with ID starting with b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757 not found: ID does not exist" containerID="b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757" Oct 30 02:41:47.254738 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.254700 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757"} err="failed to get container status \"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757\": rpc error: code = NotFound desc = could not find container \"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757\": container with ID starting with b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757 not found: ID does not exist" Oct 30 02:41:47.254738 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.254721 5882 scope.go:122] "RemoveContainer" containerID="53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb" Oct 30 02:41:47.255281 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:47.255152 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb\": container with ID starting with 53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb not found: ID does not exist" containerID="53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb" Oct 30 02:41:47.255281 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.255177 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb"} err="failed to get container status \"53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb\": rpc error: code = NotFound desc = could not find container \"53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb\": container with ID starting with 53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb not found: ID does not exist" Oct 30 02:41:47.255281 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.255197 5882 scope.go:122] "RemoveContainer" containerID="cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c" Oct 30 02:41:47.255517 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.255495 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c"} err="failed to get container status \"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c\": rpc error: code = NotFound desc = could not find container \"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c\": container with ID starting with cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c not found: ID does not exist" Oct 30 02:41:47.255634 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.255517 5882 scope.go:122] "RemoveContainer" containerID="d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8" Oct 30 02:41:47.255804 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.255784 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8"} err="failed to get container status \"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8\": rpc error: code = NotFound desc = could not find container \"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8\": container with ID starting with d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8 not found: ID does not exist" Oct 30 02:41:47.255804 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.255802 5882 scope.go:122] "RemoveContainer" containerID="871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae" Oct 30 02:41:47.256056 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.256036 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae"} err="failed to get container status \"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae\": rpc error: code = NotFound desc = could not find container \"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae\": container with ID starting with 871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae not found: ID does not exist" Oct 30 02:41:47.256056 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.256054 5882 scope.go:122] "RemoveContainer" containerID="87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e" Oct 30 02:41:47.256305 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.256285 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e"} err="failed to get container status \"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e\": rpc error: code = NotFound desc = could not find container \"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e\": container with ID starting with 87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e not found: ID does not exist" Oct 30 02:41:47.256305 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.256304 5882 scope.go:122] "RemoveContainer" containerID="a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd" Oct 30 02:41:47.256558 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.256534 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd"} err="failed to get container status \"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd\": rpc error: code = NotFound desc = could not find container \"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd\": container with ID starting with a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd not found: ID does not exist" Oct 30 02:41:47.256597 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.256561 5882 scope.go:122] "RemoveContainer" containerID="b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757" Oct 30 02:41:47.256902 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.256859 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757"} err="failed to get container status \"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757\": rpc error: code = NotFound desc = could not find container \"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757\": container with ID starting with b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757 not found: ID does not exist" Oct 30 02:41:47.256902 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.256877 5882 scope.go:122] "RemoveContainer" containerID="53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb" Oct 30 02:41:47.257097 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.257082 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb"} err="failed to get container status \"53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb\": rpc error: code = NotFound desc = could not find container \"53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb\": container with ID starting with 53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb not found: ID does not exist" Oct 30 02:41:47.257138 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.257098 5882 scope.go:122] "RemoveContainer" containerID="cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c" Oct 30 02:41:47.257315 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.257297 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c"} err="failed to get container status \"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c\": rpc error: code = NotFound desc = could not find container \"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c\": container with ID starting with cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c not found: ID does not exist" Oct 30 02:41:47.257315 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.257314 5882 scope.go:122] "RemoveContainer" containerID="d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8" Oct 30 02:41:47.257526 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.257505 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8"} err="failed to get container status \"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8\": rpc error: code = NotFound desc = could not find container \"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8\": container with ID starting with d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8 not found: ID does not exist" Oct 30 02:41:47.257526 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.257525 5882 scope.go:122] "RemoveContainer" containerID="871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae" Oct 30 02:41:47.257877 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.257853 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae"} err="failed to get container status \"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae\": rpc error: code = NotFound desc = could not find container \"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae\": container with ID starting with 871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae not found: ID does not exist" Oct 30 02:41:47.257951 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.257879 5882 scope.go:122] "RemoveContainer" containerID="87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e" Oct 30 02:41:47.258146 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.258125 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e"} err="failed to get container status \"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e\": rpc error: code = NotFound desc = could not find container \"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e\": container with ID starting with 87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e not found: ID does not exist" Oct 30 02:41:47.258206 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.258155 5882 scope.go:122] "RemoveContainer" containerID="a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd" Oct 30 02:41:47.258709 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.258595 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd"} err="failed to get container status \"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd\": rpc error: code = NotFound desc = could not find container \"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd\": container with ID starting with a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd not found: ID does not exist" Oct 30 02:41:47.258709 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.258635 5882 scope.go:122] "RemoveContainer" containerID="b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757" Oct 30 02:41:47.258933 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.258904 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757"} err="failed to get container status \"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757\": rpc error: code = NotFound desc = could not find container \"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757\": container with ID starting with b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757 not found: ID does not exist" Oct 30 02:41:47.259050 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.258934 5882 scope.go:122] "RemoveContainer" containerID="53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb" Oct 30 02:41:47.259228 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.259205 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb"} err="failed to get container status \"53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb\": rpc error: code = NotFound desc = could not find container \"53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb\": container with ID starting with 53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb not found: ID does not exist" Oct 30 02:41:47.259282 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.259229 5882 scope.go:122] "RemoveContainer" containerID="cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c" Oct 30 02:41:47.259507 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.259478 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c"} err="failed to get container status \"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c\": rpc error: code = NotFound desc = could not find container \"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c\": container with ID starting with cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c not found: ID does not exist" Oct 30 02:41:47.259507 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.259506 5882 scope.go:122] "RemoveContainer" containerID="d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8" Oct 30 02:41:47.259834 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.259814 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8"} err="failed to get container status \"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8\": rpc error: code = NotFound desc = could not find container \"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8\": container with ID starting with d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8 not found: ID does not exist" Oct 30 02:41:47.259916 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.259835 5882 scope.go:122] "RemoveContainer" containerID="871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae" Oct 30 02:41:47.260129 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.260090 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae"} err="failed to get container status \"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae\": rpc error: code = NotFound desc = could not find container \"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae\": container with ID starting with 871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae not found: ID does not exist" Oct 30 02:41:47.260197 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.260147 5882 scope.go:122] "RemoveContainer" containerID="87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e" Oct 30 02:41:47.260485 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.260466 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e"} err="failed to get container status \"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e\": rpc error: code = NotFound desc = could not find container \"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e\": container with ID starting with 87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e not found: ID does not exist" Oct 30 02:41:47.260547 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.260485 5882 scope.go:122] "RemoveContainer" containerID="a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd" Oct 30 02:41:47.260778 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.260748 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd"} err="failed to get container status \"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd\": rpc error: code = NotFound desc = could not find container \"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd\": container with ID starting with a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd not found: ID does not exist" Oct 30 02:41:47.260778 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.260774 5882 scope.go:122] "RemoveContainer" containerID="b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757" Oct 30 02:41:47.261031 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.261012 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757"} err="failed to get container status \"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757\": rpc error: code = NotFound desc = could not find container \"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757\": container with ID starting with b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757 not found: ID does not exist" Oct 30 02:41:47.261100 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.261031 5882 scope.go:122] "RemoveContainer" containerID="53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb" Oct 30 02:41:47.261321 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.261300 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb"} err="failed to get container status \"53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb\": rpc error: code = NotFound desc = could not find container \"53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb\": container with ID starting with 53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb not found: ID does not exist" Oct 30 02:41:47.261388 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.261322 5882 scope.go:122] "RemoveContainer" containerID="cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c" Oct 30 02:41:47.261570 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.261549 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c"} err="failed to get container status \"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c\": rpc error: code = NotFound desc = could not find container \"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c\": container with ID starting with cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c not found: ID does not exist" Oct 30 02:41:47.261659 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.261572 5882 scope.go:122] "RemoveContainer" containerID="d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8" Oct 30 02:41:47.261801 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.261782 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8"} err="failed to get container status \"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8\": rpc error: code = NotFound desc = could not find container \"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8\": container with ID starting with d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8 not found: ID does not exist" Oct 30 02:41:47.261863 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.261801 5882 scope.go:122] "RemoveContainer" containerID="871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae" Oct 30 02:41:47.262046 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.262025 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae"} err="failed to get container status \"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae\": rpc error: code = NotFound desc = could not find container \"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae\": container with ID starting with 871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae not found: ID does not exist" Oct 30 02:41:47.262115 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.262046 5882 scope.go:122] "RemoveContainer" containerID="87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e" Oct 30 02:41:47.262275 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.262256 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e"} err="failed to get container status \"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e\": rpc error: code = NotFound desc = could not find container \"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e\": container with ID starting with 87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e not found: ID does not exist" Oct 30 02:41:47.262345 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.262276 5882 scope.go:122] "RemoveContainer" containerID="a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd" Oct 30 02:41:47.262516 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.262495 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd"} err="failed to get container status \"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd\": rpc error: code = NotFound desc = could not find container \"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd\": container with ID starting with a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd not found: ID does not exist" Oct 30 02:41:47.262652 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.262516 5882 scope.go:122] "RemoveContainer" containerID="b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757" Oct 30 02:41:47.262771 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.262753 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757"} err="failed to get container status \"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757\": rpc error: code = NotFound desc = could not find container \"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757\": container with ID starting with b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757 not found: ID does not exist" Oct 30 02:41:47.262833 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.262771 5882 scope.go:122] "RemoveContainer" containerID="53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb" Oct 30 02:41:47.263026 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.262996 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb"} err="failed to get container status \"53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb\": rpc error: code = NotFound desc = could not find container \"53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb\": container with ID starting with 53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb not found: ID does not exist" Oct 30 02:41:47.263026 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.263021 5882 scope.go:122] "RemoveContainer" containerID="cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c" Oct 30 02:41:47.263320 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.263295 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c"} err="failed to get container status \"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c\": rpc error: code = NotFound desc = could not find container \"cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c\": container with ID starting with cc3d4191fdb942b968fe7f9cce30e4c9f34a61c98bf00a484e7467b8be17825c not found: ID does not exist" Oct 30 02:41:47.263320 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.263314 5882 scope.go:122] "RemoveContainer" containerID="d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8" Oct 30 02:41:47.263595 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.263568 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8"} err="failed to get container status \"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8\": rpc error: code = NotFound desc = could not find container \"d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8\": container with ID starting with d3517efeb98b7666cc45f060d9f168c07b1490999ccfc26eed34c0f946016ce8 not found: ID does not exist" Oct 30 02:41:47.263595 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.263590 5882 scope.go:122] "RemoveContainer" containerID="871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae" Oct 30 02:41:47.263888 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.263854 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae"} err="failed to get container status \"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae\": rpc error: code = NotFound desc = could not find container \"871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae\": container with ID starting with 871823274e9fe38a67cb2c87b386f93f24ec4aab685d731df7006064690725ae not found: ID does not exist" Oct 30 02:41:47.263888 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.263880 5882 scope.go:122] "RemoveContainer" containerID="87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e" Oct 30 02:41:47.264130 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.264111 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e"} err="failed to get container status \"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e\": rpc error: code = NotFound desc = could not find container \"87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e\": container with ID starting with 87628aa57c2c0b88cc4ad52683a60b8ee52e74fd59b37d1427e552149fa51d3e not found: ID does not exist" Oct 30 02:41:47.264130 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.264129 5882 scope.go:122] "RemoveContainer" containerID="a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd" Oct 30 02:41:47.264359 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.264342 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd"} err="failed to get container status \"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd\": rpc error: code = NotFound desc = could not find container \"a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd\": container with ID starting with a804f4c4a98c4c4216c27c43347312317353e4c0308ae781edcf700a5680b6fd not found: ID does not exist" Oct 30 02:41:47.264359 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.264359 5882 scope.go:122] "RemoveContainer" containerID="b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757" Oct 30 02:41:47.264580 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.264561 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757"} err="failed to get container status \"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757\": rpc error: code = NotFound desc = could not find container \"b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757\": container with ID starting with b469cc8cd525db87fcc82cf1e66c0992f9708c345c56baf93c42b80afa2dc757 not found: ID does not exist" Oct 30 02:41:47.264672 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.264580 5882 scope.go:122] "RemoveContainer" containerID="53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb" Oct 30 02:41:47.264898 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.264873 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb"} err="failed to get container status \"53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb\": rpc error: code = NotFound desc = could not find container \"53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb\": container with ID starting with 53ee86f1cb6832c530087e52a8b4fd8173ba7e935f13f34a8a5599bbbd8b16eb not found: ID does not exist" Oct 30 02:41:47.316499 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.316462 5882 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.316499 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.316500 5882 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.316725 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.316517 5882 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-configmap-metrics-client-ca\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.316725 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.316533 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.316725 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.316550 5882 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43ccba25-b0ea-47c8-b664-105feea298a3-config-out\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.316725 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.316562 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.316725 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.316575 5882 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-web-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.316725 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.316588 5882 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-grpc-tls\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.316725 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.316602 5882 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-kube-rbac-proxy\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.316725 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.316629 5882 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.316725 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.316645 5882 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.316725 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.316662 5882 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.316725 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.316677 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.316725 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.316691 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl9n4\" (UniqueName: \"kubernetes.io/projected/43ccba25-b0ea-47c8-b664-105feea298a3-kube-api-access-pl9n4\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.316725 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.316705 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.316725 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.316720 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/43ccba25-b0ea-47c8-b664-105feea298a3-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.317264 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.316735 5882 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43ccba25-b0ea-47c8-b664-105feea298a3-tls-assets\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.317264 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.316748 5882 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/43ccba25-b0ea-47c8-b664-105feea298a3-secret-metrics-client-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.515440 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.515405 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 02:41:47.526836 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.526808 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 02:41:47.552477 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.552450 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 02:41:47.552834 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.552809 5882 cpu_manager.go:414] "containerMap: removing container" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="thanos-sidecar" Oct 30 02:41:47.552834 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.552829 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="thanos-sidecar" Oct 30 02:41:47.553004 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.552853 5882 cpu_manager.go:414] "containerMap: removing container" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="kube-rbac-proxy" Oct 30 02:41:47.553004 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.552862 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="kube-rbac-proxy" Oct 30 02:41:47.553004 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.552873 5882 cpu_manager.go:414] "containerMap: removing container" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="init-config-reloader" Oct 30 02:41:47.553004 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.552881 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="init-config-reloader" Oct 30 02:41:47.553004 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.552913 5882 cpu_manager.go:414] "containerMap: removing container" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="kube-rbac-proxy-web" Oct 30 02:41:47.553004 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.552924 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="kube-rbac-proxy-web" Oct 30 02:41:47.553004 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.552940 5882 cpu_manager.go:414] "containerMap: removing container" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="config-reloader" Oct 30 02:41:47.553004 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.552947 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="config-reloader" Oct 30 02:41:47.553004 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.552961 5882 cpu_manager.go:414] "containerMap: removing container" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="kube-rbac-proxy-thanos" Oct 30 02:41:47.553004 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.552969 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="kube-rbac-proxy-thanos" Oct 30 02:41:47.553004 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.552978 5882 cpu_manager.go:414] "containerMap: removing container" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="prometheus" Oct 30 02:41:47.553004 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.552985 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="prometheus" Oct 30 02:41:47.553488 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.553068 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="config-reloader" Oct 30 02:41:47.553488 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.553093 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="kube-rbac-proxy-thanos" Oct 30 02:41:47.553488 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.553110 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="prometheus" Oct 30 02:41:47.553488 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.553119 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="thanos-sidecar" Oct 30 02:41:47.553488 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.553126 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="kube-rbac-proxy" Oct 30 02:41:47.553488 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.553132 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="init-config-reloader" Oct 30 02:41:47.553488 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.553140 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" containerName="kube-rbac-proxy-web" Oct 30 02:41:47.584234 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.584201 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 02:41:47.584384 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.584271 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.600276 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.600249 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 02:41:47.600412 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.600250 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 02:41:47.600412 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.600325 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 02:41:47.600412 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.600370 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 02:41:47.601181 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.601156 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 02:41:47.601304 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.601193 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 02:41:47.601304 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.601210 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 02:41:47.601304 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.601250 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 02:41:47.601304 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.601288 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 02:41:47.601520 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.601419 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 02:41:47.601520 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.601445 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-q4z4m\"" Oct 30 02:41:47.601520 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.601498 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-96il72fi4oust\"" Oct 30 02:41:47.601665 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.601551 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 02:41:47.601665 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.601551 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 02:41:47.613594 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.613576 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 02:41:47.613594 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.613588 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 02:41:47.721339 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.721310 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.721526 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.721357 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-config\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.721526 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.721383 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.721526 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.721511 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.721715 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.721576 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.721715 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.721637 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.721795 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.721745 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.721795 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.721784 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.721882 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.721854 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.721937 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.721902 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.721977 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.721931 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.722036 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.722014 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.722088 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.722058 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.722127 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.722119 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.722175 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.722138 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.722175 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.722153 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnt8g\" (UniqueName: \"kubernetes.io/projected/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-kube-api-access-pnt8g\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.722256 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.722205 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.722256 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.722231 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.722330 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.722256 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.722372 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.722340 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.823795 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.823755 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-config\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.823987 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.823806 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.823987 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.823855 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.823987 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.823908 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.823987 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.823935 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.823987 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.823980 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.824246 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.824011 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.824246 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.824043 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.824246 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.824066 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.824246 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.824108 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.824246 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.824143 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.824246 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.824167 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.824246 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.824192 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.824246 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.824216 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.824246 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.824239 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnt8g\" (UniqueName: \"kubernetes.io/projected/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-kube-api-access-pnt8g\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.824670 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.824284 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.824670 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.824308 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.824670 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.824338 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.824670 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.824396 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.824887 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.824762 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.825336 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.825311 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.826822 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.825566 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.826822 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.825953 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.826822 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.826445 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.827430 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.827405 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.827539 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.827464 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.828040 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.827732 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-config\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.828040 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.827883 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.828040 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.827958 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.828249 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.828174 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.828249 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.828182 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.829000 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.828559 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.829000 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.828915 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.829214 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.829172 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.830049 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.830021 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.830049 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.830036 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.830726 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.830699 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.830820 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.830765 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.830820 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.830770 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:47.847950 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:47.847932 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnt8g\" (UniqueName: \"kubernetes.io/projected/7c79c37b-c9d5-41e1-b13a-4b39bfd46178-kube-api-access-pnt8g\") pod \"prometheus-k8s-0\" (UID: \"7c79c37b-c9d5-41e1-b13a-4b39bfd46178\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:48.010801 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:48.010512 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ccba25-b0ea-47c8-b664-105feea298a3" path="/var/lib/kubelet/pods/43ccba25-b0ea-47c8-b664-105feea298a3/volumes" Oct 30 02:41:48.013734 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:48.013705 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:48.364690 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:48.364595 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 02:41:48.367314 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:41:48.367282 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c79c37b_c9d5_41e1_b13a_4b39bfd46178.slice/crio-4554498d2a53beea345b940ab7a6196bcbc917f5ec5f4bad1c5dd615f9e4ef46 WatchSource:0}: Error finding container 4554498d2a53beea345b940ab7a6196bcbc917f5ec5f4bad1c5dd615f9e4ef46: Status 404 returned error can't find the container with id 4554498d2a53beea345b940ab7a6196bcbc917f5ec5f4bad1c5dd615f9e4ef46 Oct 30 02:41:49.177532 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:49.177500 5882 generic.go:356] "Generic (PLEG): container finished" podID="7c79c37b-c9d5-41e1-b13a-4b39bfd46178" containerID="192b061654fc68f869013fde702a94f2428ad8c9ec836decd613921f63673fdb" exitCode=0 Oct 30 02:41:49.177734 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:49.177591 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c79c37b-c9d5-41e1-b13a-4b39bfd46178","Type":"ContainerDied","Data":"192b061654fc68f869013fde702a94f2428ad8c9ec836decd613921f63673fdb"} Oct 30 02:41:49.177734 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:49.177685 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c79c37b-c9d5-41e1-b13a-4b39bfd46178","Type":"ContainerStarted","Data":"4554498d2a53beea345b940ab7a6196bcbc917f5ec5f4bad1c5dd615f9e4ef46"} Oct 30 02:41:49.179831 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:49.179806 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" event={"ID":"c426339e-5bc3-45ec-a994-7b205c2cdc44","Type":"ContainerStarted","Data":"6c93d64462b9c06733036dbed2fb0a60d6d1a28c99b64e8936ee2bbb6ce90009"} Oct 30 02:41:49.179909 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:49.179853 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" event={"ID":"c426339e-5bc3-45ec-a994-7b205c2cdc44","Type":"ContainerStarted","Data":"ae07855199c392d0dfbf9eb5570c6fb042b30d0771037c832ebdc12a35351e2f"} Oct 30 02:41:49.179909 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:49.179873 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" event={"ID":"c426339e-5bc3-45ec-a994-7b205c2cdc44","Type":"ContainerStarted","Data":"1f3402a280288ea9a4eb8f8bc50b620f929bfd41d669f5b53fc96697a4bc17dd"} Oct 30 02:41:49.280033 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:49.279833 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-84cd9b5f86-vtnlf" podStartSLOduration=2.121519148 podStartE2EDuration="4.279815883s" totalImagesPullingTime="2.158296735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:49.279445528 +0000 UTC m=+21.393760506" watchObservedRunningTime="2026-10-30 02:41:49.279815883 +0000 UTC m=+21.394130868" Oct 30 02:41:50.186899 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:50.186807 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c79c37b-c9d5-41e1-b13a-4b39bfd46178","Type":"ContainerStarted","Data":"7bbd258892ccf62dcf2c81352ec6a400307f0c5083cdd03db39da6e9a242e999"} Oct 30 02:41:50.186899 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:50.186887 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c79c37b-c9d5-41e1-b13a-4b39bfd46178","Type":"ContainerStarted","Data":"389d20ecfb2eb37b4d76c3ec4a86717491b357ba4a0a9903c05f9502afb98ca7"} Oct 30 02:41:50.187376 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:50.186918 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c79c37b-c9d5-41e1-b13a-4b39bfd46178","Type":"ContainerStarted","Data":"a84fd524c0297b788a470f57b2ff23475c19baf7e811a2d69d7ae225c5100c3f"} Oct 30 02:41:50.187376 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:50.186953 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c79c37b-c9d5-41e1-b13a-4b39bfd46178","Type":"ContainerStarted","Data":"1db802154735eeaaaa4ad11a1ecf585179f13d55e3e48d1358d34091c4611e8b"} Oct 30 02:41:50.187376 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:50.186981 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c79c37b-c9d5-41e1-b13a-4b39bfd46178","Type":"ContainerStarted","Data":"991d5adf0ae2ecfce7b63aebec00c2d3ecd6a3cc0ca22d4a74d4532286270e74"} Oct 30 02:41:50.187376 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:50.187006 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c79c37b-c9d5-41e1-b13a-4b39bfd46178","Type":"ContainerStarted","Data":"6263465a7530ebd68bdbdfba61ae0eadfd7b417fc0538d4a9be4f126a657c414"} Oct 30 02:41:50.253126 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:50.253078 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.230160048 podStartE2EDuration="3.253062347s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.902299ms" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:50.252321966 +0000 UTC m=+22.366636958" watchObservedRunningTime="2026-10-30 02:41:50.253062347 +0000 UTC m=+22.367377319" Oct 30 02:41:53.014409 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:53.014360 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:41:53.942122 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:41:53.942080 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:41:54.041131 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.041090 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-647f7b9c85-ldfjx" podUID="49f725c1-f41a-403d-b729-a170fc1fe687" containerName="console" containerID="cri-o://9f0c8ff7cee660c75133b779bbf875fb9a1b5daf1d5305e937b6d63788e57071" gracePeriod=15 Oct 30 02:41:54.203809 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.203734 5882 generic.go:356] "Generic (PLEG): container finished" podID="49f725c1-f41a-403d-b729-a170fc1fe687" containerID="9f0c8ff7cee660c75133b779bbf875fb9a1b5daf1d5305e937b6d63788e57071" exitCode=0 Oct 30 02:41:54.203809 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.203782 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-647f7b9c85-ldfjx" event={"ID":"49f725c1-f41a-403d-b729-a170fc1fe687","Type":"ContainerDied","Data":"9f0c8ff7cee660c75133b779bbf875fb9a1b5daf1d5305e937b6d63788e57071"} Oct 30 02:41:54.288309 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.288283 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:54.401721 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.401688 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle\") pod \"49f725c1-f41a-403d-b729-a170fc1fe687\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " Oct 30 02:41:54.401886 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.401736 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config\") pod \"49f725c1-f41a-403d-b729-a170fc1fe687\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " Oct 30 02:41:54.401886 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.401852 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg\" (UniqueName: \"kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg\") pod \"49f725c1-f41a-403d-b729-a170fc1fe687\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " Oct 30 02:41:54.402038 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.401895 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca\") pod \"49f725c1-f41a-403d-b729-a170fc1fe687\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " Oct 30 02:41:54.402038 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.401933 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert\") pod \"49f725c1-f41a-403d-b729-a170fc1fe687\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " Oct 30 02:41:54.402038 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.401997 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config\") pod \"49f725c1-f41a-403d-b729-a170fc1fe687\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " Oct 30 02:41:54.402190 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.402050 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config" pod "49f725c1-f41a-403d-b729-a170fc1fe687" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:54.402190 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.402099 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert\") pod \"49f725c1-f41a-403d-b729-a170fc1fe687\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " Oct 30 02:41:54.402190 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.402136 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49f725c1-f41a-403d-b729-a170fc1fe687-tmp\" (UniqueName: \"kubernetes.io/empty-dir/49f725c1-f41a-403d-b729-a170fc1fe687-tmp\") pod \"49f725c1-f41a-403d-b729-a170fc1fe687\" (UID: \"49f725c1-f41a-403d-b729-a170fc1fe687\") " Oct 30 02:41:54.402327 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.402153 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle" pod "49f725c1-f41a-403d-b729-a170fc1fe687" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:54.402380 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.402354 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert" pod "49f725c1-f41a-403d-b729-a170fc1fe687" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:54.402380 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.402364 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca" pod "49f725c1-f41a-403d-b729-a170fc1fe687" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:54.402574 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.402549 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f725c1-f41a-403d-b729-a170fc1fe687-tmp" pod "49f725c1-f41a-403d-b729-a170fc1fe687" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:54.402675 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.402484 5882 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-trusted-ca-bundle\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:54.402675 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.402598 5882 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-console-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:54.402675 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.402609 5882 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-service-ca\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:54.402675 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.402638 5882 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49f725c1-f41a-403d-b729-a170fc1fe687-oauth-serving-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:54.404301 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.404279 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert" pod "49f725c1-f41a-403d-b729-a170fc1fe687" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:54.404561 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.404541 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg" pod "49f725c1-f41a-403d-b729-a170fc1fe687" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687"). InnerVolumeSpecName "kube-api-access-nl2vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:54.404611 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.404558 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config" pod "49f725c1-f41a-403d-b729-a170fc1fe687" (UID: "49f725c1-f41a-403d-b729-a170fc1fe687"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:54.503192 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.503153 5882 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-oauth-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:54.503192 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.503188 5882 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49f725c1-f41a-403d-b729-a170fc1fe687-console-serving-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:54.503383 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.503203 5882 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49f725c1-f41a-403d-b729-a170fc1fe687-tmp\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:54.503383 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:54.503217 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl2vg\" (UniqueName: \"kubernetes.io/projected/49f725c1-f41a-403d-b729-a170fc1fe687-kube-api-access-nl2vg\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:41:55.209761 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:55.209721 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-647f7b9c85-ldfjx" event={"ID":"49f725c1-f41a-403d-b729-a170fc1fe687","Type":"ContainerDied","Data":"719b462c9bbf11e8fa2bba324a94d6311f2196d28e35106f50f564ce80743ec1"} Oct 30 02:41:55.209761 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:55.209757 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-647f7b9c85-ldfjx" Oct 30 02:41:55.210220 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:55.209792 5882 scope.go:122] "RemoveContainer" containerID="9f0c8ff7cee660c75133b779bbf875fb9a1b5daf1d5305e937b6d63788e57071" Oct 30 02:41:55.262291 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:55.262266 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-647f7b9c85-ldfjx"] Oct 30 02:41:55.273295 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:55.273266 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-647f7b9c85-ldfjx"] Oct 30 02:41:56.009136 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:56.009107 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49f725c1-f41a-403d-b729-a170fc1fe687" path="/var/lib/kubelet/pods/49f725c1-f41a-403d-b729-a170fc1fe687/volumes" Oct 30 02:41:57.174297 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:41:57.174257 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t765c" Oct 30 02:42:03.678925 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:42:03.678890 5882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:42:03.696069 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:42:03.696043 5882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:42:04.262526 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:42:04.262497 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:42:23.941868 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:42:23.941827 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:42:53.941661 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:42:53.941607 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:43:23.942276 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:43:23.942230 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:43:53.942312 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:43:53.942275 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:44:23.942239 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:44:23.942199 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:44:53.942574 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:44:53.942530 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:45:23.942957 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:45:23.942919 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:45:53.942313 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:45:53.942275 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:46:23.942233 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:46:23.942196 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:46:28.033913 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:46:28.033886 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 02:46:28.034471 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:46:28.033987 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 02:46:53.942093 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:46:53.942044 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:47:23.942044 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:47:23.942005 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:47:53.942044 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:47:53.942006 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:48:23.941427 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:48:23.941388 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:48:53.942538 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:48:53.942495 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:49:23.942186 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:49:23.942108 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:49:53.942605 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:49:53.942567 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:50:23.942435 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:50:23.942396 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:50:53.943351 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:50:53.943312 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:51:23.942751 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:51:23.942716 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:51:28.057332 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:51:28.057302 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 02:51:28.058940 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:51:28.058922 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 02:51:53.942050 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:51:53.942011 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:52:23.941655 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:52:23.941559 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:52:53.941994 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:52:53.941954 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:53:23.943011 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:53:23.942969 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:53:53.942318 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:53:53.942280 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:54:23.941916 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:54:23.941880 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:54:53.942412 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:54:53.942377 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:55:23.942327 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:55:23.942246 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:55:53.942489 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:55:53.942451 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:56:18.001147 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.001115 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-j6x7b"] Oct 30 02:56:18.001811 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.001516 5882 cpu_manager.go:414] "containerMap: removing container" podUID="49f725c1-f41a-403d-b729-a170fc1fe687" containerName="console" Oct 30 02:56:18.001811 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.001531 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f725c1-f41a-403d-b729-a170fc1fe687" containerName="console" Oct 30 02:56:18.001811 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.001629 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f725c1-f41a-403d-b729-a170fc1fe687" containerName="console" Oct 30 02:56:18.006257 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.006233 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-j6x7b" Oct 30 02:56:18.041677 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.041648 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-j6x7b"] Oct 30 02:56:18.054533 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.054495 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-admission-webhook-service-cert\"" Oct 30 02:56:18.054533 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.054522 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-admission-webhook-dockercfg-t48zq\"" Oct 30 02:56:18.098144 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.098103 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ece5a90-9c20-4d5d-9e00-c2ae6fe5fcb1-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-7795c9f77c-j6x7b\" (UID: \"3ece5a90-9c20-4d5d-9e00-c2ae6fe5fcb1\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-j6x7b" Oct 30 02:56:18.098325 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.098237 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ece5a90-9c20-4d5d-9e00-c2ae6fe5fcb1-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-7795c9f77c-j6x7b\" (UID: \"3ece5a90-9c20-4d5d-9e00-c2ae6fe5fcb1\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-j6x7b" Oct 30 02:56:18.198913 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.198880 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ece5a90-9c20-4d5d-9e00-c2ae6fe5fcb1-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-7795c9f77c-j6x7b\" (UID: \"3ece5a90-9c20-4d5d-9e00-c2ae6fe5fcb1\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-j6x7b" Oct 30 02:56:18.199089 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.198944 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ece5a90-9c20-4d5d-9e00-c2ae6fe5fcb1-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-7795c9f77c-j6x7b\" (UID: \"3ece5a90-9c20-4d5d-9e00-c2ae6fe5fcb1\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-j6x7b" Oct 30 02:56:18.201280 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.201256 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ece5a90-9c20-4d5d-9e00-c2ae6fe5fcb1-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-7795c9f77c-j6x7b\" (UID: \"3ece5a90-9c20-4d5d-9e00-c2ae6fe5fcb1\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-j6x7b" Oct 30 02:56:18.201396 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.201312 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ece5a90-9c20-4d5d-9e00-c2ae6fe5fcb1-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-7795c9f77c-j6x7b\" (UID: \"3ece5a90-9c20-4d5d-9e00-c2ae6fe5fcb1\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-j6x7b" Oct 30 02:56:18.320084 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.320001 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-j6x7b" Oct 30 02:56:18.641835 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.641743 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-j6x7b"] Oct 30 02:56:18.646218 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:56:18.646192 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ece5a90_9c20_4d5d_9e00_c2ae6fe5fcb1.slice/crio-f6ccd77b5e325ddcfbb372bbdbb2c97c7624748c2391227956c0d20cff839204 WatchSource:0}: Error finding container f6ccd77b5e325ddcfbb372bbdbb2c97c7624748c2391227956c0d20cff839204: Status 404 returned error can't find the container with id f6ccd77b5e325ddcfbb372bbdbb2c97c7624748c2391227956c0d20cff839204 Oct 30 02:56:18.649052 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.648911 5882 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:56:18.652959 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.652938 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/observability-operator-599c6f89d-wb29z"] Oct 30 02:56:18.657074 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.657056 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-wb29z" Oct 30 02:56:18.699210 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.699174 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"observability-operator-sa-dockercfg-vjhcx\"" Oct 30 02:56:18.699377 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.699333 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"observability-operator-tls\"" Oct 30 02:56:18.699572 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.699523 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-observability-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:56:18.699695 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.699679 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-observability-operator\"/\"kube-root-ca.crt\"" Oct 30 02:56:18.702003 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.701974 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"observability-operator-tls\" (UniqueName: \"kubernetes.io/secret/453bc4cc-791c-409f-ba52-58bc3796d66e-observability-operator-tls\") pod \"observability-operator-599c6f89d-wb29z\" (UID: \"453bc4cc-791c-409f-ba52-58bc3796d66e\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-wb29z" Oct 30 02:56:18.702082 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.702062 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmglt\" (UniqueName: \"kubernetes.io/projected/453bc4cc-791c-409f-ba52-58bc3796d66e-kube-api-access-xmglt\") pod \"observability-operator-599c6f89d-wb29z\" (UID: \"453bc4cc-791c-409f-ba52-58bc3796d66e\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-wb29z" Oct 30 02:56:18.716595 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.716573 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/observability-operator-599c6f89d-wb29z"] Oct 30 02:56:18.802816 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.802782 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"observability-operator-tls\" (UniqueName: \"kubernetes.io/secret/453bc4cc-791c-409f-ba52-58bc3796d66e-observability-operator-tls\") pod \"observability-operator-599c6f89d-wb29z\" (UID: \"453bc4cc-791c-409f-ba52-58bc3796d66e\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-wb29z" Oct 30 02:56:18.803018 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.802848 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmglt\" (UniqueName: \"kubernetes.io/projected/453bc4cc-791c-409f-ba52-58bc3796d66e-kube-api-access-xmglt\") pod \"observability-operator-599c6f89d-wb29z\" (UID: \"453bc4cc-791c-409f-ba52-58bc3796d66e\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-wb29z" Oct 30 02:56:18.805163 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.805141 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"observability-operator-tls\" (UniqueName: \"kubernetes.io/secret/453bc4cc-791c-409f-ba52-58bc3796d66e-observability-operator-tls\") pod \"observability-operator-599c6f89d-wb29z\" (UID: \"453bc4cc-791c-409f-ba52-58bc3796d66e\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-wb29z" Oct 30 02:56:18.831320 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.831289 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmglt\" (UniqueName: \"kubernetes.io/projected/453bc4cc-791c-409f-ba52-58bc3796d66e-kube-api-access-xmglt\") pod \"observability-operator-599c6f89d-wb29z\" (UID: \"453bc4cc-791c-409f-ba52-58bc3796d66e\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-wb29z" Oct 30 02:56:18.969440 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:18.969367 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-wb29z" Oct 30 02:56:19.301226 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.301188 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/observability-operator-599c6f89d-wb29z"] Oct 30 02:56:19.306855 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:56:19.306823 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453bc4cc_791c_409f_ba52_58bc3796d66e.slice/crio-817a3c03185ee3bd38b00efbf6625f8a4e98929e30e7468b7470c22434f5afa1 WatchSource:0}: Error finding container 817a3c03185ee3bd38b00efbf6625f8a4e98929e30e7468b7470c22434f5afa1: Status 404 returned error can't find the container with id 817a3c03185ee3bd38b00efbf6625f8a4e98929e30e7468b7470c22434f5afa1 Oct 30 02:56:19.321974 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.321941 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-wb29z" event={"ID":"453bc4cc-791c-409f-ba52-58bc3796d66e","Type":"ContainerStarted","Data":"817a3c03185ee3bd38b00efbf6625f8a4e98929e30e7468b7470c22434f5afa1"} Oct 30 02:56:19.323531 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.323480 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-j6x7b" event={"ID":"3ece5a90-9c20-4d5d-9e00-c2ae6fe5fcb1","Type":"ContainerStarted","Data":"f6ccd77b5e325ddcfbb372bbdbb2c97c7624748c2391227956c0d20cff839204"} Oct 30 02:56:19.388556 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.388521 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7"] Oct 30 02:56:19.391964 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.391943 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7" Oct 30 02:56:19.406804 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.406778 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/195847d4-d0c8-4005-baff-526262bfb16c-apiservice-cert\") pod \"perses-operator-589f965ff6-krqs7\" (UID: \"195847d4-d0c8-4005-baff-526262bfb16c\") " pod="openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7" Oct 30 02:56:19.406940 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.406872 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d2gj\" (UniqueName: \"kubernetes.io/projected/195847d4-d0c8-4005-baff-526262bfb16c-kube-api-access-6d2gj\") pod \"perses-operator-589f965ff6-krqs7\" (UID: \"195847d4-d0c8-4005-baff-526262bfb16c\") " pod="openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7" Oct 30 02:56:19.406940 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.406915 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195847d4-d0c8-4005-baff-526262bfb16c-openshift-service-ca\") pod \"perses-operator-589f965ff6-krqs7\" (UID: \"195847d4-d0c8-4005-baff-526262bfb16c\") " pod="openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7" Oct 30 02:56:19.407156 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.407128 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/195847d4-d0c8-4005-baff-526262bfb16c-webhook-cert\") pod \"perses-operator-589f965ff6-krqs7\" (UID: \"195847d4-d0c8-4005-baff-526262bfb16c\") " pod="openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7" Oct 30 02:56:19.417071 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.417040 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"perses-operator-service-cert\"" Oct 30 02:56:19.417071 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.417051 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"perses-operator-dockercfg-grns7\"" Oct 30 02:56:19.451088 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.451050 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7"] Oct 30 02:56:19.507924 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.507887 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d2gj\" (UniqueName: \"kubernetes.io/projected/195847d4-d0c8-4005-baff-526262bfb16c-kube-api-access-6d2gj\") pod \"perses-operator-589f965ff6-krqs7\" (UID: \"195847d4-d0c8-4005-baff-526262bfb16c\") " pod="openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7" Oct 30 02:56:19.508120 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.507937 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195847d4-d0c8-4005-baff-526262bfb16c-openshift-service-ca\") pod \"perses-operator-589f965ff6-krqs7\" (UID: \"195847d4-d0c8-4005-baff-526262bfb16c\") " pod="openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7" Oct 30 02:56:19.508120 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.507964 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/195847d4-d0c8-4005-baff-526262bfb16c-webhook-cert\") pod \"perses-operator-589f965ff6-krqs7\" (UID: \"195847d4-d0c8-4005-baff-526262bfb16c\") " pod="openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7" Oct 30 02:56:19.508120 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.508021 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/195847d4-d0c8-4005-baff-526262bfb16c-apiservice-cert\") pod \"perses-operator-589f965ff6-krqs7\" (UID: \"195847d4-d0c8-4005-baff-526262bfb16c\") " pod="openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7" Oct 30 02:56:19.508715 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.508688 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195847d4-d0c8-4005-baff-526262bfb16c-openshift-service-ca\") pod \"perses-operator-589f965ff6-krqs7\" (UID: \"195847d4-d0c8-4005-baff-526262bfb16c\") " pod="openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7" Oct 30 02:56:19.510668 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.510643 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/195847d4-d0c8-4005-baff-526262bfb16c-webhook-cert\") pod \"perses-operator-589f965ff6-krqs7\" (UID: \"195847d4-d0c8-4005-baff-526262bfb16c\") " pod="openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7" Oct 30 02:56:19.510774 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.510753 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/195847d4-d0c8-4005-baff-526262bfb16c-apiservice-cert\") pod \"perses-operator-589f965ff6-krqs7\" (UID: \"195847d4-d0c8-4005-baff-526262bfb16c\") " pod="openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7" Oct 30 02:56:19.548723 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.548695 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d2gj\" (UniqueName: \"kubernetes.io/projected/195847d4-d0c8-4005-baff-526262bfb16c-kube-api-access-6d2gj\") pod \"perses-operator-589f965ff6-krqs7\" (UID: \"195847d4-d0c8-4005-baff-526262bfb16c\") " pod="openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7" Oct 30 02:56:19.708269 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:19.708183 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7" Oct 30 02:56:20.040002 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:20.039965 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7"] Oct 30 02:56:20.044719 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:56:20.044687 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195847d4_d0c8_4005_baff_526262bfb16c.slice/crio-36cd5ebddf318ecdf82b419f8634f367d470b39c284cc27635cc7b4c33306923 WatchSource:0}: Error finding container 36cd5ebddf318ecdf82b419f8634f367d470b39c284cc27635cc7b4c33306923: Status 404 returned error can't find the container with id 36cd5ebddf318ecdf82b419f8634f367d470b39c284cc27635cc7b4c33306923 Oct 30 02:56:20.336242 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:20.336138 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7" event={"ID":"195847d4-d0c8-4005-baff-526262bfb16c","Type":"ContainerStarted","Data":"36cd5ebddf318ecdf82b419f8634f367d470b39c284cc27635cc7b4c33306923"} Oct 30 02:56:22.353295 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:22.353190 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-j6x7b" event={"ID":"3ece5a90-9c20-4d5d-9e00-c2ae6fe5fcb1","Type":"ContainerStarted","Data":"ee4f71b50ed67f24d7f016c45b2b912037c932d5a72a1ebde2378b405a45fad8"} Oct 30 02:56:22.355147 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:22.355114 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7" event={"ID":"195847d4-d0c8-4005-baff-526262bfb16c","Type":"ContainerStarted","Data":"36839b5d8a412a046ef5baeb3b05df6b614c45e6e0b3801009049c08ee8b85b4"} Oct 30 02:56:22.355285 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:22.355267 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7" Oct 30 02:56:22.443265 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:22.443211 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-j6x7b" podStartSLOduration=2.839628153 podStartE2EDuration="5.443198618s" totalImagesPullingTime="2.603570465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:22.431202884 +0000 UTC m=+894.545517858" watchObservedRunningTime="2026-10-30 02:56:22.443198618 +0000 UTC m=+894.557513590" Oct 30 02:56:22.505728 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:22.505650 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7" podStartSLOduration=1.50920535 podStartE2EDuration="3.50562939s" totalImagesPullingTime="1.99642404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:22.504163141 +0000 UTC m=+894.618478117" watchObservedRunningTime="2026-10-30 02:56:22.50562939 +0000 UTC m=+894.619944362" Oct 30 02:56:23.361420 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:23.361386 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-wb29z" event={"ID":"453bc4cc-791c-409f-ba52-58bc3796d66e","Type":"ContainerStarted","Data":"96cd9f82809dd8bf193e5e39ca3c3889d86ab2918da1fa52de5cf9e60d1e30ef"} Oct 30 02:56:23.361897 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:23.361480 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-wb29z" Oct 30 02:56:23.363407 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:23.363387 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-wb29z" Oct 30 02:56:23.443688 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:23.443642 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-wb29z" podStartSLOduration=1.9516235910000002 podStartE2EDuration="5.443627817s" totalImagesPullingTime="3.492004226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:23.431509677 +0000 UTC m=+895.545824652" watchObservedRunningTime="2026-10-30 02:56:23.443627817 +0000 UTC m=+895.557942783" Oct 30 02:56:23.942379 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:56:23.942346 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:56:28.085565 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:28.085536 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 02:56:28.086481 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:28.086461 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 02:56:33.363689 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:33.363653 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-cluster-observability-operator/perses-operator-589f965ff6-krqs7" Oct 30 02:56:45.741800 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:45.741708 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq"] Oct 30 02:56:45.745718 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:45.745689 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq" Oct 30 02:56:45.762106 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:45.762080 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:56:45.762220 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:45.762106 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"kube-root-ca.crt\"" Oct 30 02:56:45.773717 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:45.773694 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-webhook-service-cert\"" Oct 30 02:56:45.773832 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:45.773700 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-controller-manager-dockercfg-hdzf5\"" Oct 30 02:56:45.774663 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:45.774646 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq"] Oct 30 02:56:45.844096 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:45.844057 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7926992-9308-423f-bfdb-02c9824523e1-apiservice-cert\") pod \"opentelemetry-operator-webhook-787f9767b8-hf6jq\" (UID: \"e7926992-9308-423f-bfdb-02c9824523e1\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq" Oct 30 02:56:45.844096 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:45.844097 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7926992-9308-423f-bfdb-02c9824523e1-webhook-cert\") pod \"opentelemetry-operator-webhook-787f9767b8-hf6jq\" (UID: \"e7926992-9308-423f-bfdb-02c9824523e1\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq" Oct 30 02:56:45.844370 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:45.844137 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trf4j\" (UniqueName: \"kubernetes.io/projected/e7926992-9308-423f-bfdb-02c9824523e1-kube-api-access-trf4j\") pod \"opentelemetry-operator-webhook-787f9767b8-hf6jq\" (UID: \"e7926992-9308-423f-bfdb-02c9824523e1\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq" Oct 30 02:56:45.945481 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:45.945449 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7926992-9308-423f-bfdb-02c9824523e1-apiservice-cert\") pod \"opentelemetry-operator-webhook-787f9767b8-hf6jq\" (UID: \"e7926992-9308-423f-bfdb-02c9824523e1\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq" Oct 30 02:56:45.945690 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:45.945492 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7926992-9308-423f-bfdb-02c9824523e1-webhook-cert\") pod \"opentelemetry-operator-webhook-787f9767b8-hf6jq\" (UID: \"e7926992-9308-423f-bfdb-02c9824523e1\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq" Oct 30 02:56:45.945690 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:45.945560 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trf4j\" (UniqueName: \"kubernetes.io/projected/e7926992-9308-423f-bfdb-02c9824523e1-kube-api-access-trf4j\") pod \"opentelemetry-operator-webhook-787f9767b8-hf6jq\" (UID: \"e7926992-9308-423f-bfdb-02c9824523e1\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq" Oct 30 02:56:45.947990 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:45.947964 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7926992-9308-423f-bfdb-02c9824523e1-webhook-cert\") pod \"opentelemetry-operator-webhook-787f9767b8-hf6jq\" (UID: \"e7926992-9308-423f-bfdb-02c9824523e1\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq" Oct 30 02:56:45.948096 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:45.948008 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7926992-9308-423f-bfdb-02c9824523e1-apiservice-cert\") pod \"opentelemetry-operator-webhook-787f9767b8-hf6jq\" (UID: \"e7926992-9308-423f-bfdb-02c9824523e1\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq" Oct 30 02:56:45.974158 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:45.974131 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trf4j\" (UniqueName: \"kubernetes.io/projected/e7926992-9308-423f-bfdb-02c9824523e1-kube-api-access-trf4j\") pod \"opentelemetry-operator-webhook-787f9767b8-hf6jq\" (UID: \"e7926992-9308-423f-bfdb-02c9824523e1\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq" Oct 30 02:56:46.057687 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:46.057652 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq" Oct 30 02:56:46.374288 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:46.374198 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq"] Oct 30 02:56:46.377649 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:56:46.377606 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7926992_9308_423f_bfdb_02c9824523e1.slice/crio-1bbb74231f32d67c8bb13e8120652ff01a5da3506b4f2f83bd6eb3aa7ae2955b WatchSource:0}: Error finding container 1bbb74231f32d67c8bb13e8120652ff01a5da3506b4f2f83bd6eb3aa7ae2955b: Status 404 returned error can't find the container with id 1bbb74231f32d67c8bb13e8120652ff01a5da3506b4f2f83bd6eb3aa7ae2955b Oct 30 02:56:46.464514 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:46.464477 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq" event={"ID":"e7926992-9308-423f-bfdb-02c9824523e1","Type":"ContainerStarted","Data":"1bbb74231f32d67c8bb13e8120652ff01a5da3506b4f2f83bd6eb3aa7ae2955b"} Oct 30 02:56:48.475661 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:48.475608 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq" event={"ID":"e7926992-9308-423f-bfdb-02c9824523e1","Type":"ContainerStarted","Data":"4c287f458782b2c697bb1c318bdd076e1211d1d3f5207a5d1977640040cf90b4"} Oct 30 02:56:48.476146 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:48.475722 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq" Oct 30 02:56:48.520010 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:48.519982 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq"] Oct 30 02:56:48.556806 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:48.556752 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq" podStartSLOduration=2.118084498 podStartE2EDuration="3.556738489s" totalImagesPullingTime="1.438653991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:48.556098747 +0000 UTC m=+920.670413721" watchObservedRunningTime="2026-10-30 02:56:48.556738489 +0000 UTC m=+920.671053461" Oct 30 02:56:50.483377 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:50.483332 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq" podUID="e7926992-9308-423f-bfdb-02c9824523e1" containerName="webhook" containerID="cri-o://4c287f458782b2c697bb1c318bdd076e1211d1d3f5207a5d1977640040cf90b4" gracePeriod=30 Oct 30 02:56:50.506168 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:56:50.506132 5882 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7926992_9308_423f_bfdb_02c9824523e1.slice/crio-4c287f458782b2c697bb1c318bdd076e1211d1d3f5207a5d1977640040cf90b4.scope\": RecentStats: unable to find data in memory cache]" Oct 30 02:56:50.764248 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:50.764226 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq" Oct 30 02:56:50.890772 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:50.890742 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7926992-9308-423f-bfdb-02c9824523e1-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7926992-9308-423f-bfdb-02c9824523e1-webhook-cert\") pod \"e7926992-9308-423f-bfdb-02c9824523e1\" (UID: \"e7926992-9308-423f-bfdb-02c9824523e1\") " Oct 30 02:56:50.890966 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:50.890829 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7926992-9308-423f-bfdb-02c9824523e1-apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7926992-9308-423f-bfdb-02c9824523e1-apiservice-cert\") pod \"e7926992-9308-423f-bfdb-02c9824523e1\" (UID: \"e7926992-9308-423f-bfdb-02c9824523e1\") " Oct 30 02:56:50.890966 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:50.890892 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7926992-9308-423f-bfdb-02c9824523e1-kube-api-access-trf4j\" (UniqueName: \"kubernetes.io/projected/e7926992-9308-423f-bfdb-02c9824523e1-kube-api-access-trf4j\") pod \"e7926992-9308-423f-bfdb-02c9824523e1\" (UID: \"e7926992-9308-423f-bfdb-02c9824523e1\") " Oct 30 02:56:50.893052 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:50.893024 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7926992-9308-423f-bfdb-02c9824523e1-kube-api-access-trf4j" pod "e7926992-9308-423f-bfdb-02c9824523e1" (UID: "e7926992-9308-423f-bfdb-02c9824523e1"). InnerVolumeSpecName "kube-api-access-trf4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:56:50.893153 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:50.893045 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7926992-9308-423f-bfdb-02c9824523e1-apiservice-cert" pod "e7926992-9308-423f-bfdb-02c9824523e1" (UID: "e7926992-9308-423f-bfdb-02c9824523e1"). InnerVolumeSpecName "apiservice-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:56:50.893153 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:50.893114 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7926992-9308-423f-bfdb-02c9824523e1-webhook-cert" pod "e7926992-9308-423f-bfdb-02c9824523e1" (UID: "e7926992-9308-423f-bfdb-02c9824523e1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:56:50.991561 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:50.991529 5882 reconciler_common.go:299] "Volume detached for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7926992-9308-423f-bfdb-02c9824523e1-apiservice-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:56:50.991561 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:50.991557 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trf4j\" (UniqueName: \"kubernetes.io/projected/e7926992-9308-423f-bfdb-02c9824523e1-kube-api-access-trf4j\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:56:50.991561 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:50.991567 5882 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7926992-9308-423f-bfdb-02c9824523e1-webhook-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:56:51.489025 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:51.488987 5882 generic.go:356] "Generic (PLEG): container finished" podID="e7926992-9308-423f-bfdb-02c9824523e1" containerID="4c287f458782b2c697bb1c318bdd076e1211d1d3f5207a5d1977640040cf90b4" exitCode=0 Oct 30 02:56:51.489508 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:51.489037 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq" Oct 30 02:56:51.489508 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:51.489066 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq" event={"ID":"e7926992-9308-423f-bfdb-02c9824523e1","Type":"ContainerDied","Data":"4c287f458782b2c697bb1c318bdd076e1211d1d3f5207a5d1977640040cf90b4"} Oct 30 02:56:51.489508 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:51.489113 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq" event={"ID":"e7926992-9308-423f-bfdb-02c9824523e1","Type":"ContainerDied","Data":"1bbb74231f32d67c8bb13e8120652ff01a5da3506b4f2f83bd6eb3aa7ae2955b"} Oct 30 02:56:51.489508 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:51.489133 5882 scope.go:122] "RemoveContainer" containerID="4c287f458782b2c697bb1c318bdd076e1211d1d3f5207a5d1977640040cf90b4" Oct 30 02:56:51.497700 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:51.497682 5882 scope.go:122] "RemoveContainer" containerID="4c287f458782b2c697bb1c318bdd076e1211d1d3f5207a5d1977640040cf90b4" Oct 30 02:56:51.497959 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:56:51.497939 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c287f458782b2c697bb1c318bdd076e1211d1d3f5207a5d1977640040cf90b4\": container with ID starting with 4c287f458782b2c697bb1c318bdd076e1211d1d3f5207a5d1977640040cf90b4 not found: ID does not exist" containerID="4c287f458782b2c697bb1c318bdd076e1211d1d3f5207a5d1977640040cf90b4" Oct 30 02:56:51.498001 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:51.497970 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c287f458782b2c697bb1c318bdd076e1211d1d3f5207a5d1977640040cf90b4"} err="failed to get container status \"4c287f458782b2c697bb1c318bdd076e1211d1d3f5207a5d1977640040cf90b4\": rpc error: code = NotFound desc = could not find container \"4c287f458782b2c697bb1c318bdd076e1211d1d3f5207a5d1977640040cf90b4\": container with ID starting with 4c287f458782b2c697bb1c318bdd076e1211d1d3f5207a5d1977640040cf90b4 not found: ID does not exist" Oct 30 02:56:51.550366 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:51.550343 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq"] Oct 30 02:56:51.562221 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:51.562198 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-787f9767b8-hf6jq"] Oct 30 02:56:52.009057 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:56:52.009023 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7926992-9308-423f-bfdb-02c9824523e1" path="/var/lib/kubelet/pods/e7926992-9308-423f-bfdb-02c9824523e1/volumes" Oct 30 02:56:53.942519 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:56:53.942484 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:57:23.941784 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:57:23.941750 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:57:26.610410 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:26.610371 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 02:57:26.610886 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:26.610868 5882 cpu_manager.go:414] "containerMap: removing container" podUID="e7926992-9308-423f-bfdb-02c9824523e1" containerName="webhook" Oct 30 02:57:26.610886 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:26.610885 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7926992-9308-423f-bfdb-02c9824523e1" containerName="webhook" Oct 30 02:57:26.610993 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:26.610981 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7926992-9308-423f-bfdb-02c9824523e1" containerName="webhook" Oct 30 02:57:26.617106 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:26.617084 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:26.632381 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:26.632340 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 02:57:26.643524 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:26.643499 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 30 02:57:26.643725 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:26.643546 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 02:57:26.643725 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:26.643601 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 02:57:26.644904 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:26.644887 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 02:57:26.705395 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:26.705367 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lmx6\" (UniqueName: \"kubernetes.io/projected/b55502fd-2c06-4082-8e5e-a5119afcc3b9-kube-api-access-8lmx6\") pod \"data-science-perses-0\" (UID: \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:26.705395 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:26.705398 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/b55502fd-2c06-4082-8e5e-a5119afcc3b9-plugins\") pod \"data-science-perses-0\" (UID: \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:26.705571 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:26.705439 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b55502fd-2c06-4082-8e5e-a5119afcc3b9-config\") pod \"data-science-perses-0\" (UID: \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:26.807003 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:26.806971 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lmx6\" (UniqueName: \"kubernetes.io/projected/b55502fd-2c06-4082-8e5e-a5119afcc3b9-kube-api-access-8lmx6\") pod \"data-science-perses-0\" (UID: \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:26.807163 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:26.807030 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/b55502fd-2c06-4082-8e5e-a5119afcc3b9-plugins\") pod \"data-science-perses-0\" (UID: \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:26.807163 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:26.807097 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b55502fd-2c06-4082-8e5e-a5119afcc3b9-config\") pod \"data-science-perses-0\" (UID: \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:26.807471 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:26.807449 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/b55502fd-2c06-4082-8e5e-a5119afcc3b9-plugins\") pod \"data-science-perses-0\" (UID: \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:26.807757 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:26.807738 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b55502fd-2c06-4082-8e5e-a5119afcc3b9-config\") pod \"data-science-perses-0\" (UID: \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:26.830276 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:26.830250 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lmx6\" (UniqueName: \"kubernetes.io/projected/b55502fd-2c06-4082-8e5e-a5119afcc3b9-kube-api-access-8lmx6\") pod \"data-science-perses-0\" (UID: \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:29.228783 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:29.228750 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:29.256749 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:29.256719 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:29.330070 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:29.330035 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:29.330225 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:29.330202 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:29.344251 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:29.344224 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\") DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:29.532312 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:29.532285 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2c422b71c1e0b8fdf34659f2cef7846e06b4353ed88027dc99ade5d7fc29e9d6/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:29.559795 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:29.559767 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:29.652174 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:29.652149 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 02:57:29.659305 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:29.659283 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:29.954049 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:29.953970 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 02:57:29.957838 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:57:29.957807 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb55502fd_2c06_4082_8e5e_a5119afcc3b9.slice/crio-7674889ce3ed4da90c9bc181e66154c20c43898d9d04166cc56c75a118e0af5c WatchSource:0}: Error finding container 7674889ce3ed4da90c9bc181e66154c20c43898d9d04166cc56c75a118e0af5c: Status 404 returned error can't find the container with id 7674889ce3ed4da90c9bc181e66154c20c43898d9d04166cc56c75a118e0af5c Oct 30 02:57:30.653806 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:30.653771 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"b55502fd-2c06-4082-8e5e-a5119afcc3b9","Type":"ContainerStarted","Data":"7674889ce3ed4da90c9bc181e66154c20c43898d9d04166cc56c75a118e0af5c"} Oct 30 02:57:33.669710 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:33.669669 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"b55502fd-2c06-4082-8e5e-a5119afcc3b9","Type":"ContainerStarted","Data":"f329582f1490eeeec673f7782c2a5c24bf2cdcbfc031ea3f561ff8b6ad19ffd4"} Oct 30 02:57:33.670094 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:33.669819 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:33.711387 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:33.711332 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=7.200079967 podStartE2EDuration="10.711318479s" totalImagesPullingTime="3.511238512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:33.710695577 +0000 UTC m=+965.825010564" watchObservedRunningTime="2026-10-30 02:57:33.711318479 +0000 UTC m=+965.825633448" Oct 30 02:57:41.178644 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.178470 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 02:57:41.179359 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.178729 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="b55502fd-2c06-4082-8e5e-a5119afcc3b9" containerName="perses" containerID="cri-o://f329582f1490eeeec673f7782c2a5c24bf2cdcbfc031ea3f561ff8b6ad19ffd4" gracePeriod=30 Oct 30 02:57:41.421400 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.421370 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:41.541373 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.541344 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b55502fd-2c06-4082-8e5e-a5119afcc3b9-plugins\" (UniqueName: \"kubernetes.io/empty-dir/b55502fd-2c06-4082-8e5e-a5119afcc3b9-plugins\") pod \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\" (UID: \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\") " Oct 30 02:57:41.541526 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.541403 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b55502fd-2c06-4082-8e5e-a5119afcc3b9-kube-api-access-8lmx6\" (UniqueName: \"kubernetes.io/projected/b55502fd-2c06-4082-8e5e-a5119afcc3b9-kube-api-access-8lmx6\") pod \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\" (UID: \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\") " Oct 30 02:57:41.541526 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.541444 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b55502fd-2c06-4082-8e5e-a5119afcc3b9-config\" (UniqueName: \"kubernetes.io/configmap/b55502fd-2c06-4082-8e5e-a5119afcc3b9-config\") pod \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\" (UID: \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\") " Oct 30 02:57:41.541669 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.541580 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\" (UID: \"b55502fd-2c06-4082-8e5e-a5119afcc3b9\") " Oct 30 02:57:41.541939 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.541902 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b55502fd-2c06-4082-8e5e-a5119afcc3b9-config" pod "b55502fd-2c06-4082-8e5e-a5119afcc3b9" (UID: "b55502fd-2c06-4082-8e5e-a5119afcc3b9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:57:41.544260 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.544231 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b55502fd-2c06-4082-8e5e-a5119afcc3b9-kube-api-access-8lmx6" pod "b55502fd-2c06-4082-8e5e-a5119afcc3b9" (UID: "b55502fd-2c06-4082-8e5e-a5119afcc3b9"). InnerVolumeSpecName "kube-api-access-8lmx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:57:41.545809 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.545787 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e" pod "b55502fd-2c06-4082-8e5e-a5119afcc3b9" (UID: "b55502fd-2c06-4082-8e5e-a5119afcc3b9"). InnerVolumeSpecName "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 02:57:41.642262 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.642232 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lmx6\" (UniqueName: \"kubernetes.io/projected/b55502fd-2c06-4082-8e5e-a5119afcc3b9-kube-api-access-8lmx6\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:57:41.642262 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.642261 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b55502fd-2c06-4082-8e5e-a5119afcc3b9-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:57:41.642446 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.642291 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 02:57:41.650048 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.650029 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e") on node "ip-10-0-141-200.ec2.internal" Oct 30 02:57:41.666191 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.666147 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b55502fd-2c06-4082-8e5e-a5119afcc3b9-plugins" pod "b55502fd-2c06-4082-8e5e-a5119afcc3b9" (UID: "b55502fd-2c06-4082-8e5e-a5119afcc3b9"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:57:41.702867 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.702832 5882 generic.go:356] "Generic (PLEG): container finished" podID="b55502fd-2c06-4082-8e5e-a5119afcc3b9" containerID="f329582f1490eeeec673f7782c2a5c24bf2cdcbfc031ea3f561ff8b6ad19ffd4" exitCode=0 Oct 30 02:57:41.703011 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.702893 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 02:57:41.703011 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.702929 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"b55502fd-2c06-4082-8e5e-a5119afcc3b9","Type":"ContainerDied","Data":"f329582f1490eeeec673f7782c2a5c24bf2cdcbfc031ea3f561ff8b6ad19ffd4"} Oct 30 02:57:41.703011 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.702978 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"b55502fd-2c06-4082-8e5e-a5119afcc3b9","Type":"ContainerDied","Data":"7674889ce3ed4da90c9bc181e66154c20c43898d9d04166cc56c75a118e0af5c"} Oct 30 02:57:41.703011 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.702997 5882 scope.go:122] "RemoveContainer" containerID="f329582f1490eeeec673f7782c2a5c24bf2cdcbfc031ea3f561ff8b6ad19ffd4" Oct 30 02:57:41.711310 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.711291 5882 scope.go:122] "RemoveContainer" containerID="f329582f1490eeeec673f7782c2a5c24bf2cdcbfc031ea3f561ff8b6ad19ffd4" Oct 30 02:57:41.711553 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:57:41.711537 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f329582f1490eeeec673f7782c2a5c24bf2cdcbfc031ea3f561ff8b6ad19ffd4\": container with ID starting with f329582f1490eeeec673f7782c2a5c24bf2cdcbfc031ea3f561ff8b6ad19ffd4 not found: ID does not exist" containerID="f329582f1490eeeec673f7782c2a5c24bf2cdcbfc031ea3f561ff8b6ad19ffd4" Oct 30 02:57:41.711605 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.711559 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f329582f1490eeeec673f7782c2a5c24bf2cdcbfc031ea3f561ff8b6ad19ffd4"} err="failed to get container status \"f329582f1490eeeec673f7782c2a5c24bf2cdcbfc031ea3f561ff8b6ad19ffd4\": rpc error: code = NotFound desc = could not find container \"f329582f1490eeeec673f7782c2a5c24bf2cdcbfc031ea3f561ff8b6ad19ffd4\": container with ID starting with f329582f1490eeeec673f7782c2a5c24bf2cdcbfc031ea3f561ff8b6ad19ffd4 not found: ID does not exist" Oct 30 02:57:41.743305 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.743278 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" Oct 30 02:57:41.743305 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.743302 5882 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/b55502fd-2c06-4082-8e5e-a5119afcc3b9-plugins\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 02:57:41.751693 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.751667 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 02:57:41.756344 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:41.756325 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 02:57:42.009852 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:42.009821 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b55502fd-2c06-4082-8e5e-a5119afcc3b9" path="/var/lib/kubelet/pods/b55502fd-2c06-4082-8e5e-a5119afcc3b9/volumes" Oct 30 02:57:47.352030 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:47.351995 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf"] Oct 30 02:57:47.352566 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:47.352545 5882 cpu_manager.go:414] "containerMap: removing container" podUID="b55502fd-2c06-4082-8e5e-a5119afcc3b9" containerName="perses" Oct 30 02:57:47.352663 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:47.352568 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b55502fd-2c06-4082-8e5e-a5119afcc3b9" containerName="perses" Oct 30 02:57:47.352727 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:47.352707 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b55502fd-2c06-4082-8e5e-a5119afcc3b9" containerName="perses" Oct 30 02:57:47.355753 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:47.355731 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" Oct 30 02:57:47.370958 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:47.370936 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 30 02:57:47.382878 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:47.382853 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-8rglz\"" Oct 30 02:57:47.384353 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:47.384333 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf"] Oct 30 02:57:47.388524 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:47.388501 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-ta-internal\") pod \"data-science-collector-targetallocator-768cc89879-km5qf\" (UID: \"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" Oct 30 02:57:47.388589 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:47.388574 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-768cc89879-km5qf\" (UID: \"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" Oct 30 02:57:47.388690 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:47.388667 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trzxg\" (UniqueName: \"kubernetes.io/projected/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-kube-api-access-trzxg\") pod \"data-science-collector-targetallocator-768cc89879-km5qf\" (UID: \"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" Oct 30 02:57:47.443035 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:47.442999 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 02:57:47.443166 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:47.443052 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 30 02:57:47.443166 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:47.443064 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 02:57:47.489221 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:47.489182 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-768cc89879-km5qf\" (UID: \"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" Oct 30 02:57:47.489397 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:47.489228 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trzxg\" (UniqueName: \"kubernetes.io/projected/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-kube-api-access-trzxg\") pod \"data-science-collector-targetallocator-768cc89879-km5qf\" (UID: \"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" Oct 30 02:57:47.489397 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:47.489287 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-ta-internal\") pod \"data-science-collector-targetallocator-768cc89879-km5qf\" (UID: \"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" Oct 30 02:57:47.489397 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:57:47.489336 5882 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-server-cert: secret "data-science-collector-ta-server-cert" not found Oct 30 02:57:47.489567 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:57:47.489429 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-data-science-collector-ta-server-cert podName:37e8e895-0acb-4afc-b3a4-8462cc8bf1a0 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:47.989407258 +0000 UTC m=+980.103722232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-server-cert" (UniqueName: "kubernetes.io/secret/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-data-science-collector-ta-server-cert") pod "data-science-collector-targetallocator-768cc89879-km5qf" (UID: "37e8e895-0acb-4afc-b3a4-8462cc8bf1a0") : secret "data-science-collector-ta-server-cert" not found Oct 30 02:57:47.489988 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:47.489969 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-ta-internal\") pod \"data-science-collector-targetallocator-768cc89879-km5qf\" (UID: \"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" Oct 30 02:57:47.513787 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:47.513759 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trzxg\" (UniqueName: \"kubernetes.io/projected/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-kube-api-access-trzxg\") pod \"data-science-collector-targetallocator-768cc89879-km5qf\" (UID: \"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" Oct 30 02:57:47.994405 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:47.994368 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-768cc89879-km5qf\" (UID: \"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" Oct 30 02:57:47.994579 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:57:47.994515 5882 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-server-cert: secret "data-science-collector-ta-server-cert" not found Oct 30 02:57:47.994648 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:57:47.994577 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-data-science-collector-ta-server-cert podName:37e8e895-0acb-4afc-b3a4-8462cc8bf1a0 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:48.994560308 +0000 UTC m=+981.108875281 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-server-cert" (UniqueName: "kubernetes.io/secret/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-data-science-collector-ta-server-cert") pod "data-science-collector-targetallocator-768cc89879-km5qf" (UID: "37e8e895-0acb-4afc-b3a4-8462cc8bf1a0") : secret "data-science-collector-ta-server-cert" not found Oct 30 02:57:49.002112 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:49.002071 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-768cc89879-km5qf\" (UID: \"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" Oct 30 02:57:49.004473 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:49.004452 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-768cc89879-km5qf\" (UID: \"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" Oct 30 02:57:49.180855 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:49.180808 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" Oct 30 02:57:49.474956 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:49.474863 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf"] Oct 30 02:57:49.477945 ip-10-0-141-200 kubenswrapper[5882]: W1030 02:57:49.477915 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37e8e895_0acb_4afc_b3a4_8462cc8bf1a0.slice/crio-6d6f5d278b3c10aa80f47ca52902a3f10125d41957241bcb4071a704aba33f9d WatchSource:0}: Error finding container 6d6f5d278b3c10aa80f47ca52902a3f10125d41957241bcb4071a704aba33f9d: Status 404 returned error can't find the container with id 6d6f5d278b3c10aa80f47ca52902a3f10125d41957241bcb4071a704aba33f9d Oct 30 02:57:49.737592 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:49.737508 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" event={"ID":"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0","Type":"ContainerStarted","Data":"6d6f5d278b3c10aa80f47ca52902a3f10125d41957241bcb4071a704aba33f9d"} Oct 30 02:57:53.135325 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:53.135285 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:57:53.135725 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:53.135360 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:57:53.756456 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:53.756423 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" event={"ID":"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0","Type":"ContainerStarted","Data":"c834c95227aca580b3e7985acae193b7fd57c2d007c1563316101441a4fa6dd6"} Oct 30 02:57:53.756606 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:53.756544 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" Oct 30 02:57:53.758294 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:53.758273 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" Oct 30 02:57:53.797434 ip-10-0-141-200 kubenswrapper[5882]: I1030 02:57:53.797385 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" podStartSLOduration=3.142759463 podStartE2EDuration="6.797372804s" totalImagesPullingTime="3.654613341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:53.796737597 +0000 UTC m=+985.911052571" watchObservedRunningTime="2026-10-30 02:57:53.797372804 +0000 UTC m=+985.911687778" Oct 30 02:57:53.941602 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:57:53.941570 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:58:23.942441 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:58:23.942392 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:58:53.942833 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:58:53.942791 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:59:23.943714 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:59:23.943672 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 02:59:53.942658 ip-10-0-141-200 kubenswrapper[5882]: E1030 02:59:53.942604 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:00:23.943587 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:00:23.943537 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:00:53.943553 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:00:53.943512 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:01:23.942641 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:01:23.942575 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:01:28.112209 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:01:28.112177 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:01:28.114529 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:01:28.114505 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:01:53.944406 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:01:53.944363 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:02:23.943018 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:02:23.942980 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:02:53.942422 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:02:53.942383 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:03:23.942479 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:03:23.942438 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:03:53.942943 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:03:53.942906 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:04:23.942632 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:04:23.942537 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:04:53.942536 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:04:53.942500 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:05:23.942710 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:05:23.942675 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:05:53.942627 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:05:53.942575 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:06:23.942813 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:06:23.942771 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:06:28.140726 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:06:28.140694 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:06:28.143357 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:06:28.143337 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:06:53.944917 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:06:53.944869 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:07:23.941965 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:07:23.941888 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:07:53.941807 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:07:53.941765 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:08:02.281144 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:02.281104 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf"] Oct 30 03:08:02.281828 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:02.281338 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" podUID="37e8e895-0acb-4afc-b3a4-8462cc8bf1a0" containerName="ta-container" containerID="cri-o://c834c95227aca580b3e7985acae193b7fd57c2d007c1563316101441a4fa6dd6" gracePeriod=30 Oct 30 03:08:02.520280 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:02.520253 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" Oct 30 03:08:02.635870 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:02.635777 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-ta-internal\" (UniqueName: \"kubernetes.io/configmap/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-ta-internal\") pod \"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0\" (UID: \"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0\") " Oct 30 03:08:02.635870 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:02.635844 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-data-science-collector-ta-server-cert\") pod \"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0\" (UID: \"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0\") " Oct 30 03:08:02.636092 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:02.635889 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-kube-api-access-trzxg\" (UniqueName: \"kubernetes.io/projected/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-kube-api-access-trzxg\") pod \"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0\" (UID: \"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0\") " Oct 30 03:08:02.636242 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:02.636208 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-ta-internal" pod "37e8e895-0acb-4afc-b3a4-8462cc8bf1a0" (UID: "37e8e895-0acb-4afc-b3a4-8462cc8bf1a0"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:08:02.637899 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:02.637876 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-kube-api-access-trzxg" pod "37e8e895-0acb-4afc-b3a4-8462cc8bf1a0" (UID: "37e8e895-0acb-4afc-b3a4-8462cc8bf1a0"). InnerVolumeSpecName "kube-api-access-trzxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:08:02.637967 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:02.637952 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-data-science-collector-ta-server-cert" pod "37e8e895-0acb-4afc-b3a4-8462cc8bf1a0" (UID: "37e8e895-0acb-4afc-b3a4-8462cc8bf1a0"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:08:02.737235 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:02.737207 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:08:02.737235 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:02.737235 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:08:02.737414 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:02.737246 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trzxg\" (UniqueName: \"kubernetes.io/projected/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0-kube-api-access-trzxg\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:08:03.253471 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:03.253433 5882 generic.go:356] "Generic (PLEG): container finished" podID="37e8e895-0acb-4afc-b3a4-8462cc8bf1a0" containerID="c834c95227aca580b3e7985acae193b7fd57c2d007c1563316101441a4fa6dd6" exitCode=0 Oct 30 03:08:03.253471 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:03.253472 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" Oct 30 03:08:03.253698 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:03.253519 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" event={"ID":"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0","Type":"ContainerDied","Data":"c834c95227aca580b3e7985acae193b7fd57c2d007c1563316101441a4fa6dd6"} Oct 30 03:08:03.253698 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:03.253557 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf" event={"ID":"37e8e895-0acb-4afc-b3a4-8462cc8bf1a0","Type":"ContainerDied","Data":"6d6f5d278b3c10aa80f47ca52902a3f10125d41957241bcb4071a704aba33f9d"} Oct 30 03:08:03.253698 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:03.253574 5882 scope.go:122] "RemoveContainer" containerID="c834c95227aca580b3e7985acae193b7fd57c2d007c1563316101441a4fa6dd6" Oct 30 03:08:03.261881 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:03.261858 5882 scope.go:122] "RemoveContainer" containerID="c834c95227aca580b3e7985acae193b7fd57c2d007c1563316101441a4fa6dd6" Oct 30 03:08:03.262112 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:08:03.262094 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c834c95227aca580b3e7985acae193b7fd57c2d007c1563316101441a4fa6dd6\": container with ID starting with c834c95227aca580b3e7985acae193b7fd57c2d007c1563316101441a4fa6dd6 not found: ID does not exist" containerID="c834c95227aca580b3e7985acae193b7fd57c2d007c1563316101441a4fa6dd6" Oct 30 03:08:03.262168 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:03.262121 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c834c95227aca580b3e7985acae193b7fd57c2d007c1563316101441a4fa6dd6"} err="failed to get container status \"c834c95227aca580b3e7985acae193b7fd57c2d007c1563316101441a4fa6dd6\": rpc error: code = NotFound desc = could not find container \"c834c95227aca580b3e7985acae193b7fd57c2d007c1563316101441a4fa6dd6\": container with ID starting with c834c95227aca580b3e7985acae193b7fd57c2d007c1563316101441a4fa6dd6 not found: ID does not exist" Oct 30 03:08:03.303372 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:03.303345 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf"] Oct 30 03:08:03.315291 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:03.315269 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-768cc89879-km5qf"] Oct 30 03:08:04.009141 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:04.009104 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37e8e895-0acb-4afc-b3a4-8462cc8bf1a0" path="/var/lib/kubelet/pods/37e8e895-0acb-4afc-b3a4-8462cc8bf1a0/volumes" Oct 30 03:08:23.942731 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:08:23.942688 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:08:27.248087 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.248053 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq"] Oct 30 03:08:27.248551 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.248529 5882 cpu_manager.go:414] "containerMap: removing container" podUID="37e8e895-0acb-4afc-b3a4-8462cc8bf1a0" containerName="ta-container" Oct 30 03:08:27.248638 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.248552 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37e8e895-0acb-4afc-b3a4-8462cc8bf1a0" containerName="ta-container" Oct 30 03:08:27.248706 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.248665 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="37e8e895-0acb-4afc-b3a4-8462cc8bf1a0" containerName="ta-container" Oct 30 03:08:27.251596 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.251578 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" Oct 30 03:08:27.268403 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.268384 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-bq58m\"" Oct 30 03:08:27.280600 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.280571 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 30 03:08:27.280600 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.280594 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 03:08:27.282088 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.282064 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq"] Oct 30 03:08:27.284479 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.284458 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 03:08:27.284523 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.284488 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 30 03:08:27.348098 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.348074 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-697b95795f-h8mbq\" (UID: \"dc0dcf8e-bd2d-41a9-adb7-3a171a88d745\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" Oct 30 03:08:27.348262 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.348111 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgcxc\" (UniqueName: \"kubernetes.io/projected/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-kube-api-access-tgcxc\") pod \"data-science-collector-targetallocator-697b95795f-h8mbq\" (UID: \"dc0dcf8e-bd2d-41a9-adb7-3a171a88d745\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" Oct 30 03:08:27.348262 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.348133 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-ta-internal\") pod \"data-science-collector-targetallocator-697b95795f-h8mbq\" (UID: \"dc0dcf8e-bd2d-41a9-adb7-3a171a88d745\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" Oct 30 03:08:27.449528 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.449496 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-697b95795f-h8mbq\" (UID: \"dc0dcf8e-bd2d-41a9-adb7-3a171a88d745\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" Oct 30 03:08:27.449709 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.449537 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgcxc\" (UniqueName: \"kubernetes.io/projected/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-kube-api-access-tgcxc\") pod \"data-science-collector-targetallocator-697b95795f-h8mbq\" (UID: \"dc0dcf8e-bd2d-41a9-adb7-3a171a88d745\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" Oct 30 03:08:27.449709 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.449557 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-ta-internal\") pod \"data-science-collector-targetallocator-697b95795f-h8mbq\" (UID: \"dc0dcf8e-bd2d-41a9-adb7-3a171a88d745\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" Oct 30 03:08:27.450151 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.450130 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-ta-internal\") pod \"data-science-collector-targetallocator-697b95795f-h8mbq\" (UID: \"dc0dcf8e-bd2d-41a9-adb7-3a171a88d745\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" Oct 30 03:08:27.452116 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.452086 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-697b95795f-h8mbq\" (UID: \"dc0dcf8e-bd2d-41a9-adb7-3a171a88d745\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" Oct 30 03:08:27.473815 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.473787 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgcxc\" (UniqueName: \"kubernetes.io/projected/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-kube-api-access-tgcxc\") pod \"data-science-collector-targetallocator-697b95795f-h8mbq\" (UID: \"dc0dcf8e-bd2d-41a9-adb7-3a171a88d745\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" Oct 30 03:08:27.571417 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.571342 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" Oct 30 03:08:27.861323 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.861121 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq"] Oct 30 03:08:27.863996 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:08:27.863970 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc0dcf8e_bd2d_41a9_adb7_3a171a88d745.slice/crio-f2a2e983d1667d5e677326faf3c845a53ef6414e2e3e04bc0a1616c7e7f870c8 WatchSource:0}: Error finding container f2a2e983d1667d5e677326faf3c845a53ef6414e2e3e04bc0a1616c7e7f870c8: Status 404 returned error can't find the container with id f2a2e983d1667d5e677326faf3c845a53ef6414e2e3e04bc0a1616c7e7f870c8 Oct 30 03:08:27.866122 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.866082 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:08:27.866220 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:27.866158 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:08:28.357125 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:28.357081 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" event={"ID":"dc0dcf8e-bd2d-41a9-adb7-3a171a88d745","Type":"ContainerStarted","Data":"e2f929fb8387e926583be7a47baa7f124793ce67340f86ba0bba9523d24873ec"} Oct 30 03:08:28.357642 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:28.357135 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" event={"ID":"dc0dcf8e-bd2d-41a9-adb7-3a171a88d745","Type":"ContainerStarted","Data":"f2a2e983d1667d5e677326faf3c845a53ef6414e2e3e04bc0a1616c7e7f870c8"} Oct 30 03:08:28.358500 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:28.358474 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" Oct 30 03:08:28.360637 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:28.360595 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" Oct 30 03:08:28.398955 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:08:28.398907 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" podStartSLOduration=1.398895132 podStartE2EDuration="1.398895132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:28.398123931 +0000 UTC m=+1620.512438906" watchObservedRunningTime="2026-10-30 03:08:28.398895132 +0000 UTC m=+1620.513210107" Oct 30 03:08:53.943112 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:08:53.943072 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:09:23.942939 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:09:23.942894 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:09:53.942869 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:09:53.942824 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:10:23.942532 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:10:23.942492 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:10:53.942789 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:10:53.942749 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:11:23.941994 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:11:23.941956 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:11:28.167595 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:11:28.167566 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:11:28.170929 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:11:28.170907 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:11:53.942765 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:11:53.942729 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:12:23.941993 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:12:23.941950 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:12:53.942736 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:12:53.942692 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:13:23.942706 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:13:23.942603 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:13:53.942479 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:13:53.942447 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:14:23.942555 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:14:23.942514 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:14:53.943849 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:14:53.943813 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:15:23.941776 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:15:23.941737 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:15:53.945518 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:15:53.945473 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:16:23.942755 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:16:23.942681 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:16:28.194422 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:16:28.194312 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:16:28.198143 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:16:28.198122 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:16:53.943279 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:16:53.943239 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:17:23.942899 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:17:23.942863 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:17:53.942671 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:17:53.942636 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:18:23.942946 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:18:23.942895 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:18:41.936809 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:41.936774 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq"] Oct 30 03:18:41.937443 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:41.937204 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" podUID="dc0dcf8e-bd2d-41a9-adb7-3a171a88d745" containerName="ta-container" containerID="cri-o://e2f929fb8387e926583be7a47baa7f124793ce67340f86ba0bba9523d24873ec" gracePeriod=30 Oct 30 03:18:42.172585 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:42.172558 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" Oct 30 03:18:42.270103 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:42.270073 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-data-science-collector-ta-server-cert\") pod \"dc0dcf8e-bd2d-41a9-adb7-3a171a88d745\" (UID: \"dc0dcf8e-bd2d-41a9-adb7-3a171a88d745\") " Oct 30 03:18:42.270236 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:42.270123 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-ta-internal\" (UniqueName: \"kubernetes.io/configmap/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-ta-internal\") pod \"dc0dcf8e-bd2d-41a9-adb7-3a171a88d745\" (UID: \"dc0dcf8e-bd2d-41a9-adb7-3a171a88d745\") " Oct 30 03:18:42.270236 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:42.270210 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-kube-api-access-tgcxc\" (UniqueName: \"kubernetes.io/projected/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-kube-api-access-tgcxc\") pod \"dc0dcf8e-bd2d-41a9-adb7-3a171a88d745\" (UID: \"dc0dcf8e-bd2d-41a9-adb7-3a171a88d745\") " Oct 30 03:18:42.270489 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:42.270454 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-ta-internal" pod "dc0dcf8e-bd2d-41a9-adb7-3a171a88d745" (UID: "dc0dcf8e-bd2d-41a9-adb7-3a171a88d745"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:18:42.272163 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:42.272138 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-data-science-collector-ta-server-cert" pod "dc0dcf8e-bd2d-41a9-adb7-3a171a88d745" (UID: "dc0dcf8e-bd2d-41a9-adb7-3a171a88d745"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:18:42.272253 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:42.272184 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-kube-api-access-tgcxc" pod "dc0dcf8e-bd2d-41a9-adb7-3a171a88d745" (UID: "dc0dcf8e-bd2d-41a9-adb7-3a171a88d745"). InnerVolumeSpecName "kube-api-access-tgcxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:18:42.371559 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:42.371524 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgcxc\" (UniqueName: \"kubernetes.io/projected/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-kube-api-access-tgcxc\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:18:42.371559 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:42.371554 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:18:42.371775 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:42.371570 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:18:42.883406 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:42.883372 5882 generic.go:356] "Generic (PLEG): container finished" podID="dc0dcf8e-bd2d-41a9-adb7-3a171a88d745" containerID="e2f929fb8387e926583be7a47baa7f124793ce67340f86ba0bba9523d24873ec" exitCode=0 Oct 30 03:18:42.883406 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:42.883409 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" event={"ID":"dc0dcf8e-bd2d-41a9-adb7-3a171a88d745","Type":"ContainerDied","Data":"e2f929fb8387e926583be7a47baa7f124793ce67340f86ba0bba9523d24873ec"} Oct 30 03:18:42.883682 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:42.883420 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" Oct 30 03:18:42.883682 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:42.883442 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq" event={"ID":"dc0dcf8e-bd2d-41a9-adb7-3a171a88d745","Type":"ContainerDied","Data":"f2a2e983d1667d5e677326faf3c845a53ef6414e2e3e04bc0a1616c7e7f870c8"} Oct 30 03:18:42.883682 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:42.883463 5882 scope.go:122] "RemoveContainer" containerID="e2f929fb8387e926583be7a47baa7f124793ce67340f86ba0bba9523d24873ec" Oct 30 03:18:42.891754 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:42.891737 5882 scope.go:122] "RemoveContainer" containerID="e2f929fb8387e926583be7a47baa7f124793ce67340f86ba0bba9523d24873ec" Oct 30 03:18:42.892011 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:18:42.891992 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2f929fb8387e926583be7a47baa7f124793ce67340f86ba0bba9523d24873ec\": container with ID starting with e2f929fb8387e926583be7a47baa7f124793ce67340f86ba0bba9523d24873ec not found: ID does not exist" containerID="e2f929fb8387e926583be7a47baa7f124793ce67340f86ba0bba9523d24873ec" Oct 30 03:18:42.892068 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:42.892022 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2f929fb8387e926583be7a47baa7f124793ce67340f86ba0bba9523d24873ec"} err="failed to get container status \"e2f929fb8387e926583be7a47baa7f124793ce67340f86ba0bba9523d24873ec\": rpc error: code = NotFound desc = could not find container \"e2f929fb8387e926583be7a47baa7f124793ce67340f86ba0bba9523d24873ec\": container with ID starting with e2f929fb8387e926583be7a47baa7f124793ce67340f86ba0bba9523d24873ec not found: ID does not exist" Oct 30 03:18:42.932530 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:42.932506 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq"] Oct 30 03:18:42.936391 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:42.936370 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-697b95795f-h8mbq"] Oct 30 03:18:44.008928 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:44.008894 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc0dcf8e-bd2d-41a9-adb7-3a171a88d745" path="/var/lib/kubelet/pods/dc0dcf8e-bd2d-41a9-adb7-3a171a88d745/volumes" Oct 30 03:18:48.556503 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.556467 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx"] Oct 30 03:18:48.557049 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.557025 5882 cpu_manager.go:414] "containerMap: removing container" podUID="dc0dcf8e-bd2d-41a9-adb7-3a171a88d745" containerName="ta-container" Oct 30 03:18:48.557133 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.557052 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc0dcf8e-bd2d-41a9-adb7-3a171a88d745" containerName="ta-container" Oct 30 03:18:48.557187 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.557168 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc0dcf8e-bd2d-41a9-adb7-3a171a88d745" containerName="ta-container" Oct 30 03:18:48.561553 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.561532 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:18:48.583774 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.583749 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx"] Oct 30 03:18:48.622180 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.622139 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/abd12c2b-3b6f-458c-998a-aed4b6c45409-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-954rx\" (UID: \"abd12c2b-3b6f-458c-998a-aed4b6c45409\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:18:48.622360 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.622186 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnt2l\" (UniqueName: \"kubernetes.io/projected/abd12c2b-3b6f-458c-998a-aed4b6c45409-kube-api-access-jnt2l\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-954rx\" (UID: \"abd12c2b-3b6f-458c-998a-aed4b6c45409\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:18:48.622360 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.622293 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-954rx\" (UID: \"abd12c2b-3b6f-458c-998a-aed4b6c45409\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:18:48.622360 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.622318 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-954rx\" (UID: \"abd12c2b-3b6f-458c-998a-aed4b6c45409\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:18:48.622360 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.622343 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-954rx\" (UID: \"abd12c2b-3b6f-458c-998a-aed4b6c45409\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:18:48.637814 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.637789 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 03:18:48.638974 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.638948 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-t2cgj\"" Oct 30 03:18:48.638974 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.638973 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 03:18:48.639172 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.638979 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 03:18:48.660552 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.660535 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 03:18:48.660552 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.660551 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 03:18:48.660708 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.660555 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 03:18:48.723917 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.723885 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-954rx\" (UID: \"abd12c2b-3b6f-458c-998a-aed4b6c45409\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:18:48.724062 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.723922 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-954rx\" (UID: \"abd12c2b-3b6f-458c-998a-aed4b6c45409\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:18:48.724062 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.723948 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-954rx\" (UID: \"abd12c2b-3b6f-458c-998a-aed4b6c45409\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:18:48.724062 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.724044 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/abd12c2b-3b6f-458c-998a-aed4b6c45409-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-954rx\" (UID: \"abd12c2b-3b6f-458c-998a-aed4b6c45409\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:18:48.724221 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.724073 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnt2l\" (UniqueName: \"kubernetes.io/projected/abd12c2b-3b6f-458c-998a-aed4b6c45409-kube-api-access-jnt2l\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-954rx\" (UID: \"abd12c2b-3b6f-458c-998a-aed4b6c45409\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:18:48.724786 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.724760 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/abd12c2b-3b6f-458c-998a-aed4b6c45409-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-954rx\" (UID: \"abd12c2b-3b6f-458c-998a-aed4b6c45409\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:18:48.726355 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.726328 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-954rx\" (UID: \"abd12c2b-3b6f-458c-998a-aed4b6c45409\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:18:48.726463 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.726381 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-954rx\" (UID: \"abd12c2b-3b6f-458c-998a-aed4b6c45409\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:18:48.726463 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.726405 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-954rx\" (UID: \"abd12c2b-3b6f-458c-998a-aed4b6c45409\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:18:48.747385 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.747359 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnt2l\" (UniqueName: \"kubernetes.io/projected/abd12c2b-3b6f-458c-998a-aed4b6c45409-kube-api-access-jnt2l\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-954rx\" (UID: \"abd12c2b-3b6f-458c-998a-aed4b6c45409\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:18:48.874452 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.874379 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:18:48.929209 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.929181 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6"] Oct 30 03:18:48.934201 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.934173 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6" Oct 30 03:18:48.949641 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.949286 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 03:18:48.953441 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:48.953415 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6"] Oct 30 03:18:49.026561 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:49.026523 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4zsl\" (UniqueName: \"kubernetes.io/projected/59a868a6-a691-4aba-a0eb-c8a20eb14971-kube-api-access-w4zsl\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6\" (UID: \"59a868a6-a691-4aba-a0eb-c8a20eb14971\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6" Oct 30 03:18:49.127487 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:49.127403 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4zsl\" (UniqueName: \"kubernetes.io/projected/59a868a6-a691-4aba-a0eb-c8a20eb14971-kube-api-access-w4zsl\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6\" (UID: \"59a868a6-a691-4aba-a0eb-c8a20eb14971\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6" Oct 30 03:18:49.150844 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:49.150814 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4zsl\" (UniqueName: \"kubernetes.io/projected/59a868a6-a691-4aba-a0eb-c8a20eb14971-kube-api-access-w4zsl\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6\" (UID: \"59a868a6-a691-4aba-a0eb-c8a20eb14971\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6" Oct 30 03:18:49.166803 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:49.166775 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx"] Oct 30 03:18:49.169850 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:18:49.169822 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd12c2b_3b6f_458c_998a_aed4b6c45409.slice/crio-073e5ca0ddc27b1d48dfefbba04ceb17dc98cefb5030bd74891c1dc5d16c3f14 WatchSource:0}: Error finding container 073e5ca0ddc27b1d48dfefbba04ceb17dc98cefb5030bd74891c1dc5d16c3f14: Status 404 returned error can't find the container with id 073e5ca0ddc27b1d48dfefbba04ceb17dc98cefb5030bd74891c1dc5d16c3f14 Oct 30 03:18:49.171695 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:49.171678 5882 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:18:49.262393 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:49.262361 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6" Oct 30 03:18:49.552846 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:49.552765 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6"] Oct 30 03:18:49.555409 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:18:49.555381 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59a868a6_a691_4aba_a0eb_c8a20eb14971.slice/crio-9c5df3db8e5a20a17bc38ce678cd30561a88d5bd0cf5fbb5c0cefced21cf17c3 WatchSource:0}: Error finding container 9c5df3db8e5a20a17bc38ce678cd30561a88d5bd0cf5fbb5c0cefced21cf17c3: Status 404 returned error can't find the container with id 9c5df3db8e5a20a17bc38ce678cd30561a88d5bd0cf5fbb5c0cefced21cf17c3 Oct 30 03:18:49.915960 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:49.915869 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" event={"ID":"abd12c2b-3b6f-458c-998a-aed4b6c45409","Type":"ContainerStarted","Data":"073e5ca0ddc27b1d48dfefbba04ceb17dc98cefb5030bd74891c1dc5d16c3f14"} Oct 30 03:18:49.917248 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:49.917205 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6" event={"ID":"59a868a6-a691-4aba-a0eb-c8a20eb14971","Type":"ContainerStarted","Data":"9c5df3db8e5a20a17bc38ce678cd30561a88d5bd0cf5fbb5c0cefced21cf17c3"} Oct 30 03:18:50.079549 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.079511 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:18:50.083259 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.083229 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:18:50.104822 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.104794 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:18:50.143849 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.143815 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 30 03:18:50.152598 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.152571 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp"] Oct 30 03:18:50.156229 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.156203 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" Oct 30 03:18:50.178975 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.178914 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp"] Oct 30 03:18:50.241198 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.241162 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 30 03:18:50.241371 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.241320 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 30 03:18:50.241371 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.241347 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-vl9w2\"" Oct 30 03:18:50.244813 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.244789 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/def20d1c-e93b-4e8d-9a2c-4106b158909e-config\") pod \"data-science-perses-0\" (UID: \"def20d1c-e93b-4e8d-9a2c-4106b158909e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:18:50.245007 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.244985 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/def20d1c-e93b-4e8d-9a2c-4106b158909e-plugins\") pod \"data-science-perses-0\" (UID: \"def20d1c-e93b-4e8d-9a2c-4106b158909e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:18:50.245128 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.245050 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr6bq\" (UniqueName: \"kubernetes.io/projected/def20d1c-e93b-4e8d-9a2c-4106b158909e-kube-api-access-mr6bq\") pod \"data-science-perses-0\" (UID: \"def20d1c-e93b-4e8d-9a2c-4106b158909e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:18:50.346150 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.346117 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-ta-internal\") pod \"data-science-collector-targetallocator-85d4cc69b-bwpcp\" (UID: \"25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" Oct 30 03:18:50.346348 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.346181 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/def20d1c-e93b-4e8d-9a2c-4106b158909e-config\") pod \"data-science-perses-0\" (UID: \"def20d1c-e93b-4e8d-9a2c-4106b158909e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:18:50.346348 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.346218 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc69c\" (UniqueName: \"kubernetes.io/projected/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-kube-api-access-jc69c\") pod \"data-science-collector-targetallocator-85d4cc69b-bwpcp\" (UID: \"25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" Oct 30 03:18:50.346446 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.346420 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/def20d1c-e93b-4e8d-9a2c-4106b158909e-plugins\") pod \"data-science-perses-0\" (UID: \"def20d1c-e93b-4e8d-9a2c-4106b158909e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:18:50.346491 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.346476 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-85d4cc69b-bwpcp\" (UID: \"25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" Oct 30 03:18:50.346544 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.346529 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr6bq\" (UniqueName: \"kubernetes.io/projected/def20d1c-e93b-4e8d-9a2c-4106b158909e-kube-api-access-mr6bq\") pod \"data-science-perses-0\" (UID: \"def20d1c-e93b-4e8d-9a2c-4106b158909e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:18:50.346948 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.346927 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/def20d1c-e93b-4e8d-9a2c-4106b158909e-config\") pod \"data-science-perses-0\" (UID: \"def20d1c-e93b-4e8d-9a2c-4106b158909e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:18:50.347156 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.347132 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/def20d1c-e93b-4e8d-9a2c-4106b158909e-plugins\") pod \"data-science-perses-0\" (UID: \"def20d1c-e93b-4e8d-9a2c-4106b158909e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:18:50.369886 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.369858 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr6bq\" (UniqueName: \"kubernetes.io/projected/def20d1c-e93b-4e8d-9a2c-4106b158909e-kube-api-access-mr6bq\") pod \"data-science-perses-0\" (UID: \"def20d1c-e93b-4e8d-9a2c-4106b158909e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:18:50.448031 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.447942 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-85d4cc69b-bwpcp\" (UID: \"25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" Oct 30 03:18:50.448180 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.448038 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-ta-internal\") pod \"data-science-collector-targetallocator-85d4cc69b-bwpcp\" (UID: \"25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" Oct 30 03:18:50.448180 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.448093 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc69c\" (UniqueName: \"kubernetes.io/projected/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-kube-api-access-jc69c\") pod \"data-science-collector-targetallocator-85d4cc69b-bwpcp\" (UID: \"25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" Oct 30 03:18:50.449076 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.449048 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-ta-internal\") pod \"data-science-collector-targetallocator-85d4cc69b-bwpcp\" (UID: \"25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" Oct 30 03:18:50.450755 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.450728 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-85d4cc69b-bwpcp\" (UID: \"25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" Oct 30 03:18:50.472318 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.472292 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc69c\" (UniqueName: \"kubernetes.io/projected/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-kube-api-access-jc69c\") pod \"data-science-collector-targetallocator-85d4cc69b-bwpcp\" (UID: \"25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" Oct 30 03:18:50.772030 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:50.771990 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" Oct 30 03:18:51.085692 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:51.083325 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp"] Oct 30 03:18:51.087753 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:18:51.087718 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25f0e069_c9d1_47aa_a6b2_2f7a18fdb1f2.slice/crio-d4dbe89ea623706219c182ab9ce08cd1a848d46d47a4bfdcfef40375726657a1 WatchSource:0}: Error finding container d4dbe89ea623706219c182ab9ce08cd1a848d46d47a4bfdcfef40375726657a1: Status 404 returned error can't find the container with id d4dbe89ea623706219c182ab9ce08cd1a848d46d47a4bfdcfef40375726657a1 Oct 30 03:18:51.091128 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:51.091087 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:18:51.091241 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:51.091168 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:18:51.932286 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:51.932020 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" event={"ID":"25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2","Type":"ContainerStarted","Data":"e833576e585c8ff6378d7300d61d33827351647bfd56fc75c28bb6e08a08a6cd"} Oct 30 03:18:51.932286 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:51.932075 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" event={"ID":"25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2","Type":"ContainerStarted","Data":"d4dbe89ea623706219c182ab9ce08cd1a848d46d47a4bfdcfef40375726657a1"} Oct 30 03:18:51.932521 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:51.932498 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" Oct 30 03:18:51.935416 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:51.934195 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" Oct 30 03:18:52.020496 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:52.020424 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" podStartSLOduration=2.0203989509999998 podStartE2EDuration="2.020398951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:51.974531205 +0000 UTC m=+2244.088846182" watchObservedRunningTime="2026-10-30 03:18:52.020398951 +0000 UTC m=+2244.134713923" Oct 30 03:18:52.937968 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:52.937934 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" event={"ID":"abd12c2b-3b6f-458c-998a-aed4b6c45409","Type":"ContainerStarted","Data":"f123bd5b956c3f45ba9fbb99d34ab21b144821a19fa9153ee8ed2c9783313eeb"} Oct 30 03:18:52.938430 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:52.938008 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:18:52.939449 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:52.939414 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6" event={"ID":"59a868a6-a691-4aba-a0eb-c8a20eb14971","Type":"ContainerStarted","Data":"61a55250720fdb9a459834d0174184994bc8f30f9f6914ae0ac83f59bf908d08"} Oct 30 03:18:52.980384 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:52.980337 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" podStartSLOduration=2.468095354 podStartE2EDuration="4.980323711s" totalImagesPullingTime="2.512228357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:52.979391615 +0000 UTC m=+2245.093706603" watchObservedRunningTime="2026-10-30 03:18:52.980323711 +0000 UTC m=+2245.094638683" Oct 30 03:18:53.022301 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.022252 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6" podStartSLOduration=2.488005377 podStartE2EDuration="5.022238353s" totalImagesPullingTime="2.534232976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:53.021595982 +0000 UTC m=+2245.135910958" watchObservedRunningTime="2026-10-30 03:18:53.022238353 +0000 UTC m=+2245.136553328" Oct 30 03:18:53.749790 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.749704 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 03:18:53.754587 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.754564 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.776110 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.776075 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x7kq\" (UniqueName: \"kubernetes.io/projected/218de7b6-adcb-481d-966a-6ed3d9f4cc44-kube-api-access-5x7kq\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.776235 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.776119 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.776235 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.776189 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/218de7b6-adcb-481d-966a-6ed3d9f4cc44-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.776235 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.776231 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/218de7b6-adcb-481d-966a-6ed3d9f4cc44-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.776356 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.776276 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.776356 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.776303 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.776356 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.776324 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/218de7b6-adcb-481d-966a-6ed3d9f4cc44-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.779013 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.778989 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 03:18:53.846160 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.846127 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 03:18:53.846160 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.846127 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 03:18:53.846368 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.846137 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 03:18:53.846368 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.846350 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 03:18:53.846436 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.846397 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-zdtts\"" Oct 30 03:18:53.876907 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.876872 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.876907 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.876914 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/218de7b6-adcb-481d-966a-6ed3d9f4cc44-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.877127 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.876960 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x7kq\" (UniqueName: \"kubernetes.io/projected/218de7b6-adcb-481d-966a-6ed3d9f4cc44-kube-api-access-5x7kq\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.877127 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.876991 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.877127 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.877025 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/218de7b6-adcb-481d-966a-6ed3d9f4cc44-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.877127 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.877052 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/218de7b6-adcb-481d-966a-6ed3d9f4cc44-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.877337 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.877229 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.877530 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.877503 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/218de7b6-adcb-481d-966a-6ed3d9f4cc44-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.879470 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.879444 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.879868 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.879841 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/218de7b6-adcb-481d-966a-6ed3d9f4cc44-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.879868 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.879862 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.879988 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.879926 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/218de7b6-adcb-481d-966a-6ed3d9f4cc44-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.880021 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.880008 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.900368 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:53.900337 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x7kq\" (UniqueName: \"kubernetes.io/projected/218de7b6-adcb-481d-966a-6ed3d9f4cc44-kube-api-access-5x7kq\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:53.942062 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:18:53.942028 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:18:54.071386 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:54.071293 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:18:54.388876 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:54.388851 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 03:18:54.390932 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:18:54.390907 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218de7b6_adcb_481d_966a_6ed3d9f4cc44.slice/crio-2dbfa3b350194f6c48c9ac5578c3ff7c25d1c83d4c649777adbf753fd32b1b71 WatchSource:0}: Error finding container 2dbfa3b350194f6c48c9ac5578c3ff7c25d1c83d4c649777adbf753fd32b1b71: Status 404 returned error can't find the container with id 2dbfa3b350194f6c48c9ac5578c3ff7c25d1c83d4c649777adbf753fd32b1b71 Oct 30 03:18:54.950346 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:54.950279 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"218de7b6-adcb-481d-966a-6ed3d9f4cc44","Type":"ContainerStarted","Data":"2dbfa3b350194f6c48c9ac5578c3ff7c25d1c83d4c649777adbf753fd32b1b71"} Oct 30 03:18:55.591259 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:55.591226 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"def20d1c-e93b-4e8d-9a2c-4106b158909e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:18:55.619654 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:55.619605 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"def20d1c-e93b-4e8d-9a2c-4106b158909e\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:18:55.691860 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:55.691829 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"def20d1c-e93b-4e8d-9a2c-4106b158909e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:18:55.691993 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:55.691955 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"def20d1c-e93b-4e8d-9a2c-4106b158909e\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:18:55.706149 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:55.706125 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"def20d1c-e93b-4e8d-9a2c-4106b158909e\") DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:18:55.808752 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:55.808728 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"def20d1c-e93b-4e8d-9a2c-4106b158909e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2c422b71c1e0b8fdf34659f2cef7846e06b4353ed88027dc99ade5d7fc29e9d6/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:18:55.833233 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:55.833207 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"def20d1c-e93b-4e8d-9a2c-4106b158909e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:18:55.955809 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:55.955767 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"218de7b6-adcb-481d-966a-6ed3d9f4cc44","Type":"ContainerStarted","Data":"11dda493a66bfe451f352a8d8166a7ce5e309c67983c18b8d5a4c2323c15b1b8"} Oct 30 03:18:56.098967 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:56.098879 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:18:56.388348 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:56.388272 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6"] Oct 30 03:18:56.388541 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:56.388465 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6" podUID="59a868a6-a691-4aba-a0eb-c8a20eb14971" containerName="thanos-querier" containerID="cri-o://61a55250720fdb9a459834d0174184994bc8f30f9f6914ae0ac83f59bf908d08" gracePeriod=30 Oct 30 03:18:56.392459 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:56.392435 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:18:56.396066 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:18:56.396044 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddef20d1c_e93b_4e8d_9a2c_4106b158909e.slice/crio-fe35f76380cd9aa671d36eb7d3c98d65c95cb4d4432b33edc60b6a5713db01c4 WatchSource:0}: Error finding container fe35f76380cd9aa671d36eb7d3c98d65c95cb4d4432b33edc60b6a5713db01c4: Status 404 returned error can't find the container with id fe35f76380cd9aa671d36eb7d3c98d65c95cb4d4432b33edc60b6a5713db01c4 Oct 30 03:18:56.629638 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:56.629591 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6" Oct 30 03:18:56.698853 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:56.698769 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a868a6-a691-4aba-a0eb-c8a20eb14971-kube-api-access-w4zsl\" (UniqueName: \"kubernetes.io/projected/59a868a6-a691-4aba-a0eb-c8a20eb14971-kube-api-access-w4zsl\") pod \"59a868a6-a691-4aba-a0eb-c8a20eb14971\" (UID: \"59a868a6-a691-4aba-a0eb-c8a20eb14971\") " Oct 30 03:18:56.701181 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:56.701154 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a868a6-a691-4aba-a0eb-c8a20eb14971-kube-api-access-w4zsl" pod "59a868a6-a691-4aba-a0eb-c8a20eb14971" (UID: "59a868a6-a691-4aba-a0eb-c8a20eb14971"). InnerVolumeSpecName "kube-api-access-w4zsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:18:56.799742 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:56.799707 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4zsl\" (UniqueName: \"kubernetes.io/projected/59a868a6-a691-4aba-a0eb-c8a20eb14971-kube-api-access-w4zsl\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:18:56.961609 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:56.961519 5882 generic.go:356] "Generic (PLEG): container finished" podID="59a868a6-a691-4aba-a0eb-c8a20eb14971" containerID="61a55250720fdb9a459834d0174184994bc8f30f9f6914ae0ac83f59bf908d08" exitCode=0 Oct 30 03:18:56.961609 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:56.961565 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6" Oct 30 03:18:56.962167 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:56.961604 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6" event={"ID":"59a868a6-a691-4aba-a0eb-c8a20eb14971","Type":"ContainerDied","Data":"61a55250720fdb9a459834d0174184994bc8f30f9f6914ae0ac83f59bf908d08"} Oct 30 03:18:56.962167 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:56.961668 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6" event={"ID":"59a868a6-a691-4aba-a0eb-c8a20eb14971","Type":"ContainerDied","Data":"9c5df3db8e5a20a17bc38ce678cd30561a88d5bd0cf5fbb5c0cefced21cf17c3"} Oct 30 03:18:56.962167 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:56.961685 5882 scope.go:122] "RemoveContainer" containerID="61a55250720fdb9a459834d0174184994bc8f30f9f6914ae0ac83f59bf908d08" Oct 30 03:18:56.963446 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:56.963407 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"def20d1c-e93b-4e8d-9a2c-4106b158909e","Type":"ContainerStarted","Data":"6b9a36ad00cc511313a7ba88e9b4daf09f73c42f858a2637fd75071a8ba0483f"} Oct 30 03:18:56.963559 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:56.963470 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"def20d1c-e93b-4e8d-9a2c-4106b158909e","Type":"ContainerStarted","Data":"fe35f76380cd9aa671d36eb7d3c98d65c95cb4d4432b33edc60b6a5713db01c4"} Oct 30 03:18:56.963559 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:56.963529 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:18:56.972000 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:56.971981 5882 scope.go:122] "RemoveContainer" containerID="61a55250720fdb9a459834d0174184994bc8f30f9f6914ae0ac83f59bf908d08" Oct 30 03:18:56.972270 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:18:56.972251 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a55250720fdb9a459834d0174184994bc8f30f9f6914ae0ac83f59bf908d08\": container with ID starting with 61a55250720fdb9a459834d0174184994bc8f30f9f6914ae0ac83f59bf908d08 not found: ID does not exist" containerID="61a55250720fdb9a459834d0174184994bc8f30f9f6914ae0ac83f59bf908d08" Oct 30 03:18:56.972333 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:56.972283 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a55250720fdb9a459834d0174184994bc8f30f9f6914ae0ac83f59bf908d08"} err="failed to get container status \"61a55250720fdb9a459834d0174184994bc8f30f9f6914ae0ac83f59bf908d08\": rpc error: code = NotFound desc = could not find container \"61a55250720fdb9a459834d0174184994bc8f30f9f6914ae0ac83f59bf908d08\": container with ID starting with 61a55250720fdb9a459834d0174184994bc8f30f9f6914ae0ac83f59bf908d08 not found: ID does not exist" Oct 30 03:18:57.011455 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:57.011420 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6"] Oct 30 03:18:57.023352 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:57.023319 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-6tgq6"] Oct 30 03:18:57.056881 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:57.056833 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=7.0568199 podStartE2EDuration="7.0568199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:57.055942526 +0000 UTC m=+2249.170257502" watchObservedRunningTime="2026-10-30 03:18:57.0568199 +0000 UTC m=+2249.171134940" Oct 30 03:18:58.017567 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:58.017531 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59a868a6-a691-4aba-a0eb-c8a20eb14971" path="/var/lib/kubelet/pods/59a868a6-a691-4aba-a0eb-c8a20eb14971/volumes" Oct 30 03:18:58.946782 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:58.946755 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:18:59.979775 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:59.979743 5882 generic.go:356] "Generic (PLEG): container finished" podID="218de7b6-adcb-481d-966a-6ed3d9f4cc44" containerID="11dda493a66bfe451f352a8d8166a7ce5e309c67983c18b8d5a4c2323c15b1b8" exitCode=0 Oct 30 03:18:59.980194 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:18:59.979825 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"218de7b6-adcb-481d-966a-6ed3d9f4cc44","Type":"ContainerDied","Data":"11dda493a66bfe451f352a8d8166a7ce5e309c67983c18b8d5a4c2323c15b1b8"} Oct 30 03:19:01.993669 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:19:01.993637 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"218de7b6-adcb-481d-966a-6ed3d9f4cc44","Type":"ContainerStarted","Data":"0ce735288d55dae6e91d14dc9903e3b65abdd25905afc37ae390d19d70411846"} Oct 30 03:19:03.000273 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:19:03.000240 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"218de7b6-adcb-481d-966a-6ed3d9f4cc44","Type":"ContainerStarted","Data":"d24fe754e1b9e32406316f71480bcbd8a05619b9a1a3ba250ad06ab0cf0c137a"} Oct 30 03:19:03.000787 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:19:03.000346 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:19:03.074259 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:19:03.074215 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.990983126 podStartE2EDuration="10.074202202s" totalImagesPullingTime="2.580188217s" totalInitContainerRuntime="3.503030859s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:03.062073168 +0000 UTC m=+2255.176388144" watchObservedRunningTime="2026-10-30 03:19:03.074202202 +0000 UTC m=+2255.188517177" Oct 30 03:19:04.009128 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:19:04.009102 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:19:07.971763 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:19:07.971733 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:19:23.942719 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:19:23.942635 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:19:53.942834 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:19:53.942789 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:20:23.942410 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:20:23.942359 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:20:53.941991 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:20:53.941953 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:21:23.942180 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:21:23.942098 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:21:28.222865 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:21:28.222761 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:21:28.228685 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:21:28.228666 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:21:53.942344 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:21:53.942300 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:22:23.942656 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:22:23.942549 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:22:53.942246 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:22:53.942208 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:23:23.942781 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:23:23.942740 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:23:53.942996 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:23:53.942956 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:24:23.943423 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:24:23.943383 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:24:53.942417 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:24:53.942381 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:25:23.942679 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:25:23.942577 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:25:53.942445 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:25:53.942400 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:26:23.941884 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:26:23.941833 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:26:28.251533 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:26:28.251510 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:26:28.262471 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:26:28.262452 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:26:53.942595 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:26:53.942549 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:27:23.942959 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:27:23.942916 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:27:53.943238 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:27:53.943198 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:28:23.942792 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:28:23.942713 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:28:53.942575 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:28:53.942534 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:29:20.903170 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:20.903127 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx"] Oct 30 03:29:20.903793 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:20.903386 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" podUID="abd12c2b-3b6f-458c-998a-aed4b6c45409" containerName="kube-rbac-proxy" containerID="cri-o://f123bd5b956c3f45ba9fbb99d34ab21b144821a19fa9153ee8ed2c9783313eeb" gracePeriod=30 Oct 30 03:29:21.142423 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.142400 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:29:21.222547 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.222468 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd12c2b-3b6f-458c-998a-aed4b6c45409-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/abd12c2b-3b6f-458c-998a-aed4b6c45409-prometheus-ca\") pod \"abd12c2b-3b6f-458c-998a-aed4b6c45409\" (UID: \"abd12c2b-3b6f-458c-998a-aed4b6c45409\") " Oct 30 03:29:21.222547 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.222511 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-prometheus-client-cert\") pod \"abd12c2b-3b6f-458c-998a-aed4b6c45409\" (UID: \"abd12c2b-3b6f-458c-998a-aed4b6c45409\") " Oct 30 03:29:21.222759 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.222550 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd12c2b-3b6f-458c-998a-aed4b6c45409-kube-api-access-jnt2l\" (UniqueName: \"kubernetes.io/projected/abd12c2b-3b6f-458c-998a-aed4b6c45409-kube-api-access-jnt2l\") pod \"abd12c2b-3b6f-458c-998a-aed4b6c45409\" (UID: \"abd12c2b-3b6f-458c-998a-aed4b6c45409\") " Oct 30 03:29:21.222759 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.222580 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-tls\" (UniqueName: \"kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-tls\") pod \"abd12c2b-3b6f-458c-998a-aed4b6c45409\" (UID: \"abd12c2b-3b6f-458c-998a-aed4b6c45409\") " Oct 30 03:29:21.222759 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.222652 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-kube-rbac-proxy-config\") pod \"abd12c2b-3b6f-458c-998a-aed4b6c45409\" (UID: \"abd12c2b-3b6f-458c-998a-aed4b6c45409\") " Oct 30 03:29:21.223007 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.222976 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd12c2b-3b6f-458c-998a-aed4b6c45409-prometheus-ca" pod "abd12c2b-3b6f-458c-998a-aed4b6c45409" (UID: "abd12c2b-3b6f-458c-998a-aed4b6c45409"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:29:21.224844 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.224815 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd12c2b-3b6f-458c-998a-aed4b6c45409-kube-api-access-jnt2l" pod "abd12c2b-3b6f-458c-998a-aed4b6c45409" (UID: "abd12c2b-3b6f-458c-998a-aed4b6c45409"). InnerVolumeSpecName "kube-api-access-jnt2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:29:21.225111 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.225085 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-kube-rbac-proxy-config" pod "abd12c2b-3b6f-458c-998a-aed4b6c45409" (UID: "abd12c2b-3b6f-458c-998a-aed4b6c45409"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:21.225217 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.225183 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-prometheus-client-cert" pod "abd12c2b-3b6f-458c-998a-aed4b6c45409" (UID: "abd12c2b-3b6f-458c-998a-aed4b6c45409"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:21.225283 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.225207 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-tls" pod "abd12c2b-3b6f-458c-998a-aed4b6c45409" (UID: "abd12c2b-3b6f-458c-998a-aed4b6c45409"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:21.333453 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.333413 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/abd12c2b-3b6f-458c-998a-aed4b6c45409-prometheus-ca\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:29:21.333645 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.333456 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-prometheus-client-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:29:21.333645 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.333476 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnt2l\" (UniqueName: \"kubernetes.io/projected/abd12c2b-3b6f-458c-998a-aed4b6c45409-kube-api-access-jnt2l\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:29:21.333645 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.333499 5882 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-tls\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:29:21.333645 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.333514 5882 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/abd12c2b-3b6f-458c-998a-aed4b6c45409-kube-rbac-proxy-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:29:21.431734 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.431699 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 03:29:21.432512 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.432237 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="218de7b6-adcb-481d-966a-6ed3d9f4cc44" containerName="alertmanager" containerID="cri-o://0ce735288d55dae6e91d14dc9903e3b65abdd25905afc37ae390d19d70411846" gracePeriod=120 Oct 30 03:29:21.432512 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.432336 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="218de7b6-adcb-481d-966a-6ed3d9f4cc44" containerName="config-reloader" containerID="cri-o://d24fe754e1b9e32406316f71480bcbd8a05619b9a1a3ba250ad06ab0cf0c137a" gracePeriod=120 Oct 30 03:29:21.619898 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.619862 5882 generic.go:356] "Generic (PLEG): container finished" podID="abd12c2b-3b6f-458c-998a-aed4b6c45409" containerID="f123bd5b956c3f45ba9fbb99d34ab21b144821a19fa9153ee8ed2c9783313eeb" exitCode=0 Oct 30 03:29:21.620074 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.619926 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" Oct 30 03:29:21.620074 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.619944 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" event={"ID":"abd12c2b-3b6f-458c-998a-aed4b6c45409","Type":"ContainerDied","Data":"f123bd5b956c3f45ba9fbb99d34ab21b144821a19fa9153ee8ed2c9783313eeb"} Oct 30 03:29:21.620074 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.620002 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx" event={"ID":"abd12c2b-3b6f-458c-998a-aed4b6c45409","Type":"ContainerDied","Data":"073e5ca0ddc27b1d48dfefbba04ceb17dc98cefb5030bd74891c1dc5d16c3f14"} Oct 30 03:29:21.620074 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.620034 5882 scope.go:122] "RemoveContainer" containerID="f123bd5b956c3f45ba9fbb99d34ab21b144821a19fa9153ee8ed2c9783313eeb" Oct 30 03:29:21.622250 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.622227 5882 generic.go:356] "Generic (PLEG): container finished" podID="218de7b6-adcb-481d-966a-6ed3d9f4cc44" containerID="d24fe754e1b9e32406316f71480bcbd8a05619b9a1a3ba250ad06ab0cf0c137a" exitCode=0 Oct 30 03:29:21.622371 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.622306 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"218de7b6-adcb-481d-966a-6ed3d9f4cc44","Type":"ContainerDied","Data":"d24fe754e1b9e32406316f71480bcbd8a05619b9a1a3ba250ad06ab0cf0c137a"} Oct 30 03:29:21.628850 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.628835 5882 scope.go:122] "RemoveContainer" containerID="f123bd5b956c3f45ba9fbb99d34ab21b144821a19fa9153ee8ed2c9783313eeb" Oct 30 03:29:21.629119 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:29:21.629103 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f123bd5b956c3f45ba9fbb99d34ab21b144821a19fa9153ee8ed2c9783313eeb\": container with ID starting with f123bd5b956c3f45ba9fbb99d34ab21b144821a19fa9153ee8ed2c9783313eeb not found: ID does not exist" containerID="f123bd5b956c3f45ba9fbb99d34ab21b144821a19fa9153ee8ed2c9783313eeb" Oct 30 03:29:21.629185 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.629126 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f123bd5b956c3f45ba9fbb99d34ab21b144821a19fa9153ee8ed2c9783313eeb"} err="failed to get container status \"f123bd5b956c3f45ba9fbb99d34ab21b144821a19fa9153ee8ed2c9783313eeb\": rpc error: code = NotFound desc = could not find container \"f123bd5b956c3f45ba9fbb99d34ab21b144821a19fa9153ee8ed2c9783313eeb\": container with ID starting with f123bd5b956c3f45ba9fbb99d34ab21b144821a19fa9153ee8ed2c9783313eeb not found: ID does not exist" Oct 30 03:29:21.668069 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.668045 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx"] Oct 30 03:29:21.672699 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:21.672677 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-954rx"] Oct 30 03:29:22.009178 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.009146 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd12c2b-3b6f-458c-998a-aed4b6c45409" path="/var/lib/kubelet/pods/abd12c2b-3b6f-458c-998a-aed4b6c45409/volumes" Oct 30 03:29:22.134754 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.134722 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp"] Oct 30 03:29:22.135100 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.135015 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" podUID="25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2" containerName="ta-container" containerID="cri-o://e833576e585c8ff6378d7300d61d33827351647bfd56fc75c28bb6e08a08a6cd" gracePeriod=30 Oct 30 03:29:22.223025 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.222934 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:29:22.223457 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.223425 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="def20d1c-e93b-4e8d-9a2c-4106b158909e" containerName="perses" containerID="cri-o://6b9a36ad00cc511313a7ba88e9b4daf09f73c42f858a2637fd75071a8ba0483f" gracePeriod=30 Oct 30 03:29:22.483158 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.483135 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" Oct 30 03:29:22.485992 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.485974 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:22.542430 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.542403 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def20d1c-e93b-4e8d-9a2c-4106b158909e-config\" (UniqueName: \"kubernetes.io/configmap/def20d1c-e93b-4e8d-9a2c-4106b158909e-config\") pod \"def20d1c-e93b-4e8d-9a2c-4106b158909e\" (UID: \"def20d1c-e93b-4e8d-9a2c-4106b158909e\") " Oct 30 03:29:22.542569 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.542451 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-data-science-collector-ta-server-cert\") pod \"25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2\" (UID: \"25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2\") " Oct 30 03:29:22.542569 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.542479 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-kube-api-access-jc69c\" (UniqueName: \"kubernetes.io/projected/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-kube-api-access-jc69c\") pod \"25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2\" (UID: \"25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2\") " Oct 30 03:29:22.542569 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.542517 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-ta-internal\" (UniqueName: \"kubernetes.io/configmap/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-ta-internal\") pod \"25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2\" (UID: \"25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2\") " Oct 30 03:29:22.542569 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.542543 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def20d1c-e93b-4e8d-9a2c-4106b158909e-kube-api-access-mr6bq\" (UniqueName: \"kubernetes.io/projected/def20d1c-e93b-4e8d-9a2c-4106b158909e-kube-api-access-mr6bq\") pod \"def20d1c-e93b-4e8d-9a2c-4106b158909e\" (UID: \"def20d1c-e93b-4e8d-9a2c-4106b158909e\") " Oct 30 03:29:22.542823 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.542574 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def20d1c-e93b-4e8d-9a2c-4106b158909e-plugins\" (UniqueName: \"kubernetes.io/empty-dir/def20d1c-e93b-4e8d-9a2c-4106b158909e-plugins\") pod \"def20d1c-e93b-4e8d-9a2c-4106b158909e\" (UID: \"def20d1c-e93b-4e8d-9a2c-4106b158909e\") " Oct 30 03:29:22.542823 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.542688 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"def20d1c-e93b-4e8d-9a2c-4106b158909e\" (UID: \"def20d1c-e93b-4e8d-9a2c-4106b158909e\") " Oct 30 03:29:22.542940 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.542891 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def20d1c-e93b-4e8d-9a2c-4106b158909e-config" pod "def20d1c-e93b-4e8d-9a2c-4106b158909e" (UID: "def20d1c-e93b-4e8d-9a2c-4106b158909e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:29:22.543011 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.542993 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/def20d1c-e93b-4e8d-9a2c-4106b158909e-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:29:22.543306 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.543217 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-ta-internal" pod "25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2" (UID: "25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:29:22.545185 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.545163 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-kube-api-access-jc69c" pod "25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2" (UID: "25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2"). InnerVolumeSpecName "kube-api-access-jc69c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:29:22.545329 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.545306 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-data-science-collector-ta-server-cert" pod "25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2" (UID: "25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:22.545585 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.545565 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def20d1c-e93b-4e8d-9a2c-4106b158909e-kube-api-access-mr6bq" pod "def20d1c-e93b-4e8d-9a2c-4106b158909e" (UID: "def20d1c-e93b-4e8d-9a2c-4106b158909e"). InnerVolumeSpecName "kube-api-access-mr6bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:29:22.546117 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.546095 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e" pod "def20d1c-e93b-4e8d-9a2c-4106b158909e" (UID: "def20d1c-e93b-4e8d-9a2c-4106b158909e"). InnerVolumeSpecName "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 03:29:22.628910 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.628874 5882 generic.go:356] "Generic (PLEG): container finished" podID="def20d1c-e93b-4e8d-9a2c-4106b158909e" containerID="6b9a36ad00cc511313a7ba88e9b4daf09f73c42f858a2637fd75071a8ba0483f" exitCode=0 Oct 30 03:29:22.629070 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.628926 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:22.629070 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.628961 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"def20d1c-e93b-4e8d-9a2c-4106b158909e","Type":"ContainerDied","Data":"6b9a36ad00cc511313a7ba88e9b4daf09f73c42f858a2637fd75071a8ba0483f"} Oct 30 03:29:22.629070 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.628999 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"def20d1c-e93b-4e8d-9a2c-4106b158909e","Type":"ContainerDied","Data":"fe35f76380cd9aa671d36eb7d3c98d65c95cb4d4432b33edc60b6a5713db01c4"} Oct 30 03:29:22.629070 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.629021 5882 scope.go:122] "RemoveContainer" containerID="6b9a36ad00cc511313a7ba88e9b4daf09f73c42f858a2637fd75071a8ba0483f" Oct 30 03:29:22.630312 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.630290 5882 generic.go:356] "Generic (PLEG): container finished" podID="25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2" containerID="e833576e585c8ff6378d7300d61d33827351647bfd56fc75c28bb6e08a08a6cd" exitCode=0 Oct 30 03:29:22.630415 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.630335 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" Oct 30 03:29:22.630415 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.630355 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" event={"ID":"25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2","Type":"ContainerDied","Data":"e833576e585c8ff6378d7300d61d33827351647bfd56fc75c28bb6e08a08a6cd"} Oct 30 03:29:22.630415 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.630380 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp" event={"ID":"25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2","Type":"ContainerDied","Data":"d4dbe89ea623706219c182ab9ce08cd1a848d46d47a4bfdcfef40375726657a1"} Oct 30 03:29:22.638498 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.638477 5882 scope.go:122] "RemoveContainer" containerID="6b9a36ad00cc511313a7ba88e9b4daf09f73c42f858a2637fd75071a8ba0483f" Oct 30 03:29:22.638816 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:29:22.638796 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9a36ad00cc511313a7ba88e9b4daf09f73c42f858a2637fd75071a8ba0483f\": container with ID starting with 6b9a36ad00cc511313a7ba88e9b4daf09f73c42f858a2637fd75071a8ba0483f not found: ID does not exist" containerID="6b9a36ad00cc511313a7ba88e9b4daf09f73c42f858a2637fd75071a8ba0483f" Oct 30 03:29:22.638888 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.638824 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9a36ad00cc511313a7ba88e9b4daf09f73c42f858a2637fd75071a8ba0483f"} err="failed to get container status \"6b9a36ad00cc511313a7ba88e9b4daf09f73c42f858a2637fd75071a8ba0483f\": rpc error: code = NotFound desc = could not find container \"6b9a36ad00cc511313a7ba88e9b4daf09f73c42f858a2637fd75071a8ba0483f\": container with ID starting with 6b9a36ad00cc511313a7ba88e9b4daf09f73c42f858a2637fd75071a8ba0483f not found: ID does not exist" Oct 30 03:29:22.638888 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.638840 5882 scope.go:122] "RemoveContainer" containerID="e833576e585c8ff6378d7300d61d33827351647bfd56fc75c28bb6e08a08a6cd" Oct 30 03:29:22.643534 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.643508 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:29:22.643598 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.643536 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc69c\" (UniqueName: \"kubernetes.io/projected/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-kube-api-access-jc69c\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:29:22.643598 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.643555 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:29:22.643598 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.643570 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr6bq\" (UniqueName: \"kubernetes.io/projected/def20d1c-e93b-4e8d-9a2c-4106b158909e-kube-api-access-mr6bq\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:29:22.643738 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.643607 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 03:29:22.646889 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.646865 5882 scope.go:122] "RemoveContainer" containerID="e833576e585c8ff6378d7300d61d33827351647bfd56fc75c28bb6e08a08a6cd" Oct 30 03:29:22.647168 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:29:22.647149 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e833576e585c8ff6378d7300d61d33827351647bfd56fc75c28bb6e08a08a6cd\": container with ID starting with e833576e585c8ff6378d7300d61d33827351647bfd56fc75c28bb6e08a08a6cd not found: ID does not exist" containerID="e833576e585c8ff6378d7300d61d33827351647bfd56fc75c28bb6e08a08a6cd" Oct 30 03:29:22.647233 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.647177 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e833576e585c8ff6378d7300d61d33827351647bfd56fc75c28bb6e08a08a6cd"} err="failed to get container status \"e833576e585c8ff6378d7300d61d33827351647bfd56fc75c28bb6e08a08a6cd\": rpc error: code = NotFound desc = could not find container \"e833576e585c8ff6378d7300d61d33827351647bfd56fc75c28bb6e08a08a6cd\": container with ID starting with e833576e585c8ff6378d7300d61d33827351647bfd56fc75c28bb6e08a08a6cd not found: ID does not exist" Oct 30 03:29:22.658600 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.658584 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e") on node "ip-10-0-141-200.ec2.internal" Oct 30 03:29:22.678136 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.678112 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp"] Oct 30 03:29:22.682149 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.682124 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-85d4cc69b-bwpcp"] Oct 30 03:29:22.694708 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.694680 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def20d1c-e93b-4e8d-9a2c-4106b158909e-plugins" pod "def20d1c-e93b-4e8d-9a2c-4106b158909e" (UID: "def20d1c-e93b-4e8d-9a2c-4106b158909e"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:29:22.744997 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.744973 5882 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/def20d1c-e93b-4e8d-9a2c-4106b158909e-plugins\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:29:22.745105 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.744999 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" Oct 30 03:29:22.971974 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.971934 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:29:22.976436 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:22.976408 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:29:23.942791 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:29:23.942754 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:29:24.005033 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:24.004992 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="218de7b6-adcb-481d-966a-6ed3d9f4cc44" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.31:9093/-/ready\": dial tcp 10.134.0.31:9093: connect: connection refused" Oct 30 03:29:24.009731 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:24.009707 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2" path="/var/lib/kubelet/pods/25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2/volumes" Oct 30 03:29:24.010128 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:24.010115 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def20d1c-e93b-4e8d-9a2c-4106b158909e" path="/var/lib/kubelet/pods/def20d1c-e93b-4e8d-9a2c-4106b158909e/volumes" Oct 30 03:29:27.347280 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.347246 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:29:27.348079 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.347965 5882 cpu_manager.go:414] "containerMap: removing container" podUID="def20d1c-e93b-4e8d-9a2c-4106b158909e" containerName="perses" Oct 30 03:29:27.348079 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.347993 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def20d1c-e93b-4e8d-9a2c-4106b158909e" containerName="perses" Oct 30 03:29:27.348079 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.348016 5882 cpu_manager.go:414] "containerMap: removing container" podUID="59a868a6-a691-4aba-a0eb-c8a20eb14971" containerName="thanos-querier" Oct 30 03:29:27.348079 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.348025 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a868a6-a691-4aba-a0eb-c8a20eb14971" containerName="thanos-querier" Oct 30 03:29:27.348079 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.348034 5882 cpu_manager.go:414] "containerMap: removing container" podUID="abd12c2b-3b6f-458c-998a-aed4b6c45409" containerName="kube-rbac-proxy" Oct 30 03:29:27.348079 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.348041 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd12c2b-3b6f-458c-998a-aed4b6c45409" containerName="kube-rbac-proxy" Oct 30 03:29:27.348079 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.348055 5882 cpu_manager.go:414] "containerMap: removing container" podUID="25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2" containerName="ta-container" Oct 30 03:29:27.348079 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.348062 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2" containerName="ta-container" Oct 30 03:29:27.348430 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.348147 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="def20d1c-e93b-4e8d-9a2c-4106b158909e" containerName="perses" Oct 30 03:29:27.348430 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.348162 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a868a6-a691-4aba-a0eb-c8a20eb14971" containerName="thanos-querier" Oct 30 03:29:27.348430 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.348170 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd12c2b-3b6f-458c-998a-aed4b6c45409" containerName="kube-rbac-proxy" Oct 30 03:29:27.348430 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.348180 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="25f0e069-c9d1-47aa-a6b2-2f7a18fdb1f2" containerName="ta-container" Oct 30 03:29:27.351253 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.351232 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:27.377880 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.377854 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 03:29:27.378947 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.378925 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:29:27.397434 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.397411 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 30 03:29:27.485863 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.485822 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/107a5abe-6266-46ba-b352-54ef714795aa-config\") pod \"data-science-perses-0\" (UID: \"107a5abe-6266-46ba-b352-54ef714795aa\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:27.486024 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.485872 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/107a5abe-6266-46ba-b352-54ef714795aa-plugins\") pod \"data-science-perses-0\" (UID: \"107a5abe-6266-46ba-b352-54ef714795aa\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:27.486024 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.485891 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpq4q\" (UniqueName: \"kubernetes.io/projected/107a5abe-6266-46ba-b352-54ef714795aa-kube-api-access-cpq4q\") pod \"data-science-perses-0\" (UID: \"107a5abe-6266-46ba-b352-54ef714795aa\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:27.587251 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.587215 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/107a5abe-6266-46ba-b352-54ef714795aa-config\") pod \"data-science-perses-0\" (UID: \"107a5abe-6266-46ba-b352-54ef714795aa\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:27.587422 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.587275 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/107a5abe-6266-46ba-b352-54ef714795aa-plugins\") pod \"data-science-perses-0\" (UID: \"107a5abe-6266-46ba-b352-54ef714795aa\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:27.587422 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.587308 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpq4q\" (UniqueName: \"kubernetes.io/projected/107a5abe-6266-46ba-b352-54ef714795aa-kube-api-access-cpq4q\") pod \"data-science-perses-0\" (UID: \"107a5abe-6266-46ba-b352-54ef714795aa\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:27.587849 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.587789 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/107a5abe-6266-46ba-b352-54ef714795aa-plugins\") pod \"data-science-perses-0\" (UID: \"107a5abe-6266-46ba-b352-54ef714795aa\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:27.588017 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.587995 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/107a5abe-6266-46ba-b352-54ef714795aa-config\") pod \"data-science-perses-0\" (UID: \"107a5abe-6266-46ba-b352-54ef714795aa\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:27.611130 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:27.611057 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpq4q\" (UniqueName: \"kubernetes.io/projected/107a5abe-6266-46ba-b352-54ef714795aa-kube-api-access-cpq4q\") pod \"data-science-perses-0\" (UID: \"107a5abe-6266-46ba-b352-54ef714795aa\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:29.005015 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:29.004982 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="218de7b6-adcb-481d-966a-6ed3d9f4cc44" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.31:9093/-/ready\": dial tcp 10.134.0.31:9093: connect: connection refused" Oct 30 03:29:29.504050 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:29.504020 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"107a5abe-6266-46ba-b352-54ef714795aa\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:29.544404 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:29.544376 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"107a5abe-6266-46ba-b352-54ef714795aa\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:29.604687 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:29.604661 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"107a5abe-6266-46ba-b352-54ef714795aa\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:29.604842 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:29.604820 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"107a5abe-6266-46ba-b352-54ef714795aa\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:29.618842 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:29.618817 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"107a5abe-6266-46ba-b352-54ef714795aa\") DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:29.721006 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:29.720972 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"107a5abe-6266-46ba-b352-54ef714795aa\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2c422b71c1e0b8fdf34659f2cef7846e06b4353ed88027dc99ade5d7fc29e9d6/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:29.750820 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:29.750789 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"107a5abe-6266-46ba-b352-54ef714795aa\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:29.783275 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:29.783217 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 03:29:29.790316 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:29.790289 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:30.082677 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:30.082585 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:29:30.086406 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:29:30.086378 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod107a5abe_6266_46ba_b352_54ef714795aa.slice/crio-a3e2917327c0f47c5f4cf1e22310e42b44740f2e512ef0db2da12d1965801497 WatchSource:0}: Error finding container a3e2917327c0f47c5f4cf1e22310e42b44740f2e512ef0db2da12d1965801497: Status 404 returned error can't find the container with id a3e2917327c0f47c5f4cf1e22310e42b44740f2e512ef0db2da12d1965801497 Oct 30 03:29:30.668706 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:30.668672 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"107a5abe-6266-46ba-b352-54ef714795aa","Type":"ContainerStarted","Data":"8720d50a8cbf04192b15ed5ac249ec1b7a7a45c8419d92587989b6c5b74be074"} Oct 30 03:29:30.668706 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:30.668714 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"107a5abe-6266-46ba-b352-54ef714795aa","Type":"ContainerStarted","Data":"a3e2917327c0f47c5f4cf1e22310e42b44740f2e512ef0db2da12d1965801497"} Oct 30 03:29:30.668913 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:30.668738 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:30.709014 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:30.708967 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=3.7089530809999998 podStartE2EDuration="3.708953081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:30.707867081 +0000 UTC m=+2882.822182056" watchObservedRunningTime="2026-10-30 03:29:30.708953081 +0000 UTC m=+2882.823268047" Oct 30 03:29:31.677341 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.677312 5882 generic.go:356] "Generic (PLEG): container finished" podID="218de7b6-adcb-481d-966a-6ed3d9f4cc44" containerID="0ce735288d55dae6e91d14dc9903e3b65abdd25905afc37ae390d19d70411846" exitCode=0 Oct 30 03:29:31.677762 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.677391 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"218de7b6-adcb-481d-966a-6ed3d9f4cc44","Type":"ContainerDied","Data":"0ce735288d55dae6e91d14dc9903e3b65abdd25905afc37ae390d19d70411846"} Oct 30 03:29:31.705603 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.705542 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:29:31.824577 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.824537 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-cluster-tls-config\") pod \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " Oct 30 03:29:31.824577 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.824582 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218de7b6-adcb-481d-966a-6ed3d9f4cc44-config-out\" (UniqueName: \"kubernetes.io/empty-dir/218de7b6-adcb-481d-966a-6ed3d9f4cc44-config-out\") pod \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " Oct 30 03:29:31.824815 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.824647 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218de7b6-adcb-481d-966a-6ed3d9f4cc44-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/218de7b6-adcb-481d-966a-6ed3d9f4cc44-alertmanager-data-science-monitoringstack-db\") pod \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " Oct 30 03:29:31.824815 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.824690 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/218de7b6-adcb-481d-966a-6ed3d9f4cc44-tls-assets\" (UniqueName: \"kubernetes.io/projected/218de7b6-adcb-481d-966a-6ed3d9f4cc44-tls-assets\") pod \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " Oct 30 03:29:31.824815 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.824808 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/218de7b6-adcb-481d-966a-6ed3d9f4cc44-kube-api-access-5x7kq\" (UniqueName: \"kubernetes.io/projected/218de7b6-adcb-481d-966a-6ed3d9f4cc44-kube-api-access-5x7kq\") pod \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " Oct 30 03:29:31.824992 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.824838 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-config-volume\" (UniqueName: \"kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-config-volume\") pod \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " Oct 30 03:29:31.824992 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.824871 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-web-config\" (UniqueName: \"kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-web-config\") pod \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\" (UID: \"218de7b6-adcb-481d-966a-6ed3d9f4cc44\") " Oct 30 03:29:31.825090 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.825051 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218de7b6-adcb-481d-966a-6ed3d9f4cc44-alertmanager-data-science-monitoringstack-db" pod "218de7b6-adcb-481d-966a-6ed3d9f4cc44" (UID: "218de7b6-adcb-481d-966a-6ed3d9f4cc44"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:29:31.825425 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.825192 5882 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/218de7b6-adcb-481d-966a-6ed3d9f4cc44-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:29:31.827112 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.827060 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218de7b6-adcb-481d-966a-6ed3d9f4cc44-tls-assets" pod "218de7b6-adcb-481d-966a-6ed3d9f4cc44" (UID: "218de7b6-adcb-481d-966a-6ed3d9f4cc44"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:29:31.827322 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.827296 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218de7b6-adcb-481d-966a-6ed3d9f4cc44-config-out" pod "218de7b6-adcb-481d-966a-6ed3d9f4cc44" (UID: "218de7b6-adcb-481d-966a-6ed3d9f4cc44"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:29:31.827425 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.827315 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-config-volume" pod "218de7b6-adcb-481d-966a-6ed3d9f4cc44" (UID: "218de7b6-adcb-481d-966a-6ed3d9f4cc44"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:31.827556 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.827529 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218de7b6-adcb-481d-966a-6ed3d9f4cc44-kube-api-access-5x7kq" pod "218de7b6-adcb-481d-966a-6ed3d9f4cc44" (UID: "218de7b6-adcb-481d-966a-6ed3d9f4cc44"). InnerVolumeSpecName "kube-api-access-5x7kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:29:31.831294 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.831269 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-cluster-tls-config" pod "218de7b6-adcb-481d-966a-6ed3d9f4cc44" (UID: "218de7b6-adcb-481d-966a-6ed3d9f4cc44"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:31.838379 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.838351 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-web-config" pod "218de7b6-adcb-481d-966a-6ed3d9f4cc44" (UID: "218de7b6-adcb-481d-966a-6ed3d9f4cc44"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:31.925975 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.925939 5882 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/218de7b6-adcb-481d-966a-6ed3d9f4cc44-tls-assets\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:29:31.925975 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.925971 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x7kq\" (UniqueName: \"kubernetes.io/projected/218de7b6-adcb-481d-966a-6ed3d9f4cc44-kube-api-access-5x7kq\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:29:31.925975 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.925982 5882 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-config-volume\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:29:31.926196 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.925993 5882 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-web-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:29:31.926196 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.926001 5882 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/218de7b6-adcb-481d-966a-6ed3d9f4cc44-cluster-tls-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:29:31.926196 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:31.926009 5882 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/218de7b6-adcb-481d-966a-6ed3d9f4cc44-config-out\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:29:32.683420 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:32.683386 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:29:32.683956 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:32.683386 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"218de7b6-adcb-481d-966a-6ed3d9f4cc44","Type":"ContainerDied","Data":"2dbfa3b350194f6c48c9ac5578c3ff7c25d1c83d4c649777adbf753fd32b1b71"} Oct 30 03:29:32.683956 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:32.683528 5882 scope.go:122] "RemoveContainer" containerID="d24fe754e1b9e32406316f71480bcbd8a05619b9a1a3ba250ad06ab0cf0c137a" Oct 30 03:29:32.691417 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:32.691392 5882 scope.go:122] "RemoveContainer" containerID="0ce735288d55dae6e91d14dc9903e3b65abdd25905afc37ae390d19d70411846" Oct 30 03:29:32.698515 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:32.698498 5882 scope.go:122] "RemoveContainer" containerID="11dda493a66bfe451f352a8d8166a7ce5e309c67983c18b8d5a4c2323c15b1b8" Oct 30 03:29:32.745237 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:32.745214 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 03:29:32.756878 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:32.756856 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 03:29:34.009271 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:34.009239 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="218de7b6-adcb-481d-966a-6ed3d9f4cc44" path="/var/lib/kubelet/pods/218de7b6-adcb-481d-966a-6ed3d9f4cc44/volumes" Oct 30 03:29:41.680757 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:29:41.680686 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:29:53.941396 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:29:53.941352 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:30:20.738253 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:20.738218 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:30:20.738876 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:20.738416 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="107a5abe-6266-46ba-b352-54ef714795aa" containerName="perses" containerID="cri-o://8720d50a8cbf04192b15ed5ac249ec1b7a7a45c8419d92587989b6c5b74be074" gracePeriod=30 Oct 30 03:30:20.879721 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:20.879690 5882 generic.go:356] "Generic (PLEG): container finished" podID="107a5abe-6266-46ba-b352-54ef714795aa" containerID="8720d50a8cbf04192b15ed5ac249ec1b7a7a45c8419d92587989b6c5b74be074" exitCode=0 Oct 30 03:30:20.879900 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:20.879749 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"107a5abe-6266-46ba-b352-54ef714795aa","Type":"ContainerDied","Data":"8720d50a8cbf04192b15ed5ac249ec1b7a7a45c8419d92587989b6c5b74be074"} Oct 30 03:30:20.984245 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:20.984221 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:21.038374 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:21.038347 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107a5abe-6266-46ba-b352-54ef714795aa-kube-api-access-cpq4q\" (UniqueName: \"kubernetes.io/projected/107a5abe-6266-46ba-b352-54ef714795aa-kube-api-access-cpq4q\") pod \"107a5abe-6266-46ba-b352-54ef714795aa\" (UID: \"107a5abe-6266-46ba-b352-54ef714795aa\") " Oct 30 03:30:21.038517 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:21.038474 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"107a5abe-6266-46ba-b352-54ef714795aa\" (UID: \"107a5abe-6266-46ba-b352-54ef714795aa\") " Oct 30 03:30:21.038517 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:21.038503 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/107a5abe-6266-46ba-b352-54ef714795aa-config\" (UniqueName: \"kubernetes.io/configmap/107a5abe-6266-46ba-b352-54ef714795aa-config\") pod \"107a5abe-6266-46ba-b352-54ef714795aa\" (UID: \"107a5abe-6266-46ba-b352-54ef714795aa\") " Oct 30 03:30:21.038653 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:21.038552 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107a5abe-6266-46ba-b352-54ef714795aa-plugins\" (UniqueName: \"kubernetes.io/empty-dir/107a5abe-6266-46ba-b352-54ef714795aa-plugins\") pod \"107a5abe-6266-46ba-b352-54ef714795aa\" (UID: \"107a5abe-6266-46ba-b352-54ef714795aa\") " Oct 30 03:30:21.038896 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:21.038869 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107a5abe-6266-46ba-b352-54ef714795aa-config" pod "107a5abe-6266-46ba-b352-54ef714795aa" (UID: "107a5abe-6266-46ba-b352-54ef714795aa"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:30:21.040920 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:21.040873 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107a5abe-6266-46ba-b352-54ef714795aa-kube-api-access-cpq4q" pod "107a5abe-6266-46ba-b352-54ef714795aa" (UID: "107a5abe-6266-46ba-b352-54ef714795aa"). InnerVolumeSpecName "kube-api-access-cpq4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:30:21.041672 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:21.041645 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e" pod "107a5abe-6266-46ba-b352-54ef714795aa" (UID: "107a5abe-6266-46ba-b352-54ef714795aa"). InnerVolumeSpecName "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 03:30:21.139814 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:21.139778 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 03:30:21.139814 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:21.139812 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/107a5abe-6266-46ba-b352-54ef714795aa-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:30:21.139999 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:21.139822 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpq4q\" (UniqueName: \"kubernetes.io/projected/107a5abe-6266-46ba-b352-54ef714795aa-kube-api-access-cpq4q\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:30:21.154498 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:21.154472 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e") on node "ip-10-0-141-200.ec2.internal" Oct 30 03:30:21.199854 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:21.199820 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107a5abe-6266-46ba-b352-54ef714795aa-plugins" pod "107a5abe-6266-46ba-b352-54ef714795aa" (UID: "107a5abe-6266-46ba-b352-54ef714795aa"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:30:21.241223 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:21.241187 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" Oct 30 03:30:21.241223 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:21.241217 5882 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/107a5abe-6266-46ba-b352-54ef714795aa-plugins\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:30:21.884645 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:21.884598 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"107a5abe-6266-46ba-b352-54ef714795aa","Type":"ContainerDied","Data":"a3e2917327c0f47c5f4cf1e22310e42b44740f2e512ef0db2da12d1965801497"} Oct 30 03:30:21.884645 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:21.884647 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:21.885172 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:21.884661 5882 scope.go:122] "RemoveContainer" containerID="8720d50a8cbf04192b15ed5ac249ec1b7a7a45c8419d92587989b6c5b74be074" Oct 30 03:30:21.933212 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:21.933178 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:30:21.944583 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:21.944557 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:30:22.009348 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:22.009262 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107a5abe-6266-46ba-b352-54ef714795aa" path="/var/lib/kubelet/pods/107a5abe-6266-46ba-b352-54ef714795aa/volumes" Oct 30 03:30:23.941994 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:30:23.941961 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:30:26.021048 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.021013 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t"] Oct 30 03:30:26.021487 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.021390 5882 cpu_manager.go:414] "containerMap: removing container" podUID="218de7b6-adcb-481d-966a-6ed3d9f4cc44" containerName="config-reloader" Oct 30 03:30:26.021487 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.021405 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218de7b6-adcb-481d-966a-6ed3d9f4cc44" containerName="config-reloader" Oct 30 03:30:26.021487 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.021431 5882 cpu_manager.go:414] "containerMap: removing container" podUID="218de7b6-adcb-481d-966a-6ed3d9f4cc44" containerName="alertmanager" Oct 30 03:30:26.021487 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.021437 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218de7b6-adcb-481d-966a-6ed3d9f4cc44" containerName="alertmanager" Oct 30 03:30:26.021487 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.021453 5882 cpu_manager.go:414] "containerMap: removing container" podUID="107a5abe-6266-46ba-b352-54ef714795aa" containerName="perses" Oct 30 03:30:26.021487 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.021458 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107a5abe-6266-46ba-b352-54ef714795aa" containerName="perses" Oct 30 03:30:26.021487 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.021475 5882 cpu_manager.go:414] "containerMap: removing container" podUID="218de7b6-adcb-481d-966a-6ed3d9f4cc44" containerName="init-config-reloader" Oct 30 03:30:26.021487 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.021484 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218de7b6-adcb-481d-966a-6ed3d9f4cc44" containerName="init-config-reloader" Oct 30 03:30:26.021783 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.021541 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="218de7b6-adcb-481d-966a-6ed3d9f4cc44" containerName="config-reloader" Oct 30 03:30:26.021783 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.021550 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="218de7b6-adcb-481d-966a-6ed3d9f4cc44" containerName="alertmanager" Oct 30 03:30:26.021783 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.021558 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="107a5abe-6266-46ba-b352-54ef714795aa" containerName="perses" Oct 30 03:30:26.044637 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.044562 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t"] Oct 30 03:30:26.044799 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.044733 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t" Oct 30 03:30:26.059182 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.059153 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 03:30:26.059390 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.059370 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 03:30:26.070764 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.070744 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 03:30:26.181839 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.181802 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4xgd\" (UniqueName: \"kubernetes.io/projected/75adbe29-e89e-409e-86b1-ca7fc25e5e99-kube-api-access-k4xgd\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t\" (UID: \"75adbe29-e89e-409e-86b1-ca7fc25e5e99\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t" Oct 30 03:30:26.283004 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.282925 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4xgd\" (UniqueName: \"kubernetes.io/projected/75adbe29-e89e-409e-86b1-ca7fc25e5e99-kube-api-access-k4xgd\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t\" (UID: \"75adbe29-e89e-409e-86b1-ca7fc25e5e99\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t" Oct 30 03:30:26.305978 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.305956 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4xgd\" (UniqueName: \"kubernetes.io/projected/75adbe29-e89e-409e-86b1-ca7fc25e5e99-kube-api-access-k4xgd\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t\" (UID: \"75adbe29-e89e-409e-86b1-ca7fc25e5e99\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t" Oct 30 03:30:26.369583 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.369546 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t" Oct 30 03:30:26.658866 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.658794 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t"] Oct 30 03:30:26.662348 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:30:26.662322 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75adbe29_e89e_409e_86b1_ca7fc25e5e99.slice/crio-644c284e74ddfd3aa902f765687f18a38bfba892e4c8561908c5fd036bfff77b WatchSource:0}: Error finding container 644c284e74ddfd3aa902f765687f18a38bfba892e4c8561908c5fd036bfff77b: Status 404 returned error can't find the container with id 644c284e74ddfd3aa902f765687f18a38bfba892e4c8561908c5fd036bfff77b Oct 30 03:30:26.907810 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.907778 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t" event={"ID":"75adbe29-e89e-409e-86b1-ca7fc25e5e99","Type":"ContainerStarted","Data":"8695e14cb69caea5b4f13d91d312a51c3e914de136f88e6029e2606de00314df"} Oct 30 03:30:26.908002 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.907819 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t" event={"ID":"75adbe29-e89e-409e-86b1-ca7fc25e5e99","Type":"ContainerStarted","Data":"644c284e74ddfd3aa902f765687f18a38bfba892e4c8561908c5fd036bfff77b"} Oct 30 03:30:26.945188 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:26.945098 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t" podStartSLOduration=1.9450845380000001 podStartE2EDuration="1.945084538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:26.944720167 +0000 UTC m=+2939.059035142" watchObservedRunningTime="2026-10-30 03:30:26.945084538 +0000 UTC m=+2939.059399510" Oct 30 03:30:27.165332 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.165299 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:30:27.169111 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.169090 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:27.187890 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.187867 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:30:27.189927 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.189903 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 30 03:30:27.270678 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.270639 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b"] Oct 30 03:30:27.274356 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.274334 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" Oct 30 03:30:27.292307 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.292279 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 30 03:30:27.292307 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.292303 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-rglzc\"" Oct 30 03:30:27.294650 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.294632 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 30 03:30:27.304364 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.304343 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b"] Oct 30 03:30:27.392494 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.392452 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c680be27-27a7-4953-a502-f283c2d7ad02-config\") pod \"data-science-perses-0\" (UID: \"c680be27-27a7-4953-a502-f283c2d7ad02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:27.392701 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.392592 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/c680be27-27a7-4953-a502-f283c2d7ad02-plugins\") pod \"data-science-perses-0\" (UID: \"c680be27-27a7-4953-a502-f283c2d7ad02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:27.392819 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.392792 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8dmm\" (UniqueName: \"kubernetes.io/projected/c680be27-27a7-4953-a502-f283c2d7ad02-kube-api-access-z8dmm\") pod \"data-science-perses-0\" (UID: \"c680be27-27a7-4953-a502-f283c2d7ad02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:27.493954 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.493919 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/ba5b59af-7d63-4eca-8a99-def653dad11c-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-84d679d984-6vn7b\" (UID: \"ba5b59af-7d63-4eca-8a99-def653dad11c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" Oct 30 03:30:27.494155 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.493967 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/c680be27-27a7-4953-a502-f283c2d7ad02-plugins\") pod \"data-science-perses-0\" (UID: \"c680be27-27a7-4953-a502-f283c2d7ad02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:27.494155 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.494040 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dmm\" (UniqueName: \"kubernetes.io/projected/c680be27-27a7-4953-a502-f283c2d7ad02-kube-api-access-z8dmm\") pod \"data-science-perses-0\" (UID: \"c680be27-27a7-4953-a502-f283c2d7ad02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:27.494155 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.494126 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv88k\" (UniqueName: \"kubernetes.io/projected/ba5b59af-7d63-4eca-8a99-def653dad11c-kube-api-access-jv88k\") pod \"data-science-collector-targetallocator-84d679d984-6vn7b\" (UID: \"ba5b59af-7d63-4eca-8a99-def653dad11c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" Oct 30 03:30:27.494304 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.494170 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/ba5b59af-7d63-4eca-8a99-def653dad11c-ta-internal\") pod \"data-science-collector-targetallocator-84d679d984-6vn7b\" (UID: \"ba5b59af-7d63-4eca-8a99-def653dad11c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" Oct 30 03:30:27.494304 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.494217 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c680be27-27a7-4953-a502-f283c2d7ad02-config\") pod \"data-science-perses-0\" (UID: \"c680be27-27a7-4953-a502-f283c2d7ad02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:27.494712 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.494680 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/c680be27-27a7-4953-a502-f283c2d7ad02-plugins\") pod \"data-science-perses-0\" (UID: \"c680be27-27a7-4953-a502-f283c2d7ad02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:27.494979 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.494957 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c680be27-27a7-4953-a502-f283c2d7ad02-config\") pod \"data-science-perses-0\" (UID: \"c680be27-27a7-4953-a502-f283c2d7ad02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:27.517011 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.516984 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8dmm\" (UniqueName: \"kubernetes.io/projected/c680be27-27a7-4953-a502-f283c2d7ad02-kube-api-access-z8dmm\") pod \"data-science-perses-0\" (UID: \"c680be27-27a7-4953-a502-f283c2d7ad02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:27.595519 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.595452 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv88k\" (UniqueName: \"kubernetes.io/projected/ba5b59af-7d63-4eca-8a99-def653dad11c-kube-api-access-jv88k\") pod \"data-science-collector-targetallocator-84d679d984-6vn7b\" (UID: \"ba5b59af-7d63-4eca-8a99-def653dad11c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" Oct 30 03:30:27.595519 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.595497 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/ba5b59af-7d63-4eca-8a99-def653dad11c-ta-internal\") pod \"data-science-collector-targetallocator-84d679d984-6vn7b\" (UID: \"ba5b59af-7d63-4eca-8a99-def653dad11c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" Oct 30 03:30:27.595756 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.595569 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/ba5b59af-7d63-4eca-8a99-def653dad11c-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-84d679d984-6vn7b\" (UID: \"ba5b59af-7d63-4eca-8a99-def653dad11c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" Oct 30 03:30:27.596182 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.596156 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/ba5b59af-7d63-4eca-8a99-def653dad11c-ta-internal\") pod \"data-science-collector-targetallocator-84d679d984-6vn7b\" (UID: \"ba5b59af-7d63-4eca-8a99-def653dad11c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" Oct 30 03:30:27.597923 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.597904 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/ba5b59af-7d63-4eca-8a99-def653dad11c-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-84d679d984-6vn7b\" (UID: \"ba5b59af-7d63-4eca-8a99-def653dad11c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" Oct 30 03:30:27.620781 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.620759 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv88k\" (UniqueName: \"kubernetes.io/projected/ba5b59af-7d63-4eca-8a99-def653dad11c-kube-api-access-jv88k\") pod \"data-science-collector-targetallocator-84d679d984-6vn7b\" (UID: \"ba5b59af-7d63-4eca-8a99-def653dad11c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" Oct 30 03:30:27.886116 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:27.886037 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" Oct 30 03:30:28.175595 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:28.175515 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b"] Oct 30 03:30:28.179310 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:30:28.179286 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba5b59af_7d63_4eca_8a99_def653dad11c.slice/crio-4bf0e639f6848fe35cdb58df79aaf42e5aa86612ae97206a10742d9feec146d6 WatchSource:0}: Error finding container 4bf0e639f6848fe35cdb58df79aaf42e5aa86612ae97206a10742d9feec146d6: Status 404 returned error can't find the container with id 4bf0e639f6848fe35cdb58df79aaf42e5aa86612ae97206a10742d9feec146d6 Oct 30 03:30:28.181321 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:28.181293 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:30:28.181377 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:28.181347 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:30:28.918907 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:28.918876 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" event={"ID":"ba5b59af-7d63-4eca-8a99-def653dad11c","Type":"ContainerStarted","Data":"d0b22edaab786436449b08daf0133f8e5ed99fa3a9fe5a253165c543fc2a8f06"} Oct 30 03:30:28.919107 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:28.918920 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" event={"ID":"ba5b59af-7d63-4eca-8a99-def653dad11c","Type":"ContainerStarted","Data":"4bf0e639f6848fe35cdb58df79aaf42e5aa86612ae97206a10742d9feec146d6"} Oct 30 03:30:28.919107 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:28.919041 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" Oct 30 03:30:28.920842 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:28.920821 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" Oct 30 03:30:28.980083 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:28.980030 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" podStartSLOduration=1.980016654 podStartE2EDuration="1.980016654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:28.979849331 +0000 UTC m=+2941.094164305" watchObservedRunningTime="2026-10-30 03:30:28.980016654 +0000 UTC m=+2941.094331629" Oct 30 03:30:30.517487 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:30.517446 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"c680be27-27a7-4953-a502-f283c2d7ad02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:30.557134 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:30.557104 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"c680be27-27a7-4953-a502-f283c2d7ad02\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:30.571732 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:30.571680 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4"] Oct 30 03:30:30.576023 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:30.576004 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4" Oct 30 03:30:30.593191 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:30.593170 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4"] Oct 30 03:30:30.617932 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:30.617902 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2k7d\" (UniqueName: \"kubernetes.io/projected/ef122d6e-c44e-420b-a77d-d6c7171048b6-kube-api-access-g2k7d\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4\" (UID: \"ef122d6e-c44e-420b-a77d-d6c7171048b6\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4" Oct 30 03:30:30.618058 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:30.617970 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"c680be27-27a7-4953-a502-f283c2d7ad02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:30.618106 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:30.618090 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"c680be27-27a7-4953-a502-f283c2d7ad02\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:30.631752 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:30.631727 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"c680be27-27a7-4953-a502-f283c2d7ad02\") DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:30.719234 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:30.719204 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2k7d\" (UniqueName: \"kubernetes.io/projected/ef122d6e-c44e-420b-a77d-d6c7171048b6-kube-api-access-g2k7d\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4\" (UID: \"ef122d6e-c44e-420b-a77d-d6c7171048b6\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4" Oct 30 03:30:30.731870 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:30.731841 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"c680be27-27a7-4953-a502-f283c2d7ad02\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2c422b71c1e0b8fdf34659f2cef7846e06b4353ed88027dc99ade5d7fc29e9d6/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:30.741978 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:30.741954 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2k7d\" (UniqueName: \"kubernetes.io/projected/ef122d6e-c44e-420b-a77d-d6c7171048b6-kube-api-access-g2k7d\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4\" (UID: \"ef122d6e-c44e-420b-a77d-d6c7171048b6\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4" Oct 30 03:30:30.760353 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:30.760330 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"c680be27-27a7-4953-a502-f283c2d7ad02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:30.781178 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:30.781122 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:30.888027 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:30.887998 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4" Oct 30 03:30:31.069297 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.069178 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:30:31.072883 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:30:31.072847 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc680be27_27a7_4953_a502_f283c2d7ad02.slice/crio-93abd4371caa4e91a827d5bbe495e7220aac24130d60b7368843966a952a3309 WatchSource:0}: Error finding container 93abd4371caa4e91a827d5bbe495e7220aac24130d60b7368843966a952a3309: Status 404 returned error can't find the container with id 93abd4371caa4e91a827d5bbe495e7220aac24130d60b7368843966a952a3309 Oct 30 03:30:31.176664 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.176607 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4"] Oct 30 03:30:31.180428 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:30:31.180395 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef122d6e_c44e_420b_a77d_d6c7171048b6.slice/crio-d81c7dff145d99c64abe5414e97e4b164331413eba1e5e93f279f23ffa3763d6 WatchSource:0}: Error finding container d81c7dff145d99c64abe5414e97e4b164331413eba1e5e93f279f23ffa3763d6: Status 404 returned error can't find the container with id d81c7dff145d99c64abe5414e97e4b164331413eba1e5e93f279f23ffa3763d6 Oct 30 03:30:31.430988 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.430902 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 03:30:31.438499 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.438472 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.460391 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.460357 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 03:30:31.521879 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.521846 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 03:30:31.522278 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.521896 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 03:30:31.522278 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.521921 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 03:30:31.522278 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.521964 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 03:30:31.522278 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.521977 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-zdtxd\"" Oct 30 03:30:31.528918 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.528890 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.529915 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.528941 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.529915 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.529682 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2rgz\" (UniqueName: \"kubernetes.io/projected/189e5984-4f6c-45fd-a40f-9a029341eb1c-kube-api-access-x2rgz\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.529915 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.529743 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/189e5984-4f6c-45fd-a40f-9a029341eb1c-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.529915 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.529828 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/189e5984-4f6c-45fd-a40f-9a029341eb1c-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.529915 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.529858 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/189e5984-4f6c-45fd-a40f-9a029341eb1c-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.529915 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.529897 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.631115 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.631079 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/189e5984-4f6c-45fd-a40f-9a029341eb1c-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.631115 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.631117 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/189e5984-4f6c-45fd-a40f-9a029341eb1c-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.631316 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.631152 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.631316 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.631251 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.631316 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.631285 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.631316 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.631308 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2rgz\" (UniqueName: \"kubernetes.io/projected/189e5984-4f6c-45fd-a40f-9a029341eb1c-kube-api-access-x2rgz\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.631526 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.631347 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/189e5984-4f6c-45fd-a40f-9a029341eb1c-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.631789 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.631734 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/189e5984-4f6c-45fd-a40f-9a029341eb1c-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.633842 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.633795 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/189e5984-4f6c-45fd-a40f-9a029341eb1c-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.633989 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.633931 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/189e5984-4f6c-45fd-a40f-9a029341eb1c-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.634491 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.634089 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.634491 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.634415 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.634601 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.634501 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.654033 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.654006 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2rgz\" (UniqueName: \"kubernetes.io/projected/189e5984-4f6c-45fd-a40f-9a029341eb1c-kube-api-access-x2rgz\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.756284 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.756249 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:31.935246 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.935191 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"c680be27-27a7-4953-a502-f283c2d7ad02","Type":"ContainerStarted","Data":"0a584c2a9838ba7db41f87871bd28fd4dfb9bfc69fb47f8b77598f8428e5d0cb"} Oct 30 03:30:31.935450 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.935261 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:31.935450 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.935278 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"c680be27-27a7-4953-a502-f283c2d7ad02","Type":"ContainerStarted","Data":"93abd4371caa4e91a827d5bbe495e7220aac24130d60b7368843966a952a3309"} Oct 30 03:30:31.936821 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.936797 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4" event={"ID":"ef122d6e-c44e-420b-a77d-d6c7171048b6","Type":"ContainerStarted","Data":"0b73b00f1143d0abfd5d3f5487fba678f0b5c98fc64e0e8321f8f22c3f17ec01"} Oct 30 03:30:31.936952 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.936825 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4" event={"ID":"ef122d6e-c44e-420b-a77d-d6c7171048b6","Type":"ContainerStarted","Data":"d81c7dff145d99c64abe5414e97e4b164331413eba1e5e93f279f23ffa3763d6"} Oct 30 03:30:31.974751 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:31.974687 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=4.97466989 podStartE2EDuration="4.97466989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:31.973894298 +0000 UTC m=+2944.088209273" watchObservedRunningTime="2026-10-30 03:30:31.97466989 +0000 UTC m=+2944.088984862" Oct 30 03:30:32.011480 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:32.011389 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4" podStartSLOduration=2.011376315 podStartE2EDuration="2.011376315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:32.011112451 +0000 UTC m=+2944.125427427" watchObservedRunningTime="2026-10-30 03:30:32.011376315 +0000 UTC m=+2944.125691290" Oct 30 03:30:32.036268 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:32.036187 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t"] Oct 30 03:30:32.036490 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:32.036459 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t" podUID="75adbe29-e89e-409e-86b1-ca7fc25e5e99" containerName="thanos-querier" containerID="cri-o://8695e14cb69caea5b4f13d91d312a51c3e914de136f88e6029e2606de00314df" gracePeriod=30 Oct 30 03:30:32.072925 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:32.072894 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 03:30:32.115571 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:30:32.115534 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189e5984_4f6c_45fd_a40f_9a029341eb1c.slice/crio-03daa462aa4712ead49095d5ecdd43119dc08bcd9f796369b24c4b71901bdcc4 WatchSource:0}: Error finding container 03daa462aa4712ead49095d5ecdd43119dc08bcd9f796369b24c4b71901bdcc4: Status 404 returned error can't find the container with id 03daa462aa4712ead49095d5ecdd43119dc08bcd9f796369b24c4b71901bdcc4 Oct 30 03:30:32.284374 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:32.284345 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t" Oct 30 03:30:32.438491 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:32.438455 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75adbe29-e89e-409e-86b1-ca7fc25e5e99-kube-api-access-k4xgd\" (UniqueName: \"kubernetes.io/projected/75adbe29-e89e-409e-86b1-ca7fc25e5e99-kube-api-access-k4xgd\") pod \"75adbe29-e89e-409e-86b1-ca7fc25e5e99\" (UID: \"75adbe29-e89e-409e-86b1-ca7fc25e5e99\") " Oct 30 03:30:32.440495 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:32.440468 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75adbe29-e89e-409e-86b1-ca7fc25e5e99-kube-api-access-k4xgd" pod "75adbe29-e89e-409e-86b1-ca7fc25e5e99" (UID: "75adbe29-e89e-409e-86b1-ca7fc25e5e99"). InnerVolumeSpecName "kube-api-access-k4xgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:30:32.540074 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:32.539998 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4xgd\" (UniqueName: \"kubernetes.io/projected/75adbe29-e89e-409e-86b1-ca7fc25e5e99-kube-api-access-k4xgd\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:30:32.943919 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:32.943838 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"189e5984-4f6c-45fd-a40f-9a029341eb1c","Type":"ContainerStarted","Data":"ca3540eaa146f872580393153024f6cb3d42be975d0b4a46849e1a8ddf6e112b"} Oct 30 03:30:32.943919 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:32.943910 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"189e5984-4f6c-45fd-a40f-9a029341eb1c","Type":"ContainerStarted","Data":"03daa462aa4712ead49095d5ecdd43119dc08bcd9f796369b24c4b71901bdcc4"} Oct 30 03:30:32.945183 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:32.945152 5882 generic.go:356] "Generic (PLEG): container finished" podID="75adbe29-e89e-409e-86b1-ca7fc25e5e99" containerID="8695e14cb69caea5b4f13d91d312a51c3e914de136f88e6029e2606de00314df" exitCode=0 Oct 30 03:30:32.945288 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:32.945191 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t" Oct 30 03:30:32.945288 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:32.945229 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t" event={"ID":"75adbe29-e89e-409e-86b1-ca7fc25e5e99","Type":"ContainerDied","Data":"8695e14cb69caea5b4f13d91d312a51c3e914de136f88e6029e2606de00314df"} Oct 30 03:30:32.945288 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:32.945256 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t" event={"ID":"75adbe29-e89e-409e-86b1-ca7fc25e5e99","Type":"ContainerDied","Data":"644c284e74ddfd3aa902f765687f18a38bfba892e4c8561908c5fd036bfff77b"} Oct 30 03:30:32.945288 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:32.945272 5882 scope.go:122] "RemoveContainer" containerID="8695e14cb69caea5b4f13d91d312a51c3e914de136f88e6029e2606de00314df" Oct 30 03:30:32.954085 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:32.954064 5882 scope.go:122] "RemoveContainer" containerID="8695e14cb69caea5b4f13d91d312a51c3e914de136f88e6029e2606de00314df" Oct 30 03:30:32.954349 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:30:32.954329 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8695e14cb69caea5b4f13d91d312a51c3e914de136f88e6029e2606de00314df\": container with ID starting with 8695e14cb69caea5b4f13d91d312a51c3e914de136f88e6029e2606de00314df not found: ID does not exist" containerID="8695e14cb69caea5b4f13d91d312a51c3e914de136f88e6029e2606de00314df" Oct 30 03:30:32.954434 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:32.954355 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8695e14cb69caea5b4f13d91d312a51c3e914de136f88e6029e2606de00314df"} err="failed to get container status \"8695e14cb69caea5b4f13d91d312a51c3e914de136f88e6029e2606de00314df\": rpc error: code = NotFound desc = could not find container \"8695e14cb69caea5b4f13d91d312a51c3e914de136f88e6029e2606de00314df\": container with ID starting with 8695e14cb69caea5b4f13d91d312a51c3e914de136f88e6029e2606de00314df not found: ID does not exist" Oct 30 03:30:33.024148 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:33.023754 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t"] Oct 30 03:30:33.027873 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:33.027847 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-mt87t"] Oct 30 03:30:34.009116 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:34.009081 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75adbe29-e89e-409e-86b1-ca7fc25e5e99" path="/var/lib/kubelet/pods/75adbe29-e89e-409e-86b1-ca7fc25e5e99/volumes" Oct 30 03:30:36.967870 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:36.967835 5882 generic.go:356] "Generic (PLEG): container finished" podID="189e5984-4f6c-45fd-a40f-9a029341eb1c" containerID="ca3540eaa146f872580393153024f6cb3d42be975d0b4a46849e1a8ddf6e112b" exitCode=0 Oct 30 03:30:36.968300 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:36.968272 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"189e5984-4f6c-45fd-a40f-9a029341eb1c","Type":"ContainerDied","Data":"ca3540eaa146f872580393153024f6cb3d42be975d0b4a46849e1a8ddf6e112b"} Oct 30 03:30:37.974904 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:37.974821 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"189e5984-4f6c-45fd-a40f-9a029341eb1c","Type":"ContainerStarted","Data":"c1e6abf370269ad25cbbbac56a67ec32c686b896502966565623c7a12c03b405"} Oct 30 03:30:37.974904 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:37.974879 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"189e5984-4f6c-45fd-a40f-9a029341eb1c","Type":"ContainerStarted","Data":"8dcdc26591131c77b75ad781018fdb32de183c9a40b6c96d395f476664ea8c98"} Oct 30 03:30:37.975360 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:37.975059 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:38.044518 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:38.044471 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.455395037 podStartE2EDuration="7.044458242s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.589063205s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:38.032838504 +0000 UTC m=+2950.147153475" watchObservedRunningTime="2026-10-30 03:30:38.044458242 +0000 UTC m=+2950.158773214" Oct 30 03:30:42.949029 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:42.949000 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:30:43.981685 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:30:43.981652 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:30:53.942140 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:30:53.942107 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:31:23.943378 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:31:23.943298 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:31:28.283824 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:31:28.283704 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:31:28.292495 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:31:28.292475 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:31:53.943067 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:31:53.943021 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:32:23.942673 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:32:23.942634 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:32:53.943014 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:32:53.942975 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:33:23.942353 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:33:23.942296 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:33:53.943088 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:33:53.943048 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:34:23.942978 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:34:23.942880 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:34:53.945354 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:34:53.945314 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:35:23.943597 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:35:23.943561 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:35:53.942131 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:35:53.942094 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:36:23.943393 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:36:23.943355 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:36:28.315672 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:36:28.315521 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:36:28.325248 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:36:28.325227 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:36:53.942380 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:36:53.942338 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:37:23.944005 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:37:23.943926 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:37:53.942836 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:37:53.942792 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:38:23.941673 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:38:23.941637 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:38:53.942526 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:38:53.942479 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:39:23.942981 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:39:23.942945 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:39:53.942816 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:39:53.942776 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:40:23.941990 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:40:23.941949 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:40:53.943518 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:40:53.943473 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:41:21.750549 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:21.750513 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4"] Oct 30 03:41:21.751006 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:21.750757 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4" podUID="ef122d6e-c44e-420b-a77d-d6c7171048b6" containerName="thanos-querier" containerID="cri-o://0b73b00f1143d0abfd5d3f5487fba678f0b5c98fc64e0e8321f8f22c3f17ec01" gracePeriod=30 Oct 30 03:41:21.905488 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:21.903399 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 03:41:21.905488 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:21.903727 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="189e5984-4f6c-45fd-a40f-9a029341eb1c" containerName="alertmanager" containerID="cri-o://8dcdc26591131c77b75ad781018fdb32de183c9a40b6c96d395f476664ea8c98" gracePeriod=120 Oct 30 03:41:21.905488 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:21.904066 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="189e5984-4f6c-45fd-a40f-9a029341eb1c" containerName="config-reloader" containerID="cri-o://c1e6abf370269ad25cbbbac56a67ec32c686b896502966565623c7a12c03b405" gracePeriod=120 Oct 30 03:41:22.103888 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:22.103865 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4" Oct 30 03:41:22.195201 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:22.195162 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef122d6e-c44e-420b-a77d-d6c7171048b6-kube-api-access-g2k7d\" (UniqueName: \"kubernetes.io/projected/ef122d6e-c44e-420b-a77d-d6c7171048b6-kube-api-access-g2k7d\") pod \"ef122d6e-c44e-420b-a77d-d6c7171048b6\" (UID: \"ef122d6e-c44e-420b-a77d-d6c7171048b6\") " Oct 30 03:41:22.197286 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:22.197262 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef122d6e-c44e-420b-a77d-d6c7171048b6-kube-api-access-g2k7d" pod "ef122d6e-c44e-420b-a77d-d6c7171048b6" (UID: "ef122d6e-c44e-420b-a77d-d6c7171048b6"). InnerVolumeSpecName "kube-api-access-g2k7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:41:22.296077 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:22.296049 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2k7d\" (UniqueName: \"kubernetes.io/projected/ef122d6e-c44e-420b-a77d-d6c7171048b6-kube-api-access-g2k7d\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:41:22.744511 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:22.744479 5882 generic.go:356] "Generic (PLEG): container finished" podID="189e5984-4f6c-45fd-a40f-9a029341eb1c" containerID="c1e6abf370269ad25cbbbac56a67ec32c686b896502966565623c7a12c03b405" exitCode=0 Oct 30 03:41:22.744730 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:22.744570 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"189e5984-4f6c-45fd-a40f-9a029341eb1c","Type":"ContainerDied","Data":"c1e6abf370269ad25cbbbac56a67ec32c686b896502966565623c7a12c03b405"} Oct 30 03:41:22.746213 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:22.746184 5882 generic.go:356] "Generic (PLEG): container finished" podID="ef122d6e-c44e-420b-a77d-d6c7171048b6" containerID="0b73b00f1143d0abfd5d3f5487fba678f0b5c98fc64e0e8321f8f22c3f17ec01" exitCode=0 Oct 30 03:41:22.746357 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:22.746244 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4" Oct 30 03:41:22.746357 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:22.746271 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4" event={"ID":"ef122d6e-c44e-420b-a77d-d6c7171048b6","Type":"ContainerDied","Data":"0b73b00f1143d0abfd5d3f5487fba678f0b5c98fc64e0e8321f8f22c3f17ec01"} Oct 30 03:41:22.746357 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:22.746299 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4" event={"ID":"ef122d6e-c44e-420b-a77d-d6c7171048b6","Type":"ContainerDied","Data":"d81c7dff145d99c64abe5414e97e4b164331413eba1e5e93f279f23ffa3763d6"} Oct 30 03:41:22.746357 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:22.746323 5882 scope.go:122] "RemoveContainer" containerID="0b73b00f1143d0abfd5d3f5487fba678f0b5c98fc64e0e8321f8f22c3f17ec01" Oct 30 03:41:22.762572 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:22.762554 5882 scope.go:122] "RemoveContainer" containerID="0b73b00f1143d0abfd5d3f5487fba678f0b5c98fc64e0e8321f8f22c3f17ec01" Oct 30 03:41:22.762902 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:41:22.762861 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b73b00f1143d0abfd5d3f5487fba678f0b5c98fc64e0e8321f8f22c3f17ec01\": container with ID starting with 0b73b00f1143d0abfd5d3f5487fba678f0b5c98fc64e0e8321f8f22c3f17ec01 not found: ID does not exist" containerID="0b73b00f1143d0abfd5d3f5487fba678f0b5c98fc64e0e8321f8f22c3f17ec01" Oct 30 03:41:22.762947 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:22.762898 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b73b00f1143d0abfd5d3f5487fba678f0b5c98fc64e0e8321f8f22c3f17ec01"} err="failed to get container status \"0b73b00f1143d0abfd5d3f5487fba678f0b5c98fc64e0e8321f8f22c3f17ec01\": rpc error: code = NotFound desc = could not find container \"0b73b00f1143d0abfd5d3f5487fba678f0b5c98fc64e0e8321f8f22c3f17ec01\": container with ID starting with 0b73b00f1143d0abfd5d3f5487fba678f0b5c98fc64e0e8321f8f22c3f17ec01 not found: ID does not exist" Oct 30 03:41:22.778668 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:22.778640 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b"] Oct 30 03:41:22.778882 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:22.778843 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" podUID="ba5b59af-7d63-4eca-8a99-def653dad11c" containerName="ta-container" containerID="cri-o://d0b22edaab786436449b08daf0133f8e5ed99fa3a9fe5a253165c543fc2a8f06" gracePeriod=30 Oct 30 03:41:22.804142 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:22.804115 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4"] Oct 30 03:41:22.809768 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:22.809746 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-pr6b4"] Oct 30 03:41:22.903147 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:22.903107 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:41:22.904282 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:22.903357 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="c680be27-27a7-4953-a502-f283c2d7ad02" containerName="perses" containerID="cri-o://0a584c2a9838ba7db41f87871bd28fd4dfb9bfc69fb47f8b77598f8428e5d0cb" gracePeriod=30 Oct 30 03:41:22.946844 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:22.946806 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-perses-0" podUID="c680be27-27a7-4953-a502-f283c2d7ad02" containerName="perses" probeResult="failure" output="Get \"http://10.134.0.34:8080/metrics\": dial tcp 10.134.0.34:8080: connect: connection refused" Oct 30 03:41:23.033645 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.033593 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" Oct 30 03:41:23.104126 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.104093 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba5b59af-7d63-4eca-8a99-def653dad11c-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/ba5b59af-7d63-4eca-8a99-def653dad11c-data-science-collector-ta-server-cert\") pod \"ba5b59af-7d63-4eca-8a99-def653dad11c\" (UID: \"ba5b59af-7d63-4eca-8a99-def653dad11c\") " Oct 30 03:41:23.104308 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.104161 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5b59af-7d63-4eca-8a99-def653dad11c-kube-api-access-jv88k\" (UniqueName: \"kubernetes.io/projected/ba5b59af-7d63-4eca-8a99-def653dad11c-kube-api-access-jv88k\") pod \"ba5b59af-7d63-4eca-8a99-def653dad11c\" (UID: \"ba5b59af-7d63-4eca-8a99-def653dad11c\") " Oct 30 03:41:23.104373 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.104319 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba5b59af-7d63-4eca-8a99-def653dad11c-ta-internal\" (UniqueName: \"kubernetes.io/configmap/ba5b59af-7d63-4eca-8a99-def653dad11c-ta-internal\") pod \"ba5b59af-7d63-4eca-8a99-def653dad11c\" (UID: \"ba5b59af-7d63-4eca-8a99-def653dad11c\") " Oct 30 03:41:23.104747 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.104715 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5b59af-7d63-4eca-8a99-def653dad11c-ta-internal" pod "ba5b59af-7d63-4eca-8a99-def653dad11c" (UID: "ba5b59af-7d63-4eca-8a99-def653dad11c"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:41:23.106787 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.106761 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5b59af-7d63-4eca-8a99-def653dad11c-kube-api-access-jv88k" pod "ba5b59af-7d63-4eca-8a99-def653dad11c" (UID: "ba5b59af-7d63-4eca-8a99-def653dad11c"). InnerVolumeSpecName "kube-api-access-jv88k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:41:23.106920 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.106839 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba5b59af-7d63-4eca-8a99-def653dad11c-data-science-collector-ta-server-cert" pod "ba5b59af-7d63-4eca-8a99-def653dad11c" (UID: "ba5b59af-7d63-4eca-8a99-def653dad11c"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:41:23.139927 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.139901 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:23.205929 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.205898 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c680be27-27a7-4953-a502-f283c2d7ad02-kube-api-access-z8dmm\" (UniqueName: \"kubernetes.io/projected/c680be27-27a7-4953-a502-f283c2d7ad02-kube-api-access-z8dmm\") pod \"c680be27-27a7-4953-a502-f283c2d7ad02\" (UID: \"c680be27-27a7-4953-a502-f283c2d7ad02\") " Oct 30 03:41:23.206099 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.206013 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c680be27-27a7-4953-a502-f283c2d7ad02-config\" (UniqueName: \"kubernetes.io/configmap/c680be27-27a7-4953-a502-f283c2d7ad02-config\") pod \"c680be27-27a7-4953-a502-f283c2d7ad02\" (UID: \"c680be27-27a7-4953-a502-f283c2d7ad02\") " Oct 30 03:41:23.206099 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.206047 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c680be27-27a7-4953-a502-f283c2d7ad02-plugins\" (UniqueName: \"kubernetes.io/empty-dir/c680be27-27a7-4953-a502-f283c2d7ad02-plugins\") pod \"c680be27-27a7-4953-a502-f283c2d7ad02\" (UID: \"c680be27-27a7-4953-a502-f283c2d7ad02\") " Oct 30 03:41:23.206218 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.206146 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"c680be27-27a7-4953-a502-f283c2d7ad02\" (UID: \"c680be27-27a7-4953-a502-f283c2d7ad02\") " Oct 30 03:41:23.206405 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.206381 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c680be27-27a7-4953-a502-f283c2d7ad02-config" pod "c680be27-27a7-4953-a502-f283c2d7ad02" (UID: "c680be27-27a7-4953-a502-f283c2d7ad02"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:41:23.206405 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.206401 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/ba5b59af-7d63-4eca-8a99-def653dad11c-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:41:23.206507 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.206415 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/ba5b59af-7d63-4eca-8a99-def653dad11c-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:41:23.206507 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.206425 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv88k\" (UniqueName: \"kubernetes.io/projected/ba5b59af-7d63-4eca-8a99-def653dad11c-kube-api-access-jv88k\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:41:23.208260 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.208232 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c680be27-27a7-4953-a502-f283c2d7ad02-kube-api-access-z8dmm" pod "c680be27-27a7-4953-a502-f283c2d7ad02" (UID: "c680be27-27a7-4953-a502-f283c2d7ad02"). InnerVolumeSpecName "kube-api-access-z8dmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:41:23.209165 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.209138 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e" pod "c680be27-27a7-4953-a502-f283c2d7ad02" (UID: "c680be27-27a7-4953-a502-f283c2d7ad02"). InnerVolumeSpecName "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 03:41:23.307791 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.307726 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c680be27-27a7-4953-a502-f283c2d7ad02-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:41:23.307791 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.307773 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 03:41:23.307791 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.307785 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8dmm\" (UniqueName: \"kubernetes.io/projected/c680be27-27a7-4953-a502-f283c2d7ad02-kube-api-access-z8dmm\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:41:23.320970 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.320949 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e") on node "ip-10-0-141-200.ec2.internal" Oct 30 03:41:23.354624 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.354589 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c680be27-27a7-4953-a502-f283c2d7ad02-plugins" pod "c680be27-27a7-4953-a502-f283c2d7ad02" (UID: "c680be27-27a7-4953-a502-f283c2d7ad02"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:41:23.408663 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.408637 5882 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/c680be27-27a7-4953-a502-f283c2d7ad02-plugins\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:41:23.408663 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.408663 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" Oct 30 03:41:23.751848 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.751816 5882 generic.go:356] "Generic (PLEG): container finished" podID="c680be27-27a7-4953-a502-f283c2d7ad02" containerID="0a584c2a9838ba7db41f87871bd28fd4dfb9bfc69fb47f8b77598f8428e5d0cb" exitCode=0 Oct 30 03:41:23.752006 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.751894 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:23.752006 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.751893 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"c680be27-27a7-4953-a502-f283c2d7ad02","Type":"ContainerDied","Data":"0a584c2a9838ba7db41f87871bd28fd4dfb9bfc69fb47f8b77598f8428e5d0cb"} Oct 30 03:41:23.752122 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.752007 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"c680be27-27a7-4953-a502-f283c2d7ad02","Type":"ContainerDied","Data":"93abd4371caa4e91a827d5bbe495e7220aac24130d60b7368843966a952a3309"} Oct 30 03:41:23.752122 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.752038 5882 scope.go:122] "RemoveContainer" containerID="0a584c2a9838ba7db41f87871bd28fd4dfb9bfc69fb47f8b77598f8428e5d0cb" Oct 30 03:41:23.754399 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.754377 5882 generic.go:356] "Generic (PLEG): container finished" podID="ba5b59af-7d63-4eca-8a99-def653dad11c" containerID="d0b22edaab786436449b08daf0133f8e5ed99fa3a9fe5a253165c543fc2a8f06" exitCode=0 Oct 30 03:41:23.754513 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.754404 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" event={"ID":"ba5b59af-7d63-4eca-8a99-def653dad11c","Type":"ContainerDied","Data":"d0b22edaab786436449b08daf0133f8e5ed99fa3a9fe5a253165c543fc2a8f06"} Oct 30 03:41:23.754513 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.754426 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" event={"ID":"ba5b59af-7d63-4eca-8a99-def653dad11c","Type":"ContainerDied","Data":"4bf0e639f6848fe35cdb58df79aaf42e5aa86612ae97206a10742d9feec146d6"} Oct 30 03:41:23.754513 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.754443 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b" Oct 30 03:41:23.761751 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.761568 5882 scope.go:122] "RemoveContainer" containerID="0a584c2a9838ba7db41f87871bd28fd4dfb9bfc69fb47f8b77598f8428e5d0cb" Oct 30 03:41:23.761880 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:41:23.761860 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a584c2a9838ba7db41f87871bd28fd4dfb9bfc69fb47f8b77598f8428e5d0cb\": container with ID starting with 0a584c2a9838ba7db41f87871bd28fd4dfb9bfc69fb47f8b77598f8428e5d0cb not found: ID does not exist" containerID="0a584c2a9838ba7db41f87871bd28fd4dfb9bfc69fb47f8b77598f8428e5d0cb" Oct 30 03:41:23.761922 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.761893 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a584c2a9838ba7db41f87871bd28fd4dfb9bfc69fb47f8b77598f8428e5d0cb"} err="failed to get container status \"0a584c2a9838ba7db41f87871bd28fd4dfb9bfc69fb47f8b77598f8428e5d0cb\": rpc error: code = NotFound desc = could not find container \"0a584c2a9838ba7db41f87871bd28fd4dfb9bfc69fb47f8b77598f8428e5d0cb\": container with ID starting with 0a584c2a9838ba7db41f87871bd28fd4dfb9bfc69fb47f8b77598f8428e5d0cb not found: ID does not exist" Oct 30 03:41:23.761922 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.761915 5882 scope.go:122] "RemoveContainer" containerID="d0b22edaab786436449b08daf0133f8e5ed99fa3a9fe5a253165c543fc2a8f06" Oct 30 03:41:23.769586 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.769468 5882 scope.go:122] "RemoveContainer" containerID="d0b22edaab786436449b08daf0133f8e5ed99fa3a9fe5a253165c543fc2a8f06" Oct 30 03:41:23.769884 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:41:23.769837 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0b22edaab786436449b08daf0133f8e5ed99fa3a9fe5a253165c543fc2a8f06\": container with ID starting with d0b22edaab786436449b08daf0133f8e5ed99fa3a9fe5a253165c543fc2a8f06 not found: ID does not exist" containerID="d0b22edaab786436449b08daf0133f8e5ed99fa3a9fe5a253165c543fc2a8f06" Oct 30 03:41:23.769884 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.769860 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0b22edaab786436449b08daf0133f8e5ed99fa3a9fe5a253165c543fc2a8f06"} err="failed to get container status \"d0b22edaab786436449b08daf0133f8e5ed99fa3a9fe5a253165c543fc2a8f06\": rpc error: code = NotFound desc = could not find container \"d0b22edaab786436449b08daf0133f8e5ed99fa3a9fe5a253165c543fc2a8f06\": container with ID starting with d0b22edaab786436449b08daf0133f8e5ed99fa3a9fe5a253165c543fc2a8f06 not found: ID does not exist" Oct 30 03:41:23.804402 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.804378 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b"] Oct 30 03:41:23.815831 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.815809 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-84d679d984-6vn7b"] Oct 30 03:41:23.848385 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.848361 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:41:23.852541 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.852517 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:41:23.948832 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:41:23.948791 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:41:23.981398 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:23.981355 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="189e5984-4f6c-45fd-a40f-9a029341eb1c" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.37:9093/-/ready\": dial tcp 10.134.0.37:9093: connect: connection refused" Oct 30 03:41:24.010579 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:24.010506 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba5b59af-7d63-4eca-8a99-def653dad11c" path="/var/lib/kubelet/pods/ba5b59af-7d63-4eca-8a99-def653dad11c/volumes" Oct 30 03:41:24.010974 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:24.010958 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c680be27-27a7-4953-a502-f283c2d7ad02" path="/var/lib/kubelet/pods/c680be27-27a7-4953-a502-f283c2d7ad02/volumes" Oct 30 03:41:24.011242 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:24.011229 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef122d6e-c44e-420b-a77d-d6c7171048b6" path="/var/lib/kubelet/pods/ef122d6e-c44e-420b-a77d-d6c7171048b6/volumes" Oct 30 03:41:28.343977 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:28.343872 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:41:28.354130 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:28.354108 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:41:28.981669 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:28.981612 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="189e5984-4f6c-45fd-a40f-9a029341eb1c" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.37:9093/-/ready\": dial tcp 10.134.0.37:9093: connect: connection refused" Oct 30 03:41:29.508862 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:29.508817 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k"] Oct 30 03:41:29.509285 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:29.509267 5882 cpu_manager.go:414] "containerMap: removing container" podUID="ef122d6e-c44e-420b-a77d-d6c7171048b6" containerName="thanos-querier" Oct 30 03:41:29.509404 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:29.509287 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef122d6e-c44e-420b-a77d-d6c7171048b6" containerName="thanos-querier" Oct 30 03:41:29.509404 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:29.509303 5882 cpu_manager.go:414] "containerMap: removing container" podUID="75adbe29-e89e-409e-86b1-ca7fc25e5e99" containerName="thanos-querier" Oct 30 03:41:29.509404 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:29.509308 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75adbe29-e89e-409e-86b1-ca7fc25e5e99" containerName="thanos-querier" Oct 30 03:41:29.509404 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:29.509321 5882 cpu_manager.go:414] "containerMap: removing container" podUID="ba5b59af-7d63-4eca-8a99-def653dad11c" containerName="ta-container" Oct 30 03:41:29.509404 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:29.509326 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba5b59af-7d63-4eca-8a99-def653dad11c" containerName="ta-container" Oct 30 03:41:29.509404 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:29.509336 5882 cpu_manager.go:414] "containerMap: removing container" podUID="c680be27-27a7-4953-a502-f283c2d7ad02" containerName="perses" Oct 30 03:41:29.509404 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:29.509341 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c680be27-27a7-4953-a502-f283c2d7ad02" containerName="perses" Oct 30 03:41:29.510024 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:29.509419 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="75adbe29-e89e-409e-86b1-ca7fc25e5e99" containerName="thanos-querier" Oct 30 03:41:29.510024 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:29.509435 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba5b59af-7d63-4eca-8a99-def653dad11c" containerName="ta-container" Oct 30 03:41:29.510024 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:29.509445 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c680be27-27a7-4953-a502-f283c2d7ad02" containerName="perses" Oct 30 03:41:29.510024 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:29.509458 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef122d6e-c44e-420b-a77d-d6c7171048b6" containerName="thanos-querier" Oct 30 03:41:29.514064 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:29.514045 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k" Oct 30 03:41:29.530224 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:29.530202 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 03:41:29.542612 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:29.542585 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k"] Oct 30 03:41:29.661289 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:29.661254 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkdl8\" (UniqueName: \"kubernetes.io/projected/41a567a2-1278-4096-a4c7-806e9206f94b-kube-api-access-fkdl8\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k\" (UID: \"41a567a2-1278-4096-a4c7-806e9206f94b\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k" Oct 30 03:41:29.762186 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:29.762116 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkdl8\" (UniqueName: \"kubernetes.io/projected/41a567a2-1278-4096-a4c7-806e9206f94b-kube-api-access-fkdl8\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k\" (UID: \"41a567a2-1278-4096-a4c7-806e9206f94b\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k" Oct 30 03:41:29.787213 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:29.787183 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkdl8\" (UniqueName: \"kubernetes.io/projected/41a567a2-1278-4096-a4c7-806e9206f94b-kube-api-access-fkdl8\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k\" (UID: \"41a567a2-1278-4096-a4c7-806e9206f94b\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k" Oct 30 03:41:29.836116 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:29.836087 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k" Oct 30 03:41:30.124667 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.124407 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k"] Oct 30 03:41:30.127467 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:41:30.127427 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41a567a2_1278_4096_a4c7_806e9206f94b.slice/crio-ea96ae4d3a7ced40ce73bb09bda65a20ebdf20135e5acad34e5329cbcb45935e WatchSource:0}: Error finding container ea96ae4d3a7ced40ce73bb09bda65a20ebdf20135e5acad34e5329cbcb45935e: Status 404 returned error can't find the container with id ea96ae4d3a7ced40ce73bb09bda65a20ebdf20135e5acad34e5329cbcb45935e Oct 30 03:41:30.645040 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.645006 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:41:30.648660 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.648637 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:30.667752 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.667728 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:41:30.698000 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.697975 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 30 03:41:30.730727 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.730694 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp"] Oct 30 03:41:30.734434 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.734411 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" Oct 30 03:41:30.749303 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.749280 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 30 03:41:30.749411 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.749316 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-pzrgb\"" Oct 30 03:41:30.753492 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.753470 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp"] Oct 30 03:41:30.769916 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.769892 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-plugins\") pod \"data-science-perses-0\" (UID: \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:30.770035 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.769932 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-config\") pod \"data-science-perses-0\" (UID: \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:30.770035 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.769950 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7scg\" (UniqueName: \"kubernetes.io/projected/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-kube-api-access-b7scg\") pod \"data-science-perses-0\" (UID: \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:30.788031 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.788006 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k" event={"ID":"41a567a2-1278-4096-a4c7-806e9206f94b","Type":"ContainerStarted","Data":"657f909c369093fdaf88a8abf058ebabf1fe4086aad9c68a2693d19abf076f53"} Oct 30 03:41:30.788146 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.788039 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k" event={"ID":"41a567a2-1278-4096-a4c7-806e9206f94b","Type":"ContainerStarted","Data":"ea96ae4d3a7ced40ce73bb09bda65a20ebdf20135e5acad34e5329cbcb45935e"} Oct 30 03:41:30.798036 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.798016 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 30 03:41:30.828588 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.828545 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k" podStartSLOduration=1.8285345629999998 podStartE2EDuration="1.828534563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:30.827727458 +0000 UTC m=+3602.942042433" watchObservedRunningTime="2026-10-30 03:41:30.828534563 +0000 UTC m=+3602.942849532" Oct 30 03:41:30.870394 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.870366 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-plugins\") pod \"data-science-perses-0\" (UID: \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:30.870534 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.870404 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/90248b9c-85f2-47a2-be83-d9f74d39c46f-ta-internal\") pod \"data-science-collector-targetallocator-7fbd55f9cd-vncpp\" (UID: \"90248b9c-85f2-47a2-be83-d9f74d39c46f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" Oct 30 03:41:30.870534 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.870451 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-config\") pod \"data-science-perses-0\" (UID: \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:30.870803 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.870765 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plw85\" (UniqueName: \"kubernetes.io/projected/90248b9c-85f2-47a2-be83-d9f74d39c46f-kube-api-access-plw85\") pod \"data-science-collector-targetallocator-7fbd55f9cd-vncpp\" (UID: \"90248b9c-85f2-47a2-be83-d9f74d39c46f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" Oct 30 03:41:30.870876 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.870832 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7scg\" (UniqueName: \"kubernetes.io/projected/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-kube-api-access-b7scg\") pod \"data-science-perses-0\" (UID: \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:30.870876 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.870862 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/90248b9c-85f2-47a2-be83-d9f74d39c46f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7fbd55f9cd-vncpp\" (UID: \"90248b9c-85f2-47a2-be83-d9f74d39c46f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" Oct 30 03:41:30.870987 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.870908 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-plugins\") pod \"data-science-perses-0\" (UID: \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:30.871209 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.871187 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-config\") pod \"data-science-perses-0\" (UID: \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:30.897074 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.897009 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7scg\" (UniqueName: \"kubernetes.io/projected/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-kube-api-access-b7scg\") pod \"data-science-perses-0\" (UID: \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:30.972112 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.972085 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plw85\" (UniqueName: \"kubernetes.io/projected/90248b9c-85f2-47a2-be83-d9f74d39c46f-kube-api-access-plw85\") pod \"data-science-collector-targetallocator-7fbd55f9cd-vncpp\" (UID: \"90248b9c-85f2-47a2-be83-d9f74d39c46f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" Oct 30 03:41:30.972244 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.972118 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/90248b9c-85f2-47a2-be83-d9f74d39c46f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7fbd55f9cd-vncpp\" (UID: \"90248b9c-85f2-47a2-be83-d9f74d39c46f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" Oct 30 03:41:30.972244 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.972183 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/90248b9c-85f2-47a2-be83-d9f74d39c46f-ta-internal\") pod \"data-science-collector-targetallocator-7fbd55f9cd-vncpp\" (UID: \"90248b9c-85f2-47a2-be83-d9f74d39c46f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" Oct 30 03:41:30.972828 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.972807 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/90248b9c-85f2-47a2-be83-d9f74d39c46f-ta-internal\") pod \"data-science-collector-targetallocator-7fbd55f9cd-vncpp\" (UID: \"90248b9c-85f2-47a2-be83-d9f74d39c46f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" Oct 30 03:41:30.974732 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.974702 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/90248b9c-85f2-47a2-be83-d9f74d39c46f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7fbd55f9cd-vncpp\" (UID: \"90248b9c-85f2-47a2-be83-d9f74d39c46f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" Oct 30 03:41:30.995402 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:30.995380 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plw85\" (UniqueName: \"kubernetes.io/projected/90248b9c-85f2-47a2-be83-d9f74d39c46f-kube-api-access-plw85\") pod \"data-science-collector-targetallocator-7fbd55f9cd-vncpp\" (UID: \"90248b9c-85f2-47a2-be83-d9f74d39c46f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" Oct 30 03:41:31.045565 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:31.045535 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" Oct 30 03:41:31.336778 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:31.336749 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp"] Oct 30 03:41:31.339759 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:41:31.339729 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90248b9c_85f2_47a2_be83_d9f74d39c46f.slice/crio-60e43a4f5ce77951b8777e8ad30f429697e9152e0c9c06f4e35baf9b4c19d404 WatchSource:0}: Error finding container 60e43a4f5ce77951b8777e8ad30f429697e9152e0c9c06f4e35baf9b4c19d404: Status 404 returned error can't find the container with id 60e43a4f5ce77951b8777e8ad30f429697e9152e0c9c06f4e35baf9b4c19d404 Oct 30 03:41:31.341925 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:31.341892 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:41:31.342021 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:31.341946 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:41:31.795998 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:31.795962 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" event={"ID":"90248b9c-85f2-47a2-be83-d9f74d39c46f","Type":"ContainerStarted","Data":"5a1596ef121bc194d1408847549097d0a129dca1cbe7a427dd1c9f0f9a0d0c49"} Oct 30 03:41:31.796390 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:31.796009 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" event={"ID":"90248b9c-85f2-47a2-be83-d9f74d39c46f","Type":"ContainerStarted","Data":"60e43a4f5ce77951b8777e8ad30f429697e9152e0c9c06f4e35baf9b4c19d404"} Oct 30 03:41:31.839115 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:31.839067 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" podStartSLOduration=1.839054107 podStartE2EDuration="1.839054107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:31.837566158 +0000 UTC m=+3603.951881133" watchObservedRunningTime="2026-10-30 03:41:31.839054107 +0000 UTC m=+3603.953369082" Oct 30 03:41:32.166916 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.166893 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:32.282926 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.282848 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-web-config\" (UniqueName: \"kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-web-config\") pod \"189e5984-4f6c-45fd-a40f-9a029341eb1c\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " Oct 30 03:41:32.282926 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.282893 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189e5984-4f6c-45fd-a40f-9a029341eb1c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/189e5984-4f6c-45fd-a40f-9a029341eb1c-config-out\") pod \"189e5984-4f6c-45fd-a40f-9a029341eb1c\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " Oct 30 03:41:32.283147 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.282952 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189e5984-4f6c-45fd-a40f-9a029341eb1c-tls-assets\" (UniqueName: \"kubernetes.io/projected/189e5984-4f6c-45fd-a40f-9a029341eb1c-tls-assets\") pod \"189e5984-4f6c-45fd-a40f-9a029341eb1c\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " Oct 30 03:41:32.283147 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.282995 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-cluster-tls-config\") pod \"189e5984-4f6c-45fd-a40f-9a029341eb1c\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " Oct 30 03:41:32.283147 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.283046 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-config-volume\" (UniqueName: \"kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-config-volume\") pod \"189e5984-4f6c-45fd-a40f-9a029341eb1c\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " Oct 30 03:41:32.283147 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.283075 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189e5984-4f6c-45fd-a40f-9a029341eb1c-kube-api-access-x2rgz\" (UniqueName: \"kubernetes.io/projected/189e5984-4f6c-45fd-a40f-9a029341eb1c-kube-api-access-x2rgz\") pod \"189e5984-4f6c-45fd-a40f-9a029341eb1c\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " Oct 30 03:41:32.283147 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.283105 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/189e5984-4f6c-45fd-a40f-9a029341eb1c-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/189e5984-4f6c-45fd-a40f-9a029341eb1c-alertmanager-data-science-monitoringstack-db\") pod \"189e5984-4f6c-45fd-a40f-9a029341eb1c\" (UID: \"189e5984-4f6c-45fd-a40f-9a029341eb1c\") " Oct 30 03:41:32.283529 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.283496 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189e5984-4f6c-45fd-a40f-9a029341eb1c-alertmanager-data-science-monitoringstack-db" pod "189e5984-4f6c-45fd-a40f-9a029341eb1c" (UID: "189e5984-4f6c-45fd-a40f-9a029341eb1c"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:41:32.285252 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.285222 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189e5984-4f6c-45fd-a40f-9a029341eb1c-config-out" pod "189e5984-4f6c-45fd-a40f-9a029341eb1c" (UID: "189e5984-4f6c-45fd-a40f-9a029341eb1c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:41:32.285503 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.285466 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189e5984-4f6c-45fd-a40f-9a029341eb1c-tls-assets" pod "189e5984-4f6c-45fd-a40f-9a029341eb1c" (UID: "189e5984-4f6c-45fd-a40f-9a029341eb1c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:41:32.285634 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.285532 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-config-volume" pod "189e5984-4f6c-45fd-a40f-9a029341eb1c" (UID: "189e5984-4f6c-45fd-a40f-9a029341eb1c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:41:32.285706 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.285690 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189e5984-4f6c-45fd-a40f-9a029341eb1c-kube-api-access-x2rgz" pod "189e5984-4f6c-45fd-a40f-9a029341eb1c" (UID: "189e5984-4f6c-45fd-a40f-9a029341eb1c"). InnerVolumeSpecName "kube-api-access-x2rgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:41:32.289529 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.289501 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-cluster-tls-config" pod "189e5984-4f6c-45fd-a40f-9a029341eb1c" (UID: "189e5984-4f6c-45fd-a40f-9a029341eb1c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:41:32.296670 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.296646 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-web-config" pod "189e5984-4f6c-45fd-a40f-9a029341eb1c" (UID: "189e5984-4f6c-45fd-a40f-9a029341eb1c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:41:32.384002 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.383970 5882 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-web-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:41:32.384002 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.383996 5882 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/189e5984-4f6c-45fd-a40f-9a029341eb1c-config-out\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:41:32.384002 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.384007 5882 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/189e5984-4f6c-45fd-a40f-9a029341eb1c-tls-assets\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:41:32.384188 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.384016 5882 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-cluster-tls-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:41:32.384188 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.384024 5882 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/189e5984-4f6c-45fd-a40f-9a029341eb1c-config-volume\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:41:32.384188 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.384034 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2rgz\" (UniqueName: \"kubernetes.io/projected/189e5984-4f6c-45fd-a40f-9a029341eb1c-kube-api-access-x2rgz\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:41:32.384188 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.384045 5882 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/189e5984-4f6c-45fd-a40f-9a029341eb1c-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:41:32.803611 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.803579 5882 generic.go:356] "Generic (PLEG): container finished" podID="189e5984-4f6c-45fd-a40f-9a029341eb1c" containerID="8dcdc26591131c77b75ad781018fdb32de183c9a40b6c96d395f476664ea8c98" exitCode=0 Oct 30 03:41:32.803984 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.803660 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:32.803984 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.803653 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"189e5984-4f6c-45fd-a40f-9a029341eb1c","Type":"ContainerDied","Data":"8dcdc26591131c77b75ad781018fdb32de183c9a40b6c96d395f476664ea8c98"} Oct 30 03:41:32.803984 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.803845 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"189e5984-4f6c-45fd-a40f-9a029341eb1c","Type":"ContainerDied","Data":"03daa462aa4712ead49095d5ecdd43119dc08bcd9f796369b24c4b71901bdcc4"} Oct 30 03:41:32.803984 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.803881 5882 scope.go:122] "RemoveContainer" containerID="c1e6abf370269ad25cbbbac56a67ec32c686b896502966565623c7a12c03b405" Oct 30 03:41:32.804211 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.804174 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" Oct 30 03:41:32.806061 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.806029 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" Oct 30 03:41:32.818082 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.818051 5882 scope.go:122] "RemoveContainer" containerID="8dcdc26591131c77b75ad781018fdb32de183c9a40b6c96d395f476664ea8c98" Oct 30 03:41:32.825873 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.825850 5882 scope.go:122] "RemoveContainer" containerID="ca3540eaa146f872580393153024f6cb3d42be975d0b4a46849e1a8ddf6e112b" Oct 30 03:41:32.833351 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.833330 5882 scope.go:122] "RemoveContainer" containerID="c1e6abf370269ad25cbbbac56a67ec32c686b896502966565623c7a12c03b405" Oct 30 03:41:32.833595 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:41:32.833574 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1e6abf370269ad25cbbbac56a67ec32c686b896502966565623c7a12c03b405\": container with ID starting with c1e6abf370269ad25cbbbac56a67ec32c686b896502966565623c7a12c03b405 not found: ID does not exist" containerID="c1e6abf370269ad25cbbbac56a67ec32c686b896502966565623c7a12c03b405" Oct 30 03:41:32.833663 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.833603 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e6abf370269ad25cbbbac56a67ec32c686b896502966565623c7a12c03b405"} err="failed to get container status \"c1e6abf370269ad25cbbbac56a67ec32c686b896502966565623c7a12c03b405\": rpc error: code = NotFound desc = could not find container \"c1e6abf370269ad25cbbbac56a67ec32c686b896502966565623c7a12c03b405\": container with ID starting with c1e6abf370269ad25cbbbac56a67ec32c686b896502966565623c7a12c03b405 not found: ID does not exist" Oct 30 03:41:32.833663 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.833632 5882 scope.go:122] "RemoveContainer" containerID="8dcdc26591131c77b75ad781018fdb32de183c9a40b6c96d395f476664ea8c98" Oct 30 03:41:32.833886 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:41:32.833863 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dcdc26591131c77b75ad781018fdb32de183c9a40b6c96d395f476664ea8c98\": container with ID starting with 8dcdc26591131c77b75ad781018fdb32de183c9a40b6c96d395f476664ea8c98 not found: ID does not exist" containerID="8dcdc26591131c77b75ad781018fdb32de183c9a40b6c96d395f476664ea8c98" Oct 30 03:41:32.833973 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.833887 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dcdc26591131c77b75ad781018fdb32de183c9a40b6c96d395f476664ea8c98"} err="failed to get container status \"8dcdc26591131c77b75ad781018fdb32de183c9a40b6c96d395f476664ea8c98\": rpc error: code = NotFound desc = could not find container \"8dcdc26591131c77b75ad781018fdb32de183c9a40b6c96d395f476664ea8c98\": container with ID starting with 8dcdc26591131c77b75ad781018fdb32de183c9a40b6c96d395f476664ea8c98 not found: ID does not exist" Oct 30 03:41:32.833973 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.833904 5882 scope.go:122] "RemoveContainer" containerID="ca3540eaa146f872580393153024f6cb3d42be975d0b4a46849e1a8ddf6e112b" Oct 30 03:41:32.834146 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:41:32.834129 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca3540eaa146f872580393153024f6cb3d42be975d0b4a46849e1a8ddf6e112b\": container with ID starting with ca3540eaa146f872580393153024f6cb3d42be975d0b4a46849e1a8ddf6e112b not found: ID does not exist" containerID="ca3540eaa146f872580393153024f6cb3d42be975d0b4a46849e1a8ddf6e112b" Oct 30 03:41:32.834187 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.834151 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3540eaa146f872580393153024f6cb3d42be975d0b4a46849e1a8ddf6e112b"} err="failed to get container status \"ca3540eaa146f872580393153024f6cb3d42be975d0b4a46849e1a8ddf6e112b\": rpc error: code = NotFound desc = could not find container \"ca3540eaa146f872580393153024f6cb3d42be975d0b4a46849e1a8ddf6e112b\": container with ID starting with ca3540eaa146f872580393153024f6cb3d42be975d0b4a46849e1a8ddf6e112b not found: ID does not exist" Oct 30 03:41:32.894022 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.893988 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 03:41:32.906186 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:32.906155 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 03:41:33.302587 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.302553 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp"] Oct 30 03:41:33.302926 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.302910 5882 cpu_manager.go:414] "containerMap: removing container" podUID="189e5984-4f6c-45fd-a40f-9a029341eb1c" containerName="config-reloader" Oct 30 03:41:33.302997 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.302926 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189e5984-4f6c-45fd-a40f-9a029341eb1c" containerName="config-reloader" Oct 30 03:41:33.302997 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.302939 5882 cpu_manager.go:414] "containerMap: removing container" podUID="189e5984-4f6c-45fd-a40f-9a029341eb1c" containerName="alertmanager" Oct 30 03:41:33.302997 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.302944 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189e5984-4f6c-45fd-a40f-9a029341eb1c" containerName="alertmanager" Oct 30 03:41:33.302997 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.302961 5882 cpu_manager.go:414] "containerMap: removing container" podUID="189e5984-4f6c-45fd-a40f-9a029341eb1c" containerName="init-config-reloader" Oct 30 03:41:33.302997 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.302967 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189e5984-4f6c-45fd-a40f-9a029341eb1c" containerName="init-config-reloader" Oct 30 03:41:33.303221 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.303034 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="189e5984-4f6c-45fd-a40f-9a029341eb1c" containerName="config-reloader" Oct 30 03:41:33.303221 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.303042 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="189e5984-4f6c-45fd-a40f-9a029341eb1c" containerName="alertmanager" Oct 30 03:41:33.307743 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.307722 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp" Oct 30 03:41:33.322645 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.322607 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp"] Oct 30 03:41:33.393085 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.393051 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrs9p\" (UniqueName: \"kubernetes.io/projected/b49d0601-a592-4f95-af65-26fbda077c59-kube-api-access-lrs9p\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp\" (UID: \"b49d0601-a592-4f95-af65-26fbda077c59\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp" Oct 30 03:41:33.493744 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.493714 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrs9p\" (UniqueName: \"kubernetes.io/projected/b49d0601-a592-4f95-af65-26fbda077c59-kube-api-access-lrs9p\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp\" (UID: \"b49d0601-a592-4f95-af65-26fbda077c59\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp" Oct 30 03:41:33.516419 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.516387 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrs9p\" (UniqueName: \"kubernetes.io/projected/b49d0601-a592-4f95-af65-26fbda077c59-kube-api-access-lrs9p\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp\" (UID: \"b49d0601-a592-4f95-af65-26fbda077c59\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp" Oct 30 03:41:33.594461 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.594384 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:33.618240 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.618215 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp" Oct 30 03:41:33.622104 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.622077 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:33.695684 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.695654 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:33.695833 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.695804 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:33.709978 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.709941 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\") DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:33.810035 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.810005 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2c422b71c1e0b8fdf34659f2cef7846e06b4353ed88027dc99ade5d7fc29e9d6/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:33.839443 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.839413 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:33.907596 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.907394 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp"] Oct 30 03:41:33.910385 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:41:33.910357 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb49d0601_a592_4f95_af65_26fbda077c59.slice/crio-d2754e4a009fd062961fc05674afc1cce5cf5bfe15a7af768535ef55341d72ea WatchSource:0}: Error finding container d2754e4a009fd062961fc05674afc1cce5cf5bfe15a7af768535ef55341d72ea: Status 404 returned error can't find the container with id d2754e4a009fd062961fc05674afc1cce5cf5bfe15a7af768535ef55341d72ea Oct 30 03:41:33.960095 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:33.960071 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:34.009671 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.009640 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189e5984-4f6c-45fd-a40f-9a029341eb1c" path="/var/lib/kubelet/pods/189e5984-4f6c-45fd-a40f-9a029341eb1c/volumes" Oct 30 03:41:34.149012 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.148925 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 03:41:34.154162 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.154129 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.175179 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.175032 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 03:41:34.200767 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.200737 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.200936 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.200772 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.200936 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.200892 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.201085 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.200937 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.201085 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.200964 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7fl6\" (UniqueName: \"kubernetes.io/projected/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-kube-api-access-m7fl6\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.201194 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.201125 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.201194 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.201167 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.239364 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.239333 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 03:41:34.239505 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.239435 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 03:41:34.239505 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.239444 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-v4hz2\"" Oct 30 03:41:34.239505 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.239480 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 03:41:34.239505 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.239492 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 03:41:34.256491 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.256469 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:41:34.259160 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:41:34.259129 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad51ce29_2421_46e3_97cf_e5e2fbadf67d.slice/crio-2ba7bd2380c658436eeef5f656cbb54097d47e1757b11840aad010117905ff0f WatchSource:0}: Error finding container 2ba7bd2380c658436eeef5f656cbb54097d47e1757b11840aad010117905ff0f: Status 404 returned error can't find the container with id 2ba7bd2380c658436eeef5f656cbb54097d47e1757b11840aad010117905ff0f Oct 30 03:41:34.302680 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.302657 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.302788 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.302691 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.302788 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.302733 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.302788 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.302763 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.302963 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.302822 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.302963 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.302844 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.302963 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.302870 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fl6\" (UniqueName: \"kubernetes.io/projected/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-kube-api-access-m7fl6\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.303208 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.303160 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.305706 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.305680 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.305945 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.305920 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.306075 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.306011 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.306231 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.306213 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.307478 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.307455 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.325799 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.325772 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7fl6\" (UniqueName: \"kubernetes.io/projected/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-kube-api-access-m7fl6\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.474899 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.474814 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:34.795746 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.795255 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 03:41:34.803444 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:41:34.803416 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a64a0a4_a619_46cc_aa53_c3b5f46b87f7.slice/crio-e9aa5bf41d32f4ed194a6d278430d227404676efe4734e37241f0e8ab2996cea WatchSource:0}: Error finding container e9aa5bf41d32f4ed194a6d278430d227404676efe4734e37241f0e8ab2996cea: Status 404 returned error can't find the container with id e9aa5bf41d32f4ed194a6d278430d227404676efe4734e37241f0e8ab2996cea Oct 30 03:41:34.818988 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.818768 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"ad51ce29-2421-46e3-97cf-e5e2fbadf67d","Type":"ContainerStarted","Data":"2af460072686db74224dea9496d6d819f4773ac16265751b412b9e0d8305c0d7"} Oct 30 03:41:34.818988 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.818824 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:34.818988 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.818843 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"ad51ce29-2421-46e3-97cf-e5e2fbadf67d","Type":"ContainerStarted","Data":"2ba7bd2380c658436eeef5f656cbb54097d47e1757b11840aad010117905ff0f"} Oct 30 03:41:34.820106 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.820081 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7","Type":"ContainerStarted","Data":"e9aa5bf41d32f4ed194a6d278430d227404676efe4734e37241f0e8ab2996cea"} Oct 30 03:41:34.822604 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.822588 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp" event={"ID":"b49d0601-a592-4f95-af65-26fbda077c59","Type":"ContainerStarted","Data":"1207645ce3ce1aeceb76a12b9cee9ab0296676525e4bb7109aaa63c8699295f7"} Oct 30 03:41:34.822714 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.822668 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp" event={"ID":"b49d0601-a592-4f95-af65-26fbda077c59","Type":"ContainerStarted","Data":"d2754e4a009fd062961fc05674afc1cce5cf5bfe15a7af768535ef55341d72ea"} Oct 30 03:41:34.857344 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.857303 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=4.857287503 podStartE2EDuration="4.857287503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:34.856965556 +0000 UTC m=+3606.971280532" watchObservedRunningTime="2026-10-30 03:41:34.857287503 +0000 UTC m=+3606.971602480" Oct 30 03:41:34.899178 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.899117 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp" podStartSLOduration=1.899100528 podStartE2EDuration="1.899100528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:34.897807325 +0000 UTC m=+3607.012122303" watchObservedRunningTime="2026-10-30 03:41:34.899100528 +0000 UTC m=+3607.013415501" Oct 30 03:41:34.925917 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.925884 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k"] Oct 30 03:41:34.926096 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:34.926073 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k" podUID="41a567a2-1278-4096-a4c7-806e9206f94b" containerName="thanos-querier" containerID="cri-o://657f909c369093fdaf88a8abf058ebabf1fe4086aad9c68a2693d19abf076f53" gracePeriod=30 Oct 30 03:41:35.184216 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:35.184184 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k" Oct 30 03:41:35.210997 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:35.210969 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a567a2-1278-4096-a4c7-806e9206f94b-kube-api-access-fkdl8\" (UniqueName: \"kubernetes.io/projected/41a567a2-1278-4096-a4c7-806e9206f94b-kube-api-access-fkdl8\") pod \"41a567a2-1278-4096-a4c7-806e9206f94b\" (UID: \"41a567a2-1278-4096-a4c7-806e9206f94b\") " Oct 30 03:41:35.212904 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:35.212875 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a567a2-1278-4096-a4c7-806e9206f94b-kube-api-access-fkdl8" pod "41a567a2-1278-4096-a4c7-806e9206f94b" (UID: "41a567a2-1278-4096-a4c7-806e9206f94b"). InnerVolumeSpecName "kube-api-access-fkdl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:41:35.312305 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:35.312221 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkdl8\" (UniqueName: \"kubernetes.io/projected/41a567a2-1278-4096-a4c7-806e9206f94b-kube-api-access-fkdl8\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:41:35.829223 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:35.829191 5882 generic.go:356] "Generic (PLEG): container finished" podID="41a567a2-1278-4096-a4c7-806e9206f94b" containerID="657f909c369093fdaf88a8abf058ebabf1fe4086aad9c68a2693d19abf076f53" exitCode=0 Oct 30 03:41:35.829722 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:35.829228 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k" Oct 30 03:41:35.829722 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:35.829273 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k" event={"ID":"41a567a2-1278-4096-a4c7-806e9206f94b","Type":"ContainerDied","Data":"657f909c369093fdaf88a8abf058ebabf1fe4086aad9c68a2693d19abf076f53"} Oct 30 03:41:35.829722 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:35.829318 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k" event={"ID":"41a567a2-1278-4096-a4c7-806e9206f94b","Type":"ContainerDied","Data":"ea96ae4d3a7ced40ce73bb09bda65a20ebdf20135e5acad34e5329cbcb45935e"} Oct 30 03:41:35.829722 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:35.829342 5882 scope.go:122] "RemoveContainer" containerID="657f909c369093fdaf88a8abf058ebabf1fe4086aad9c68a2693d19abf076f53" Oct 30 03:41:35.831083 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:35.831058 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7","Type":"ContainerStarted","Data":"9ceaf71601ca5b306f02aa9d00a2513184be1c350be1672210a3ce2f685c0b2c"} Oct 30 03:41:35.838784 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:35.838766 5882 scope.go:122] "RemoveContainer" containerID="657f909c369093fdaf88a8abf058ebabf1fe4086aad9c68a2693d19abf076f53" Oct 30 03:41:35.839063 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:41:35.839036 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657f909c369093fdaf88a8abf058ebabf1fe4086aad9c68a2693d19abf076f53\": container with ID starting with 657f909c369093fdaf88a8abf058ebabf1fe4086aad9c68a2693d19abf076f53 not found: ID does not exist" containerID="657f909c369093fdaf88a8abf058ebabf1fe4086aad9c68a2693d19abf076f53" Oct 30 03:41:35.839117 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:35.839076 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657f909c369093fdaf88a8abf058ebabf1fe4086aad9c68a2693d19abf076f53"} err="failed to get container status \"657f909c369093fdaf88a8abf058ebabf1fe4086aad9c68a2693d19abf076f53\": rpc error: code = NotFound desc = could not find container \"657f909c369093fdaf88a8abf058ebabf1fe4086aad9c68a2693d19abf076f53\": container with ID starting with 657f909c369093fdaf88a8abf058ebabf1fe4086aad9c68a2693d19abf076f53 not found: ID does not exist" Oct 30 03:41:35.883295 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:35.883257 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k"] Oct 30 03:41:35.895562 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:35.895528 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-chm5k"] Oct 30 03:41:36.010321 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:36.010290 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41a567a2-1278-4096-a4c7-806e9206f94b" path="/var/lib/kubelet/pods/41a567a2-1278-4096-a4c7-806e9206f94b/volumes" Oct 30 03:41:38.845866 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:38.845828 5882 generic.go:356] "Generic (PLEG): container finished" podID="3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" containerID="9ceaf71601ca5b306f02aa9d00a2513184be1c350be1672210a3ce2f685c0b2c" exitCode=0 Oct 30 03:41:38.846453 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:38.845869 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7","Type":"ContainerDied","Data":"9ceaf71601ca5b306f02aa9d00a2513184be1c350be1672210a3ce2f685c0b2c"} Oct 30 03:41:39.854246 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:39.854205 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7","Type":"ContainerStarted","Data":"59489c0d247cd6c31a81ba7c637048a8553ff57cf1595fce97cec2d168ab6992"} Oct 30 03:41:39.854650 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:39.854268 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7","Type":"ContainerStarted","Data":"f9898b1c1ab8fa1f0702685e306c91131cd81332df2dfbbb8a0b0daec7f3a6fb"} Oct 30 03:41:39.854650 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:39.854464 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:39.924340 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:39.924291 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.724289751 podStartE2EDuration="5.924277294s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.199987543s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:39.912255815 +0000 UTC m=+3612.026570789" watchObservedRunningTime="2026-10-30 03:41:39.924277294 +0000 UTC m=+3612.038592269" Oct 30 03:41:45.835018 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:45.834988 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:41:45.860375 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:41:45.860340 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:41:53.942973 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:41:53.942931 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:42:00.416527 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:00.416496 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866"] Oct 30 03:42:00.417100 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:00.417069 5882 cpu_manager.go:414] "containerMap: removing container" podUID="41a567a2-1278-4096-a4c7-806e9206f94b" containerName="thanos-querier" Oct 30 03:42:00.417100 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:00.417096 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a567a2-1278-4096-a4c7-806e9206f94b" containerName="thanos-querier" Oct 30 03:42:00.417269 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:00.417172 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a567a2-1278-4096-a4c7-806e9206f94b" containerName="thanos-querier" Oct 30 03:42:00.433119 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:00.433092 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" Oct 30 03:42:00.439372 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:00.439345 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866"] Oct 30 03:42:00.532954 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:00.532916 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/23d6687f-104e-4b6a-a59e-2641896c52ea-ta-internal\") pod \"data-science-collector-targetallocator-55cb99746c-pt866\" (UID: \"23d6687f-104e-4b6a-a59e-2641896c52ea\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" Oct 30 03:42:00.533130 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:00.532977 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmbfr\" (UniqueName: \"kubernetes.io/projected/23d6687f-104e-4b6a-a59e-2641896c52ea-kube-api-access-bmbfr\") pod \"data-science-collector-targetallocator-55cb99746c-pt866\" (UID: \"23d6687f-104e-4b6a-a59e-2641896c52ea\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" Oct 30 03:42:00.533130 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:00.533084 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/23d6687f-104e-4b6a-a59e-2641896c52ea-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-55cb99746c-pt866\" (UID: \"23d6687f-104e-4b6a-a59e-2641896c52ea\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" Oct 30 03:42:00.633494 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:00.633460 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/23d6687f-104e-4b6a-a59e-2641896c52ea-ta-internal\") pod \"data-science-collector-targetallocator-55cb99746c-pt866\" (UID: \"23d6687f-104e-4b6a-a59e-2641896c52ea\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" Oct 30 03:42:00.633681 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:00.633509 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmbfr\" (UniqueName: \"kubernetes.io/projected/23d6687f-104e-4b6a-a59e-2641896c52ea-kube-api-access-bmbfr\") pod \"data-science-collector-targetallocator-55cb99746c-pt866\" (UID: \"23d6687f-104e-4b6a-a59e-2641896c52ea\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" Oct 30 03:42:00.633681 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:00.633531 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/23d6687f-104e-4b6a-a59e-2641896c52ea-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-55cb99746c-pt866\" (UID: \"23d6687f-104e-4b6a-a59e-2641896c52ea\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" Oct 30 03:42:00.634194 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:00.634168 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/23d6687f-104e-4b6a-a59e-2641896c52ea-ta-internal\") pod \"data-science-collector-targetallocator-55cb99746c-pt866\" (UID: \"23d6687f-104e-4b6a-a59e-2641896c52ea\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" Oct 30 03:42:00.635997 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:00.635977 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/23d6687f-104e-4b6a-a59e-2641896c52ea-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-55cb99746c-pt866\" (UID: \"23d6687f-104e-4b6a-a59e-2641896c52ea\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" Oct 30 03:42:00.656164 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:00.656139 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmbfr\" (UniqueName: \"kubernetes.io/projected/23d6687f-104e-4b6a-a59e-2641896c52ea-kube-api-access-bmbfr\") pod \"data-science-collector-targetallocator-55cb99746c-pt866\" (UID: \"23d6687f-104e-4b6a-a59e-2641896c52ea\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" Oct 30 03:42:00.746211 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:00.746182 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" Oct 30 03:42:01.035831 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:01.035800 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866"] Oct 30 03:42:01.038784 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:42:01.038755 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23d6687f_104e_4b6a_a59e_2641896c52ea.slice/crio-dc77d0e334ed85d856f33aedf6a90051e0f7f90f72c0113c4ef7315ec4b56c1e WatchSource:0}: Error finding container dc77d0e334ed85d856f33aedf6a90051e0f7f90f72c0113c4ef7315ec4b56c1e: Status 404 returned error can't find the container with id dc77d0e334ed85d856f33aedf6a90051e0f7f90f72c0113c4ef7315ec4b56c1e Oct 30 03:42:01.040944 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:01.040913 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:42:01.041027 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:01.040969 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:42:01.956067 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:01.956029 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" event={"ID":"23d6687f-104e-4b6a-a59e-2641896c52ea","Type":"ContainerStarted","Data":"c63da6c9ecd88e8429d5fa8e7fac5a4d36a1eca32223dd3bf7e50b7baf1fd5a0"} Oct 30 03:42:01.956479 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:01.956078 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" event={"ID":"23d6687f-104e-4b6a-a59e-2641896c52ea","Type":"ContainerStarted","Data":"dc77d0e334ed85d856f33aedf6a90051e0f7f90f72c0113c4ef7315ec4b56c1e"} Oct 30 03:42:01.956479 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:01.956210 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" Oct 30 03:42:01.957710 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:01.957687 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" Oct 30 03:42:01.996362 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:01.996322 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" podStartSLOduration=1.9963123980000002 podStartE2EDuration="1.996312398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:42:01.995663082 +0000 UTC m=+3634.109978049" watchObservedRunningTime="2026-10-30 03:42:01.996312398 +0000 UTC m=+3634.110627372" Oct 30 03:42:02.065064 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:02.065034 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp"] Oct 30 03:42:02.065226 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:02.065206 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" podUID="90248b9c-85f2-47a2-be83-d9f74d39c46f" containerName="ta-container" containerID="cri-o://5a1596ef121bc194d1408847549097d0a129dca1cbe7a427dd1c9f0f9a0d0c49" gracePeriod=30 Oct 30 03:42:02.310259 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:02.310235 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" Oct 30 03:42:02.346065 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:02.346036 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90248b9c-85f2-47a2-be83-d9f74d39c46f-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/90248b9c-85f2-47a2-be83-d9f74d39c46f-data-science-collector-ta-server-cert\") pod \"90248b9c-85f2-47a2-be83-d9f74d39c46f\" (UID: \"90248b9c-85f2-47a2-be83-d9f74d39c46f\") " Oct 30 03:42:02.346205 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:02.346082 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90248b9c-85f2-47a2-be83-d9f74d39c46f-kube-api-access-plw85\" (UniqueName: \"kubernetes.io/projected/90248b9c-85f2-47a2-be83-d9f74d39c46f-kube-api-access-plw85\") pod \"90248b9c-85f2-47a2-be83-d9f74d39c46f\" (UID: \"90248b9c-85f2-47a2-be83-d9f74d39c46f\") " Oct 30 03:42:02.346205 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:02.346108 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90248b9c-85f2-47a2-be83-d9f74d39c46f-ta-internal\" (UniqueName: \"kubernetes.io/configmap/90248b9c-85f2-47a2-be83-d9f74d39c46f-ta-internal\") pod \"90248b9c-85f2-47a2-be83-d9f74d39c46f\" (UID: \"90248b9c-85f2-47a2-be83-d9f74d39c46f\") " Oct 30 03:42:02.346548 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:02.346525 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90248b9c-85f2-47a2-be83-d9f74d39c46f-ta-internal" pod "90248b9c-85f2-47a2-be83-d9f74d39c46f" (UID: "90248b9c-85f2-47a2-be83-d9f74d39c46f"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:42:02.348145 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:02.348124 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90248b9c-85f2-47a2-be83-d9f74d39c46f-kube-api-access-plw85" pod "90248b9c-85f2-47a2-be83-d9f74d39c46f" (UID: "90248b9c-85f2-47a2-be83-d9f74d39c46f"). InnerVolumeSpecName "kube-api-access-plw85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:42:02.348229 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:02.348141 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90248b9c-85f2-47a2-be83-d9f74d39c46f-data-science-collector-ta-server-cert" pod "90248b9c-85f2-47a2-be83-d9f74d39c46f" (UID: "90248b9c-85f2-47a2-be83-d9f74d39c46f"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:42:02.447158 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:02.447126 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/90248b9c-85f2-47a2-be83-d9f74d39c46f-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:42:02.447158 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:02.447152 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plw85\" (UniqueName: \"kubernetes.io/projected/90248b9c-85f2-47a2-be83-d9f74d39c46f-kube-api-access-plw85\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:42:02.447158 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:02.447164 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/90248b9c-85f2-47a2-be83-d9f74d39c46f-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:42:02.961862 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:02.961828 5882 generic.go:356] "Generic (PLEG): container finished" podID="90248b9c-85f2-47a2-be83-d9f74d39c46f" containerID="5a1596ef121bc194d1408847549097d0a129dca1cbe7a427dd1c9f0f9a0d0c49" exitCode=0 Oct 30 03:42:02.962313 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:02.961871 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" Oct 30 03:42:02.962313 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:02.961917 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" event={"ID":"90248b9c-85f2-47a2-be83-d9f74d39c46f","Type":"ContainerDied","Data":"5a1596ef121bc194d1408847549097d0a129dca1cbe7a427dd1c9f0f9a0d0c49"} Oct 30 03:42:02.962313 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:02.961957 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp" event={"ID":"90248b9c-85f2-47a2-be83-d9f74d39c46f","Type":"ContainerDied","Data":"60e43a4f5ce77951b8777e8ad30f429697e9152e0c9c06f4e35baf9b4c19d404"} Oct 30 03:42:02.962313 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:02.961975 5882 scope.go:122] "RemoveContainer" containerID="5a1596ef121bc194d1408847549097d0a129dca1cbe7a427dd1c9f0f9a0d0c49" Oct 30 03:42:02.973635 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:02.973608 5882 scope.go:122] "RemoveContainer" containerID="5a1596ef121bc194d1408847549097d0a129dca1cbe7a427dd1c9f0f9a0d0c49" Oct 30 03:42:02.973904 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:42:02.973886 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a1596ef121bc194d1408847549097d0a129dca1cbe7a427dd1c9f0f9a0d0c49\": container with ID starting with 5a1596ef121bc194d1408847549097d0a129dca1cbe7a427dd1c9f0f9a0d0c49 not found: ID does not exist" containerID="5a1596ef121bc194d1408847549097d0a129dca1cbe7a427dd1c9f0f9a0d0c49" Oct 30 03:42:02.973959 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:02.973914 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a1596ef121bc194d1408847549097d0a129dca1cbe7a427dd1c9f0f9a0d0c49"} err="failed to get container status \"5a1596ef121bc194d1408847549097d0a129dca1cbe7a427dd1c9f0f9a0d0c49\": rpc error: code = NotFound desc = could not find container \"5a1596ef121bc194d1408847549097d0a129dca1cbe7a427dd1c9f0f9a0d0c49\": container with ID starting with 5a1596ef121bc194d1408847549097d0a129dca1cbe7a427dd1c9f0f9a0d0c49 not found: ID does not exist" Oct 30 03:42:03.010418 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:03.010389 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp"] Oct 30 03:42:03.021874 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:03.021848 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7fbd55f9cd-vncpp"] Oct 30 03:42:04.009659 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:42:04.009604 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90248b9c-85f2-47a2-be83-d9f74d39c46f" path="/var/lib/kubelet/pods/90248b9c-85f2-47a2-be83-d9f74d39c46f/volumes" Oct 30 03:42:23.941917 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:42:23.941875 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:42:53.942868 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:42:53.942832 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:43:13.348641 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:13.348548 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-ingress-18e331c3b7b9cfdf"] Oct 30 03:43:13.349103 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:13.348962 5882 cpu_manager.go:414] "containerMap: removing container" podUID="90248b9c-85f2-47a2-be83-d9f74d39c46f" containerName="ta-container" Oct 30 03:43:13.349103 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:13.348974 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90248b9c-85f2-47a2-be83-d9f74d39c46f" containerName="ta-container" Oct 30 03:43:13.349103 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:13.349045 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="90248b9c-85f2-47a2-be83-d9f74d39c46f" containerName="ta-container" Oct 30 03:43:13.352224 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:13.352203 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-ingress-18e331c3b7b9cfdf" Oct 30 03:43:13.362721 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:13.362700 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-ingress-18e331c3b7b9cfdf" Oct 30 03:43:13.369678 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:13.369655 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-ingress-18e331c3b7b9cfdf"] Oct 30 03:43:13.651046 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:13.650969 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-ingress-18e331c3b7b9cfdf"] Oct 30 03:43:13.654575 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:43:13.654537 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4938121_1478_4a13_8634_949d53dbc4bb.slice/crio-d44a1ba2af22082cb92cc38690736789c88a6b91a2761dce10f13551e92ad368 WatchSource:0}: Error finding container d44a1ba2af22082cb92cc38690736789c88a6b91a2761dce10f13551e92ad368: Status 404 returned error can't find the container with id d44a1ba2af22082cb92cc38690736789c88a6b91a2761dce10f13551e92ad368 Oct 30 03:43:13.656511 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:13.656494 5882 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:43:14.263388 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:14.263356 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-ingress-18e331c3b7b9cfdf" event={"ID":"c4938121-1478-4a13-8634-949d53dbc4bb","Type":"ContainerStarted","Data":"d44a1ba2af22082cb92cc38690736789c88a6b91a2761dce10f13551e92ad368"} Oct 30 03:43:15.268841 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:15.268807 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-ingress-18e331c3b7b9cfdf" event={"ID":"c4938121-1478-4a13-8634-949d53dbc4bb","Type":"ContainerStarted","Data":"bb7e40a8e240789c028e607fd4c02fd2ff5839b1b91e8be46b38fc0faab3896b"} Oct 30 03:43:15.305932 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:15.305885 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-ingress-18e331c3b7b9cfdf" podStartSLOduration=1.515826211 podStartE2EDuration="2.305870714s" totalImagesPullingTime="790.044503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:15.305838667 +0000 UTC m=+3707.420153642" watchObservedRunningTime="2026-10-30 03:43:15.305870714 +0000 UTC m=+3707.420185689" Oct 30 03:43:20.290869 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:20.290834 5882 generic.go:356] "Generic (PLEG): container finished" podID="c4938121-1478-4a13-8634-949d53dbc4bb" containerID="bb7e40a8e240789c028e607fd4c02fd2ff5839b1b91e8be46b38fc0faab3896b" exitCode=0 Oct 30 03:43:20.291264 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:20.290899 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-ingress-18e331c3b7b9cfdf" event={"ID":"c4938121-1478-4a13-8634-949d53dbc4bb","Type":"ContainerDied","Data":"bb7e40a8e240789c028e607fd4c02fd2ff5839b1b91e8be46b38fc0faab3896b"} Oct 30 03:43:21.422592 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:21.422568 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-ingress-18e331c3b7b9cfdf" Oct 30 03:43:22.301210 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:22.301123 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-ingress-18e331c3b7b9cfdf" event={"ID":"c4938121-1478-4a13-8634-949d53dbc4bb","Type":"ContainerDied","Data":"d44a1ba2af22082cb92cc38690736789c88a6b91a2761dce10f13551e92ad368"} Oct 30 03:43:22.301210 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:22.301168 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d44a1ba2af22082cb92cc38690736789c88a6b91a2761dce10f13551e92ad368" Oct 30 03:43:22.301210 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:22.301137 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-ingress-18e331c3b7b9cfdf" Oct 30 03:43:23.455959 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:23.455928 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-ingress-18e331c3b7b9cfdf"] Oct 30 03:43:23.460111 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:23.460079 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-ingress-18e331c3b7b9cfdf"] Oct 30 03:43:23.942475 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:43:23.942440 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:43:24.009655 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:24.009598 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4938121-1478-4a13-8634-949d53dbc4bb" path="/var/lib/kubelet/pods/c4938121-1478-4a13-8634-949d53dbc4bb/volumes" Oct 30 03:43:29.000417 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:29.000376 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d"] Oct 30 03:43:29.000850 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:29.000816 5882 cpu_manager.go:414] "containerMap: removing container" podUID="c4938121-1478-4a13-8634-949d53dbc4bb" containerName="curl" Oct 30 03:43:29.000850 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:29.000830 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4938121-1478-4a13-8634-949d53dbc4bb" containerName="curl" Oct 30 03:43:29.000954 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:29.000948 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4938121-1478-4a13-8634-949d53dbc4bb" containerName="curl" Oct 30 03:43:29.004211 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:29.004186 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" Oct 30 03:43:29.031470 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:29.031433 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d"] Oct 30 03:43:29.075091 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:29.075059 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/77757029-564d-46f0-9b5e-a95764c4c1e8-ta-internal\") pod \"data-science-collector-targetallocator-64b47bdcbd-8252d\" (UID: \"77757029-564d-46f0-9b5e-a95764c4c1e8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" Oct 30 03:43:29.075262 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:29.075132 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/77757029-564d-46f0-9b5e-a95764c4c1e8-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-64b47bdcbd-8252d\" (UID: \"77757029-564d-46f0-9b5e-a95764c4c1e8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" Oct 30 03:43:29.075262 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:29.075211 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8zj6\" (UniqueName: \"kubernetes.io/projected/77757029-564d-46f0-9b5e-a95764c4c1e8-kube-api-access-n8zj6\") pod \"data-science-collector-targetallocator-64b47bdcbd-8252d\" (UID: \"77757029-564d-46f0-9b5e-a95764c4c1e8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" Oct 30 03:43:29.175975 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:29.175937 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/77757029-564d-46f0-9b5e-a95764c4c1e8-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-64b47bdcbd-8252d\" (UID: \"77757029-564d-46f0-9b5e-a95764c4c1e8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" Oct 30 03:43:29.176132 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:29.175986 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8zj6\" (UniqueName: \"kubernetes.io/projected/77757029-564d-46f0-9b5e-a95764c4c1e8-kube-api-access-n8zj6\") pod \"data-science-collector-targetallocator-64b47bdcbd-8252d\" (UID: \"77757029-564d-46f0-9b5e-a95764c4c1e8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" Oct 30 03:43:29.176132 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:29.176027 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/77757029-564d-46f0-9b5e-a95764c4c1e8-ta-internal\") pod \"data-science-collector-targetallocator-64b47bdcbd-8252d\" (UID: \"77757029-564d-46f0-9b5e-a95764c4c1e8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" Oct 30 03:43:29.176610 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:29.176592 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/77757029-564d-46f0-9b5e-a95764c4c1e8-ta-internal\") pod \"data-science-collector-targetallocator-64b47bdcbd-8252d\" (UID: \"77757029-564d-46f0-9b5e-a95764c4c1e8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" Oct 30 03:43:29.178632 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:29.178588 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/77757029-564d-46f0-9b5e-a95764c4c1e8-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-64b47bdcbd-8252d\" (UID: \"77757029-564d-46f0-9b5e-a95764c4c1e8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" Oct 30 03:43:29.200105 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:29.200075 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8zj6\" (UniqueName: \"kubernetes.io/projected/77757029-564d-46f0-9b5e-a95764c4c1e8-kube-api-access-n8zj6\") pod \"data-science-collector-targetallocator-64b47bdcbd-8252d\" (UID: \"77757029-564d-46f0-9b5e-a95764c4c1e8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" Oct 30 03:43:29.315792 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:29.315708 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" Oct 30 03:43:29.608104 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:29.607942 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d"] Oct 30 03:43:29.610715 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:43:29.610683 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77757029_564d_46f0_9b5e_a95764c4c1e8.slice/crio-48c09fc315c15b8d3ca850f62fde32e585dee3a17902ffc98599e8165a4356df WatchSource:0}: Error finding container 48c09fc315c15b8d3ca850f62fde32e585dee3a17902ffc98599e8165a4356df: Status 404 returned error can't find the container with id 48c09fc315c15b8d3ca850f62fde32e585dee3a17902ffc98599e8165a4356df Oct 30 03:43:29.612951 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:29.612920 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:43:29.613020 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:29.612979 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:43:30.339752 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:30.339709 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" event={"ID":"77757029-564d-46f0-9b5e-a95764c4c1e8","Type":"ContainerStarted","Data":"ffd1e638f15356abb3c43dbf8b0ea4b822cc07bc801109da575b81981653f701"} Oct 30 03:43:30.340150 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:30.339761 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" event={"ID":"77757029-564d-46f0-9b5e-a95764c4c1e8","Type":"ContainerStarted","Data":"48c09fc315c15b8d3ca850f62fde32e585dee3a17902ffc98599e8165a4356df"} Oct 30 03:43:30.340150 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:30.339920 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" Oct 30 03:43:30.341785 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:30.341764 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" Oct 30 03:43:30.379624 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:30.379574 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" podStartSLOduration=2.379562881 podStartE2EDuration="2.379562881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:30.379278897 +0000 UTC m=+3722.493593885" watchObservedRunningTime="2026-10-30 03:43:30.379562881 +0000 UTC m=+3722.493877835" Oct 30 03:43:30.449031 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:30.448998 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866"] Oct 30 03:43:30.449440 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:30.449227 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" podUID="23d6687f-104e-4b6a-a59e-2641896c52ea" containerName="ta-container" containerID="cri-o://c63da6c9ecd88e8429d5fa8e7fac5a4d36a1eca32223dd3bf7e50b7baf1fd5a0" gracePeriod=30 Oct 30 03:43:30.692223 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:30.692195 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" Oct 30 03:43:30.791949 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:30.791909 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23d6687f-104e-4b6a-a59e-2641896c52ea-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/23d6687f-104e-4b6a-a59e-2641896c52ea-data-science-collector-ta-server-cert\") pod \"23d6687f-104e-4b6a-a59e-2641896c52ea\" (UID: \"23d6687f-104e-4b6a-a59e-2641896c52ea\") " Oct 30 03:43:30.792127 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:30.792103 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d6687f-104e-4b6a-a59e-2641896c52ea-ta-internal\" (UniqueName: \"kubernetes.io/configmap/23d6687f-104e-4b6a-a59e-2641896c52ea-ta-internal\") pod \"23d6687f-104e-4b6a-a59e-2641896c52ea\" (UID: \"23d6687f-104e-4b6a-a59e-2641896c52ea\") " Oct 30 03:43:30.792193 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:30.792146 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d6687f-104e-4b6a-a59e-2641896c52ea-kube-api-access-bmbfr\" (UniqueName: \"kubernetes.io/projected/23d6687f-104e-4b6a-a59e-2641896c52ea-kube-api-access-bmbfr\") pod \"23d6687f-104e-4b6a-a59e-2641896c52ea\" (UID: \"23d6687f-104e-4b6a-a59e-2641896c52ea\") " Oct 30 03:43:30.792450 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:30.792427 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d6687f-104e-4b6a-a59e-2641896c52ea-ta-internal" pod "23d6687f-104e-4b6a-a59e-2641896c52ea" (UID: "23d6687f-104e-4b6a-a59e-2641896c52ea"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:43:30.793983 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:30.793955 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d6687f-104e-4b6a-a59e-2641896c52ea-data-science-collector-ta-server-cert" pod "23d6687f-104e-4b6a-a59e-2641896c52ea" (UID: "23d6687f-104e-4b6a-a59e-2641896c52ea"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:43:30.794135 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:30.794117 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d6687f-104e-4b6a-a59e-2641896c52ea-kube-api-access-bmbfr" pod "23d6687f-104e-4b6a-a59e-2641896c52ea" (UID: "23d6687f-104e-4b6a-a59e-2641896c52ea"). InnerVolumeSpecName "kube-api-access-bmbfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:43:30.893487 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:30.893404 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmbfr\" (UniqueName: \"kubernetes.io/projected/23d6687f-104e-4b6a-a59e-2641896c52ea-kube-api-access-bmbfr\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:43:30.893487 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:30.893430 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/23d6687f-104e-4b6a-a59e-2641896c52ea-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:43:30.893487 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:30.893441 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/23d6687f-104e-4b6a-a59e-2641896c52ea-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:43:31.345314 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:31.345282 5882 generic.go:356] "Generic (PLEG): container finished" podID="23d6687f-104e-4b6a-a59e-2641896c52ea" containerID="c63da6c9ecd88e8429d5fa8e7fac5a4d36a1eca32223dd3bf7e50b7baf1fd5a0" exitCode=0 Oct 30 03:43:31.345795 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:31.345340 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" Oct 30 03:43:31.345795 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:31.345367 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" event={"ID":"23d6687f-104e-4b6a-a59e-2641896c52ea","Type":"ContainerDied","Data":"c63da6c9ecd88e8429d5fa8e7fac5a4d36a1eca32223dd3bf7e50b7baf1fd5a0"} Oct 30 03:43:31.345795 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:31.345428 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866" event={"ID":"23d6687f-104e-4b6a-a59e-2641896c52ea","Type":"ContainerDied","Data":"dc77d0e334ed85d856f33aedf6a90051e0f7f90f72c0113c4ef7315ec4b56c1e"} Oct 30 03:43:31.345795 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:31.345455 5882 scope.go:122] "RemoveContainer" containerID="c63da6c9ecd88e8429d5fa8e7fac5a4d36a1eca32223dd3bf7e50b7baf1fd5a0" Oct 30 03:43:31.354710 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:31.354687 5882 scope.go:122] "RemoveContainer" containerID="c63da6c9ecd88e8429d5fa8e7fac5a4d36a1eca32223dd3bf7e50b7baf1fd5a0" Oct 30 03:43:31.355023 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:43:31.354988 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c63da6c9ecd88e8429d5fa8e7fac5a4d36a1eca32223dd3bf7e50b7baf1fd5a0\": container with ID starting with c63da6c9ecd88e8429d5fa8e7fac5a4d36a1eca32223dd3bf7e50b7baf1fd5a0 not found: ID does not exist" containerID="c63da6c9ecd88e8429d5fa8e7fac5a4d36a1eca32223dd3bf7e50b7baf1fd5a0" Oct 30 03:43:31.355123 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:31.355024 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c63da6c9ecd88e8429d5fa8e7fac5a4d36a1eca32223dd3bf7e50b7baf1fd5a0"} err="failed to get container status \"c63da6c9ecd88e8429d5fa8e7fac5a4d36a1eca32223dd3bf7e50b7baf1fd5a0\": rpc error: code = NotFound desc = could not find container \"c63da6c9ecd88e8429d5fa8e7fac5a4d36a1eca32223dd3bf7e50b7baf1fd5a0\": container with ID starting with c63da6c9ecd88e8429d5fa8e7fac5a4d36a1eca32223dd3bf7e50b7baf1fd5a0 not found: ID does not exist" Oct 30 03:43:31.395142 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:31.395108 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866"] Oct 30 03:43:31.401474 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:31.401440 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-55cb99746c-pt866"] Oct 30 03:43:32.010541 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:32.010512 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d6687f-104e-4b6a-a59e-2641896c52ea" path="/var/lib/kubelet/pods/23d6687f-104e-4b6a-a59e-2641896c52ea/volumes" Oct 30 03:43:38.288700 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:38.288656 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt"] Oct 30 03:43:38.289104 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:38.289044 5882 cpu_manager.go:414] "containerMap: removing container" podUID="23d6687f-104e-4b6a-a59e-2641896c52ea" containerName="ta-container" Oct 30 03:43:38.289104 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:38.289056 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d6687f-104e-4b6a-a59e-2641896c52ea" containerName="ta-container" Oct 30 03:43:38.289173 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:38.289144 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d6687f-104e-4b6a-a59e-2641896c52ea" containerName="ta-container" Oct 30 03:43:38.292211 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:38.292195 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" Oct 30 03:43:38.322233 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:38.322201 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt"] Oct 30 03:43:38.368976 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:38.368940 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-599bc5d685-zhqqt\" (UID: \"60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" Oct 30 03:43:38.369115 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:38.368990 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-ta-internal\") pod \"data-science-collector-targetallocator-599bc5d685-zhqqt\" (UID: \"60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" Oct 30 03:43:38.369115 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:38.369058 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mvdz\" (UniqueName: \"kubernetes.io/projected/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-kube-api-access-7mvdz\") pod \"data-science-collector-targetallocator-599bc5d685-zhqqt\" (UID: \"60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" Oct 30 03:43:38.470180 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:38.470147 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-599bc5d685-zhqqt\" (UID: \"60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" Oct 30 03:43:38.470326 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:38.470196 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-ta-internal\") pod \"data-science-collector-targetallocator-599bc5d685-zhqqt\" (UID: \"60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" Oct 30 03:43:38.470326 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:38.470221 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mvdz\" (UniqueName: \"kubernetes.io/projected/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-kube-api-access-7mvdz\") pod \"data-science-collector-targetallocator-599bc5d685-zhqqt\" (UID: \"60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" Oct 30 03:43:38.470858 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:38.470835 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-ta-internal\") pod \"data-science-collector-targetallocator-599bc5d685-zhqqt\" (UID: \"60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" Oct 30 03:43:38.472689 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:38.472668 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-599bc5d685-zhqqt\" (UID: \"60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" Oct 30 03:43:38.493736 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:38.493708 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mvdz\" (UniqueName: \"kubernetes.io/projected/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-kube-api-access-7mvdz\") pod \"data-science-collector-targetallocator-599bc5d685-zhqqt\" (UID: \"60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" Oct 30 03:43:38.604131 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:38.604044 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" Oct 30 03:43:38.907031 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:38.906961 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt"] Oct 30 03:43:38.909675 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:43:38.909641 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60fa5e27_b4a0_4b87_87f2_8eeed0ea9f5d.slice/crio-ccf987b3ceb22c467ec39829100a8527d28c49c47e47f554dc6b1dd5b3e7b66c WatchSource:0}: Error finding container ccf987b3ceb22c467ec39829100a8527d28c49c47e47f554dc6b1dd5b3e7b66c: Status 404 returned error can't find the container with id ccf987b3ceb22c467ec39829100a8527d28c49c47e47f554dc6b1dd5b3e7b66c Oct 30 03:43:38.911815 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:38.911784 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:43:38.911923 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:38.911842 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:43:39.383767 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:39.383728 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" event={"ID":"60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d","Type":"ContainerStarted","Data":"299003e7881a00a762fc6e5255950031462670044c26ed5148c246ad8c8dc945"} Oct 30 03:43:39.384179 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:39.383776 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" event={"ID":"60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d","Type":"ContainerStarted","Data":"ccf987b3ceb22c467ec39829100a8527d28c49c47e47f554dc6b1dd5b3e7b66c"} Oct 30 03:43:39.384179 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:39.384028 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" Oct 30 03:43:39.385719 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:39.385694 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" Oct 30 03:43:39.424493 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:39.424444 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" podStartSLOduration=1.424429103 podStartE2EDuration="1.424429103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:39.423689066 +0000 UTC m=+3731.538004040" watchObservedRunningTime="2026-10-30 03:43:39.424429103 +0000 UTC m=+3731.538744077" Oct 30 03:43:39.542892 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:39.542858 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d"] Oct 30 03:43:39.543274 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:39.543223 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" podUID="77757029-564d-46f0-9b5e-a95764c4c1e8" containerName="ta-container" containerID="cri-o://ffd1e638f15356abb3c43dbf8b0ea4b822cc07bc801109da575b81981653f701" gracePeriod=30 Oct 30 03:43:39.788290 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:39.788265 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" Oct 30 03:43:39.883592 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:39.883562 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77757029-564d-46f0-9b5e-a95764c4c1e8-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/77757029-564d-46f0-9b5e-a95764c4c1e8-data-science-collector-ta-server-cert\") pod \"77757029-564d-46f0-9b5e-a95764c4c1e8\" (UID: \"77757029-564d-46f0-9b5e-a95764c4c1e8\") " Oct 30 03:43:39.883766 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:39.883695 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77757029-564d-46f0-9b5e-a95764c4c1e8-ta-internal\" (UniqueName: \"kubernetes.io/configmap/77757029-564d-46f0-9b5e-a95764c4c1e8-ta-internal\") pod \"77757029-564d-46f0-9b5e-a95764c4c1e8\" (UID: \"77757029-564d-46f0-9b5e-a95764c4c1e8\") " Oct 30 03:43:39.883766 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:39.883753 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77757029-564d-46f0-9b5e-a95764c4c1e8-kube-api-access-n8zj6\" (UniqueName: \"kubernetes.io/projected/77757029-564d-46f0-9b5e-a95764c4c1e8-kube-api-access-n8zj6\") pod \"77757029-564d-46f0-9b5e-a95764c4c1e8\" (UID: \"77757029-564d-46f0-9b5e-a95764c4c1e8\") " Oct 30 03:43:39.884124 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:39.884091 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77757029-564d-46f0-9b5e-a95764c4c1e8-ta-internal" pod "77757029-564d-46f0-9b5e-a95764c4c1e8" (UID: "77757029-564d-46f0-9b5e-a95764c4c1e8"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:43:39.885699 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:39.885673 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77757029-564d-46f0-9b5e-a95764c4c1e8-kube-api-access-n8zj6" pod "77757029-564d-46f0-9b5e-a95764c4c1e8" (UID: "77757029-564d-46f0-9b5e-a95764c4c1e8"). InnerVolumeSpecName "kube-api-access-n8zj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:43:39.885699 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:39.885684 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77757029-564d-46f0-9b5e-a95764c4c1e8-data-science-collector-ta-server-cert" pod "77757029-564d-46f0-9b5e-a95764c4c1e8" (UID: "77757029-564d-46f0-9b5e-a95764c4c1e8"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:43:39.985292 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:39.985209 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/77757029-564d-46f0-9b5e-a95764c4c1e8-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:43:39.985292 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:39.985240 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/77757029-564d-46f0-9b5e-a95764c4c1e8-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:43:39.985292 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:39.985252 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8zj6\" (UniqueName: \"kubernetes.io/projected/77757029-564d-46f0-9b5e-a95764c4c1e8-kube-api-access-n8zj6\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:43:40.389065 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:40.389030 5882 generic.go:356] "Generic (PLEG): container finished" podID="77757029-564d-46f0-9b5e-a95764c4c1e8" containerID="ffd1e638f15356abb3c43dbf8b0ea4b822cc07bc801109da575b81981653f701" exitCode=0 Oct 30 03:43:40.389497 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:40.389089 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" Oct 30 03:43:40.389497 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:40.389103 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" event={"ID":"77757029-564d-46f0-9b5e-a95764c4c1e8","Type":"ContainerDied","Data":"ffd1e638f15356abb3c43dbf8b0ea4b822cc07bc801109da575b81981653f701"} Oct 30 03:43:40.389497 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:40.389145 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d" event={"ID":"77757029-564d-46f0-9b5e-a95764c4c1e8","Type":"ContainerDied","Data":"48c09fc315c15b8d3ca850f62fde32e585dee3a17902ffc98599e8165a4356df"} Oct 30 03:43:40.389497 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:40.389162 5882 scope.go:122] "RemoveContainer" containerID="ffd1e638f15356abb3c43dbf8b0ea4b822cc07bc801109da575b81981653f701" Oct 30 03:43:40.397416 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:40.397394 5882 scope.go:122] "RemoveContainer" containerID="ffd1e638f15356abb3c43dbf8b0ea4b822cc07bc801109da575b81981653f701" Oct 30 03:43:40.397711 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:43:40.397683 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffd1e638f15356abb3c43dbf8b0ea4b822cc07bc801109da575b81981653f701\": container with ID starting with ffd1e638f15356abb3c43dbf8b0ea4b822cc07bc801109da575b81981653f701 not found: ID does not exist" containerID="ffd1e638f15356abb3c43dbf8b0ea4b822cc07bc801109da575b81981653f701" Oct 30 03:43:40.397818 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:40.397719 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd1e638f15356abb3c43dbf8b0ea4b822cc07bc801109da575b81981653f701"} err="failed to get container status \"ffd1e638f15356abb3c43dbf8b0ea4b822cc07bc801109da575b81981653f701\": rpc error: code = NotFound desc = could not find container \"ffd1e638f15356abb3c43dbf8b0ea4b822cc07bc801109da575b81981653f701\": container with ID starting with ffd1e638f15356abb3c43dbf8b0ea4b822cc07bc801109da575b81981653f701 not found: ID does not exist" Oct 30 03:43:40.440897 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:40.440870 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d"] Oct 30 03:43:40.451392 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:40.451362 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-64b47bdcbd-8252d"] Oct 30 03:43:42.009384 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:42.009344 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77757029-564d-46f0-9b5e-a95764c4c1e8" path="/var/lib/kubelet/pods/77757029-564d-46f0-9b5e-a95764c4c1e8/volumes" Oct 30 03:43:48.402538 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:48.402506 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv"] Oct 30 03:43:48.403183 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:48.403027 5882 cpu_manager.go:414] "containerMap: removing container" podUID="77757029-564d-46f0-9b5e-a95764c4c1e8" containerName="ta-container" Oct 30 03:43:48.403183 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:48.403043 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77757029-564d-46f0-9b5e-a95764c4c1e8" containerName="ta-container" Oct 30 03:43:48.403183 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:48.403128 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="77757029-564d-46f0-9b5e-a95764c4c1e8" containerName="ta-container" Oct 30 03:43:48.407288 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:48.407269 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" Oct 30 03:43:48.444579 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:48.444551 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv"] Oct 30 03:43:48.558294 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:48.558253 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-ta-internal\") pod \"data-science-collector-targetallocator-69dff74c69-thxtv\" (UID: \"c9a7a517-832f-46ab-9f09-d6a74d28e6c9\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" Oct 30 03:43:48.558474 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:48.558366 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh44v\" (UniqueName: \"kubernetes.io/projected/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-kube-api-access-dh44v\") pod \"data-science-collector-targetallocator-69dff74c69-thxtv\" (UID: \"c9a7a517-832f-46ab-9f09-d6a74d28e6c9\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" Oct 30 03:43:48.558474 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:48.558435 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-69dff74c69-thxtv\" (UID: \"c9a7a517-832f-46ab-9f09-d6a74d28e6c9\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" Oct 30 03:43:48.659422 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:48.659325 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh44v\" (UniqueName: \"kubernetes.io/projected/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-kube-api-access-dh44v\") pod \"data-science-collector-targetallocator-69dff74c69-thxtv\" (UID: \"c9a7a517-832f-46ab-9f09-d6a74d28e6c9\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" Oct 30 03:43:48.659422 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:48.659400 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-69dff74c69-thxtv\" (UID: \"c9a7a517-832f-46ab-9f09-d6a74d28e6c9\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" Oct 30 03:43:48.659733 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:48.659457 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-ta-internal\") pod \"data-science-collector-targetallocator-69dff74c69-thxtv\" (UID: \"c9a7a517-832f-46ab-9f09-d6a74d28e6c9\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" Oct 30 03:43:48.660164 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:48.660138 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-ta-internal\") pod \"data-science-collector-targetallocator-69dff74c69-thxtv\" (UID: \"c9a7a517-832f-46ab-9f09-d6a74d28e6c9\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" Oct 30 03:43:48.661904 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:48.661883 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-69dff74c69-thxtv\" (UID: \"c9a7a517-832f-46ab-9f09-d6a74d28e6c9\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" Oct 30 03:43:48.683949 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:48.683916 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh44v\" (UniqueName: \"kubernetes.io/projected/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-kube-api-access-dh44v\") pod \"data-science-collector-targetallocator-69dff74c69-thxtv\" (UID: \"c9a7a517-832f-46ab-9f09-d6a74d28e6c9\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" Oct 30 03:43:48.719868 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:48.719833 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" Oct 30 03:43:49.026391 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:49.026356 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv"] Oct 30 03:43:49.029734 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:43:49.029702 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9a7a517_832f_46ab_9f09_d6a74d28e6c9.slice/crio-388e2c09f2e27186e27e5848f76050a67645937edfb722b5b87e522d9b3ea965 WatchSource:0}: Error finding container 388e2c09f2e27186e27e5848f76050a67645937edfb722b5b87e522d9b3ea965: Status 404 returned error can't find the container with id 388e2c09f2e27186e27e5848f76050a67645937edfb722b5b87e522d9b3ea965 Oct 30 03:43:49.032097 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:49.032058 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:43:49.032186 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:49.032123 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:43:49.435303 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:49.435219 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" event={"ID":"c9a7a517-832f-46ab-9f09-d6a74d28e6c9","Type":"ContainerStarted","Data":"a4eed2decc3d02be9c5c4102fa17596dad951e6cc591917e9937267376f59af9"} Oct 30 03:43:49.435303 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:49.435267 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" event={"ID":"c9a7a517-832f-46ab-9f09-d6a74d28e6c9","Type":"ContainerStarted","Data":"388e2c09f2e27186e27e5848f76050a67645937edfb722b5b87e522d9b3ea965"} Oct 30 03:43:49.435819 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:49.435384 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" Oct 30 03:43:49.436606 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:49.436579 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" podUID="c9a7a517-832f-46ab-9f09-d6a74d28e6c9" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/readyz\": dial tcp 10.134.0.47:8080: connect: connection refused" Oct 30 03:43:49.490286 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:49.490230 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" podStartSLOduration=1.490215891 podStartE2EDuration="1.490215891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:49.489552106 +0000 UTC m=+3741.603867081" watchObservedRunningTime="2026-10-30 03:43:49.490215891 +0000 UTC m=+3741.604530865" Oct 30 03:43:50.440366 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:50.440340 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" Oct 30 03:43:50.517564 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:50.517526 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt"] Oct 30 03:43:50.517955 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:50.517788 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" podUID="60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d" containerName="ta-container" containerID="cri-o://299003e7881a00a762fc6e5255950031462670044c26ed5148c246ad8c8dc945" gracePeriod=30 Oct 30 03:43:50.776882 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:50.776860 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" Oct 30 03:43:50.879052 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:50.879021 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-ta-internal\" (UniqueName: \"kubernetes.io/configmap/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-ta-internal\") pod \"60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d\" (UID: \"60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d\") " Oct 30 03:43:50.879214 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:50.879107 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-data-science-collector-ta-server-cert\") pod \"60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d\" (UID: \"60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d\") " Oct 30 03:43:50.879214 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:50.879187 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-kube-api-access-7mvdz\" (UniqueName: \"kubernetes.io/projected/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-kube-api-access-7mvdz\") pod \"60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d\" (UID: \"60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d\") " Oct 30 03:43:50.879363 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:50.879341 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-ta-internal" pod "60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d" (UID: "60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:43:50.879436 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:50.879424 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:43:50.881171 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:50.881142 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-kube-api-access-7mvdz" pod "60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d" (UID: "60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d"). InnerVolumeSpecName "kube-api-access-7mvdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:43:50.881283 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:50.881184 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-data-science-collector-ta-server-cert" pod "60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d" (UID: "60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:43:50.980069 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:50.980035 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:43:50.980069 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:50.980067 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mvdz\" (UniqueName: \"kubernetes.io/projected/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d-kube-api-access-7mvdz\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:43:51.445140 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:51.445105 5882 generic.go:356] "Generic (PLEG): container finished" podID="60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d" containerID="299003e7881a00a762fc6e5255950031462670044c26ed5148c246ad8c8dc945" exitCode=0 Oct 30 03:43:51.445548 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:51.445163 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" Oct 30 03:43:51.445548 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:51.445191 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" event={"ID":"60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d","Type":"ContainerDied","Data":"299003e7881a00a762fc6e5255950031462670044c26ed5148c246ad8c8dc945"} Oct 30 03:43:51.445548 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:51.445234 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt" event={"ID":"60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d","Type":"ContainerDied","Data":"ccf987b3ceb22c467ec39829100a8527d28c49c47e47f554dc6b1dd5b3e7b66c"} Oct 30 03:43:51.445548 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:51.445251 5882 scope.go:122] "RemoveContainer" containerID="299003e7881a00a762fc6e5255950031462670044c26ed5148c246ad8c8dc945" Oct 30 03:43:51.454699 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:51.454669 5882 scope.go:122] "RemoveContainer" containerID="299003e7881a00a762fc6e5255950031462670044c26ed5148c246ad8c8dc945" Oct 30 03:43:51.456723 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:43:51.456700 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"299003e7881a00a762fc6e5255950031462670044c26ed5148c246ad8c8dc945\": container with ID starting with 299003e7881a00a762fc6e5255950031462670044c26ed5148c246ad8c8dc945 not found: ID does not exist" containerID="299003e7881a00a762fc6e5255950031462670044c26ed5148c246ad8c8dc945" Oct 30 03:43:51.456817 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:51.456728 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"299003e7881a00a762fc6e5255950031462670044c26ed5148c246ad8c8dc945"} err="failed to get container status \"299003e7881a00a762fc6e5255950031462670044c26ed5148c246ad8c8dc945\": rpc error: code = NotFound desc = could not find container \"299003e7881a00a762fc6e5255950031462670044c26ed5148c246ad8c8dc945\": container with ID starting with 299003e7881a00a762fc6e5255950031462670044c26ed5148c246ad8c8dc945 not found: ID does not exist" Oct 30 03:43:51.495181 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:51.495151 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt"] Oct 30 03:43:51.506357 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:51.506331 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-599bc5d685-zhqqt"] Oct 30 03:43:52.009854 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:52.009822 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d" path="/var/lib/kubelet/pods/60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d/volumes" Oct 30 03:43:53.942231 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:43:53.942191 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:43:57.937635 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:57.937584 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c"] Oct 30 03:43:57.938184 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:57.938157 5882 cpu_manager.go:414] "containerMap: removing container" podUID="60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d" containerName="ta-container" Oct 30 03:43:57.938184 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:57.938184 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d" containerName="ta-container" Oct 30 03:43:57.938386 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:57.938308 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="60fa5e27-b4a0-4b87-87f2-8eeed0ea9f5d" containerName="ta-container" Oct 30 03:43:57.942740 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:57.942719 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" Oct 30 03:43:57.965147 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:57.965116 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c"] Oct 30 03:43:58.043761 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:58.043728 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/61be905a-63c4-432b-89d3-a67ac4a286bf-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-784b75b896-c5f7c\" (UID: \"61be905a-63c4-432b-89d3-a67ac4a286bf\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" Oct 30 03:43:58.043761 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:58.043763 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/61be905a-63c4-432b-89d3-a67ac4a286bf-ta-internal\") pod \"data-science-collector-targetallocator-784b75b896-c5f7c\" (UID: \"61be905a-63c4-432b-89d3-a67ac4a286bf\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" Oct 30 03:43:58.043956 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:58.043790 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45bjt\" (UniqueName: \"kubernetes.io/projected/61be905a-63c4-432b-89d3-a67ac4a286bf-kube-api-access-45bjt\") pod \"data-science-collector-targetallocator-784b75b896-c5f7c\" (UID: \"61be905a-63c4-432b-89d3-a67ac4a286bf\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" Oct 30 03:43:58.144901 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:58.144864 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/61be905a-63c4-432b-89d3-a67ac4a286bf-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-784b75b896-c5f7c\" (UID: \"61be905a-63c4-432b-89d3-a67ac4a286bf\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" Oct 30 03:43:58.144901 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:58.144902 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/61be905a-63c4-432b-89d3-a67ac4a286bf-ta-internal\") pod \"data-science-collector-targetallocator-784b75b896-c5f7c\" (UID: \"61be905a-63c4-432b-89d3-a67ac4a286bf\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" Oct 30 03:43:58.145109 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:58.145038 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45bjt\" (UniqueName: \"kubernetes.io/projected/61be905a-63c4-432b-89d3-a67ac4a286bf-kube-api-access-45bjt\") pod \"data-science-collector-targetallocator-784b75b896-c5f7c\" (UID: \"61be905a-63c4-432b-89d3-a67ac4a286bf\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" Oct 30 03:43:58.145609 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:58.145579 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/61be905a-63c4-432b-89d3-a67ac4a286bf-ta-internal\") pod \"data-science-collector-targetallocator-784b75b896-c5f7c\" (UID: \"61be905a-63c4-432b-89d3-a67ac4a286bf\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" Oct 30 03:43:58.147370 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:58.147347 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/61be905a-63c4-432b-89d3-a67ac4a286bf-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-784b75b896-c5f7c\" (UID: \"61be905a-63c4-432b-89d3-a67ac4a286bf\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" Oct 30 03:43:58.168070 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:58.168045 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45bjt\" (UniqueName: \"kubernetes.io/projected/61be905a-63c4-432b-89d3-a67ac4a286bf-kube-api-access-45bjt\") pod \"data-science-collector-targetallocator-784b75b896-c5f7c\" (UID: \"61be905a-63c4-432b-89d3-a67ac4a286bf\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" Oct 30 03:43:58.254554 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:58.254522 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" Oct 30 03:43:58.543210 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:58.543121 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c"] Oct 30 03:43:58.546749 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:43:58.546719 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61be905a_63c4_432b_89d3_a67ac4a286bf.slice/crio-dc921d72c7c074914446eee3051d2ef3026a795f538c141290fb902657db728b WatchSource:0}: Error finding container dc921d72c7c074914446eee3051d2ef3026a795f538c141290fb902657db728b: Status 404 returned error can't find the container with id dc921d72c7c074914446eee3051d2ef3026a795f538c141290fb902657db728b Oct 30 03:43:58.549008 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:58.548978 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:43:58.549087 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:58.549034 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:43:59.482595 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:59.482557 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" event={"ID":"61be905a-63c4-432b-89d3-a67ac4a286bf","Type":"ContainerStarted","Data":"fa5a95ffb84f835cb20f2216d077cbc8ef4c87d5d06e44152196700fc5698dd2"} Oct 30 03:43:59.482595 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:59.482601 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" event={"ID":"61be905a-63c4-432b-89d3-a67ac4a286bf","Type":"ContainerStarted","Data":"dc921d72c7c074914446eee3051d2ef3026a795f538c141290fb902657db728b"} Oct 30 03:43:59.483072 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:59.482751 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" Oct 30 03:43:59.484371 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:59.484355 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" Oct 30 03:43:59.525236 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:59.525187 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" podStartSLOduration=2.525174996 podStartE2EDuration="2.525174996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:59.524724708 +0000 UTC m=+3751.639039683" watchObservedRunningTime="2026-10-30 03:43:59.525174996 +0000 UTC m=+3751.639489970" Oct 30 03:43:59.595264 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:59.595229 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv"] Oct 30 03:43:59.595558 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:59.595518 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" podUID="c9a7a517-832f-46ab-9f09-d6a74d28e6c9" containerName="ta-container" containerID="cri-o://a4eed2decc3d02be9c5c4102fa17596dad951e6cc591917e9937267376f59af9" gracePeriod=30 Oct 30 03:43:59.838891 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:59.838858 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" Oct 30 03:43:59.962918 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:59.962882 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-data-science-collector-ta-server-cert\") pod \"c9a7a517-832f-46ab-9f09-d6a74d28e6c9\" (UID: \"c9a7a517-832f-46ab-9f09-d6a74d28e6c9\") " Oct 30 03:43:59.963082 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:59.963036 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-ta-internal\" (UniqueName: \"kubernetes.io/configmap/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-ta-internal\") pod \"c9a7a517-832f-46ab-9f09-d6a74d28e6c9\" (UID: \"c9a7a517-832f-46ab-9f09-d6a74d28e6c9\") " Oct 30 03:43:59.963124 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:59.963107 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-kube-api-access-dh44v\" (UniqueName: \"kubernetes.io/projected/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-kube-api-access-dh44v\") pod \"c9a7a517-832f-46ab-9f09-d6a74d28e6c9\" (UID: \"c9a7a517-832f-46ab-9f09-d6a74d28e6c9\") " Oct 30 03:43:59.963318 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:59.963296 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-ta-internal" pod "c9a7a517-832f-46ab-9f09-d6a74d28e6c9" (UID: "c9a7a517-832f-46ab-9f09-d6a74d28e6c9"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:43:59.965060 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:59.965023 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-data-science-collector-ta-server-cert" pod "c9a7a517-832f-46ab-9f09-d6a74d28e6c9" (UID: "c9a7a517-832f-46ab-9f09-d6a74d28e6c9"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:43:59.965060 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:43:59.965045 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-kube-api-access-dh44v" pod "c9a7a517-832f-46ab-9f09-d6a74d28e6c9" (UID: "c9a7a517-832f-46ab-9f09-d6a74d28e6c9"). InnerVolumeSpecName "kube-api-access-dh44v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:00.063905 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:00.063873 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh44v\" (UniqueName: \"kubernetes.io/projected/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-kube-api-access-dh44v\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:00.063905 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:00.063900 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:00.063905 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:00.063912 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/c9a7a517-832f-46ab-9f09-d6a74d28e6c9-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:00.488796 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:00.488713 5882 generic.go:356] "Generic (PLEG): container finished" podID="c9a7a517-832f-46ab-9f09-d6a74d28e6c9" containerID="a4eed2decc3d02be9c5c4102fa17596dad951e6cc591917e9937267376f59af9" exitCode=0 Oct 30 03:44:00.488796 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:00.488764 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" Oct 30 03:44:00.489248 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:00.488793 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" event={"ID":"c9a7a517-832f-46ab-9f09-d6a74d28e6c9","Type":"ContainerDied","Data":"a4eed2decc3d02be9c5c4102fa17596dad951e6cc591917e9937267376f59af9"} Oct 30 03:44:00.489248 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:00.488834 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv" event={"ID":"c9a7a517-832f-46ab-9f09-d6a74d28e6c9","Type":"ContainerDied","Data":"388e2c09f2e27186e27e5848f76050a67645937edfb722b5b87e522d9b3ea965"} Oct 30 03:44:00.489248 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:00.488851 5882 scope.go:122] "RemoveContainer" containerID="a4eed2decc3d02be9c5c4102fa17596dad951e6cc591917e9937267376f59af9" Oct 30 03:44:00.497037 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:00.497019 5882 scope.go:122] "RemoveContainer" containerID="a4eed2decc3d02be9c5c4102fa17596dad951e6cc591917e9937267376f59af9" Oct 30 03:44:00.497284 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:44:00.497267 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4eed2decc3d02be9c5c4102fa17596dad951e6cc591917e9937267376f59af9\": container with ID starting with a4eed2decc3d02be9c5c4102fa17596dad951e6cc591917e9937267376f59af9 not found: ID does not exist" containerID="a4eed2decc3d02be9c5c4102fa17596dad951e6cc591917e9937267376f59af9" Oct 30 03:44:00.497332 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:00.497292 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4eed2decc3d02be9c5c4102fa17596dad951e6cc591917e9937267376f59af9"} err="failed to get container status \"a4eed2decc3d02be9c5c4102fa17596dad951e6cc591917e9937267376f59af9\": rpc error: code = NotFound desc = could not find container \"a4eed2decc3d02be9c5c4102fa17596dad951e6cc591917e9937267376f59af9\": container with ID starting with a4eed2decc3d02be9c5c4102fa17596dad951e6cc591917e9937267376f59af9 not found: ID does not exist" Oct 30 03:44:00.537692 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:00.537664 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv"] Oct 30 03:44:00.549093 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:00.549070 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-69dff74c69-thxtv"] Oct 30 03:44:02.009707 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:02.009674 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9a7a517-832f-46ab-9f09-d6a74d28e6c9" path="/var/lib/kubelet/pods/c9a7a517-832f-46ab-9f09-d6a74d28e6c9/volumes" Oct 30 03:44:06.939437 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:06.939401 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj"] Oct 30 03:44:06.939838 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:06.939796 5882 cpu_manager.go:414] "containerMap: removing container" podUID="c9a7a517-832f-46ab-9f09-d6a74d28e6c9" containerName="ta-container" Oct 30 03:44:06.939838 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:06.939808 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9a7a517-832f-46ab-9f09-d6a74d28e6c9" containerName="ta-container" Oct 30 03:44:06.939920 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:06.939881 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9a7a517-832f-46ab-9f09-d6a74d28e6c9" containerName="ta-container" Oct 30 03:44:06.942911 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:06.942892 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" Oct 30 03:44:06.964251 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:06.964225 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj"] Oct 30 03:44:07.017838 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:07.017808 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts48v\" (UniqueName: \"kubernetes.io/projected/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-kube-api-access-ts48v\") pod \"data-science-collector-targetallocator-76649584c9-t4kkj\" (UID: \"6aa3b896-b2f3-4e8a-b733-12572a0d88d4\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" Oct 30 03:44:07.018008 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:07.017867 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-ta-internal\") pod \"data-science-collector-targetallocator-76649584c9-t4kkj\" (UID: \"6aa3b896-b2f3-4e8a-b733-12572a0d88d4\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" Oct 30 03:44:07.018094 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:07.018073 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-76649584c9-t4kkj\" (UID: \"6aa3b896-b2f3-4e8a-b733-12572a0d88d4\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" Oct 30 03:44:07.118610 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:07.118561 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts48v\" (UniqueName: \"kubernetes.io/projected/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-kube-api-access-ts48v\") pod \"data-science-collector-targetallocator-76649584c9-t4kkj\" (UID: \"6aa3b896-b2f3-4e8a-b733-12572a0d88d4\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" Oct 30 03:44:07.118797 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:07.118678 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-ta-internal\") pod \"data-science-collector-targetallocator-76649584c9-t4kkj\" (UID: \"6aa3b896-b2f3-4e8a-b733-12572a0d88d4\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" Oct 30 03:44:07.118797 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:07.118742 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-76649584c9-t4kkj\" (UID: \"6aa3b896-b2f3-4e8a-b733-12572a0d88d4\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" Oct 30 03:44:07.119394 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:07.119369 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-ta-internal\") pod \"data-science-collector-targetallocator-76649584c9-t4kkj\" (UID: \"6aa3b896-b2f3-4e8a-b733-12572a0d88d4\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" Oct 30 03:44:07.121277 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:07.121247 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-76649584c9-t4kkj\" (UID: \"6aa3b896-b2f3-4e8a-b733-12572a0d88d4\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" Oct 30 03:44:07.154725 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:07.154694 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts48v\" (UniqueName: \"kubernetes.io/projected/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-kube-api-access-ts48v\") pod \"data-science-collector-targetallocator-76649584c9-t4kkj\" (UID: \"6aa3b896-b2f3-4e8a-b733-12572a0d88d4\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" Oct 30 03:44:07.254193 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:07.254164 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" Oct 30 03:44:07.552153 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:07.552120 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj"] Oct 30 03:44:07.554693 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:44:07.554665 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6aa3b896_b2f3_4e8a_b733_12572a0d88d4.slice/crio-9ac0ea8a43495e3e584ff0a0da5040ccf4a10dea1289a944c7c96b35084ce210 WatchSource:0}: Error finding container 9ac0ea8a43495e3e584ff0a0da5040ccf4a10dea1289a944c7c96b35084ce210: Status 404 returned error can't find the container with id 9ac0ea8a43495e3e584ff0a0da5040ccf4a10dea1289a944c7c96b35084ce210 Oct 30 03:44:07.556820 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:07.556788 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:07.556917 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:07.556845 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:08.525794 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:08.525758 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" event={"ID":"6aa3b896-b2f3-4e8a-b733-12572a0d88d4","Type":"ContainerStarted","Data":"e9d04a66f80e63b2a43783694b56c1e4f7f4e9fbaa9ebfc2e4bd8a17fb5613f5"} Oct 30 03:44:08.526214 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:08.525804 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" event={"ID":"6aa3b896-b2f3-4e8a-b733-12572a0d88d4","Type":"ContainerStarted","Data":"9ac0ea8a43495e3e584ff0a0da5040ccf4a10dea1289a944c7c96b35084ce210"} Oct 30 03:44:08.526214 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:08.525971 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" Oct 30 03:44:08.527760 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:08.527738 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" Oct 30 03:44:08.566813 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:08.566766 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" podStartSLOduration=2.566748454 podStartE2EDuration="2.566748454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:08.565761386 +0000 UTC m=+3760.680076362" watchObservedRunningTime="2026-10-30 03:44:08.566748454 +0000 UTC m=+3760.681063431" Oct 30 03:44:08.669194 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:08.669161 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c"] Oct 30 03:44:08.669369 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:08.669348 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" podUID="61be905a-63c4-432b-89d3-a67ac4a286bf" containerName="ta-container" containerID="cri-o://fa5a95ffb84f835cb20f2216d077cbc8ef4c87d5d06e44152196700fc5698dd2" gracePeriod=30 Oct 30 03:44:08.913003 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:08.912975 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" Oct 30 03:44:08.931555 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:08.931533 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61be905a-63c4-432b-89d3-a67ac4a286bf-ta-internal\" (UniqueName: \"kubernetes.io/configmap/61be905a-63c4-432b-89d3-a67ac4a286bf-ta-internal\") pod \"61be905a-63c4-432b-89d3-a67ac4a286bf\" (UID: \"61be905a-63c4-432b-89d3-a67ac4a286bf\") " Oct 30 03:44:08.931680 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:08.931583 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61be905a-63c4-432b-89d3-a67ac4a286bf-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/61be905a-63c4-432b-89d3-a67ac4a286bf-data-science-collector-ta-server-cert\") pod \"61be905a-63c4-432b-89d3-a67ac4a286bf\" (UID: \"61be905a-63c4-432b-89d3-a67ac4a286bf\") " Oct 30 03:44:08.931680 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:08.931650 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61be905a-63c4-432b-89d3-a67ac4a286bf-kube-api-access-45bjt\" (UniqueName: \"kubernetes.io/projected/61be905a-63c4-432b-89d3-a67ac4a286bf-kube-api-access-45bjt\") pod \"61be905a-63c4-432b-89d3-a67ac4a286bf\" (UID: \"61be905a-63c4-432b-89d3-a67ac4a286bf\") " Oct 30 03:44:08.931934 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:08.931907 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61be905a-63c4-432b-89d3-a67ac4a286bf-ta-internal" pod "61be905a-63c4-432b-89d3-a67ac4a286bf" (UID: "61be905a-63c4-432b-89d3-a67ac4a286bf"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:08.933527 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:08.933502 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61be905a-63c4-432b-89d3-a67ac4a286bf-data-science-collector-ta-server-cert" pod "61be905a-63c4-432b-89d3-a67ac4a286bf" (UID: "61be905a-63c4-432b-89d3-a67ac4a286bf"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:08.933600 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:08.933580 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61be905a-63c4-432b-89d3-a67ac4a286bf-kube-api-access-45bjt" pod "61be905a-63c4-432b-89d3-a67ac4a286bf" (UID: "61be905a-63c4-432b-89d3-a67ac4a286bf"). InnerVolumeSpecName "kube-api-access-45bjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:09.032462 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:09.032426 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/61be905a-63c4-432b-89d3-a67ac4a286bf-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:09.032462 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:09.032460 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/61be905a-63c4-432b-89d3-a67ac4a286bf-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:09.032462 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:09.032471 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45bjt\" (UniqueName: \"kubernetes.io/projected/61be905a-63c4-432b-89d3-a67ac4a286bf-kube-api-access-45bjt\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:09.531264 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:09.531229 5882 generic.go:356] "Generic (PLEG): container finished" podID="61be905a-63c4-432b-89d3-a67ac4a286bf" containerID="fa5a95ffb84f835cb20f2216d077cbc8ef4c87d5d06e44152196700fc5698dd2" exitCode=0 Oct 30 03:44:09.531695 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:09.531273 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" Oct 30 03:44:09.531695 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:09.531314 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" event={"ID":"61be905a-63c4-432b-89d3-a67ac4a286bf","Type":"ContainerDied","Data":"fa5a95ffb84f835cb20f2216d077cbc8ef4c87d5d06e44152196700fc5698dd2"} Oct 30 03:44:09.531695 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:09.531357 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c" event={"ID":"61be905a-63c4-432b-89d3-a67ac4a286bf","Type":"ContainerDied","Data":"dc921d72c7c074914446eee3051d2ef3026a795f538c141290fb902657db728b"} Oct 30 03:44:09.531695 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:09.531378 5882 scope.go:122] "RemoveContainer" containerID="fa5a95ffb84f835cb20f2216d077cbc8ef4c87d5d06e44152196700fc5698dd2" Oct 30 03:44:09.541708 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:09.541682 5882 scope.go:122] "RemoveContainer" containerID="fa5a95ffb84f835cb20f2216d077cbc8ef4c87d5d06e44152196700fc5698dd2" Oct 30 03:44:09.542103 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:44:09.542065 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa5a95ffb84f835cb20f2216d077cbc8ef4c87d5d06e44152196700fc5698dd2\": container with ID starting with fa5a95ffb84f835cb20f2216d077cbc8ef4c87d5d06e44152196700fc5698dd2 not found: ID does not exist" containerID="fa5a95ffb84f835cb20f2216d077cbc8ef4c87d5d06e44152196700fc5698dd2" Oct 30 03:44:09.542216 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:09.542114 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa5a95ffb84f835cb20f2216d077cbc8ef4c87d5d06e44152196700fc5698dd2"} err="failed to get container status \"fa5a95ffb84f835cb20f2216d077cbc8ef4c87d5d06e44152196700fc5698dd2\": rpc error: code = NotFound desc = could not find container \"fa5a95ffb84f835cb20f2216d077cbc8ef4c87d5d06e44152196700fc5698dd2\": container with ID starting with fa5a95ffb84f835cb20f2216d077cbc8ef4c87d5d06e44152196700fc5698dd2 not found: ID does not exist" Oct 30 03:44:09.581083 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:09.581051 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c"] Oct 30 03:44:09.592040 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:09.592017 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-784b75b896-c5f7c"] Oct 30 03:44:10.008813 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:10.008784 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61be905a-63c4-432b-89d3-a67ac4a286bf" path="/var/lib/kubelet/pods/61be905a-63c4-432b-89d3-a67ac4a286bf/volumes" Oct 30 03:44:15.840192 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:15.840156 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc"] Oct 30 03:44:15.840592 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:15.840564 5882 cpu_manager.go:414] "containerMap: removing container" podUID="61be905a-63c4-432b-89d3-a67ac4a286bf" containerName="ta-container" Oct 30 03:44:15.840592 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:15.840576 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61be905a-63c4-432b-89d3-a67ac4a286bf" containerName="ta-container" Oct 30 03:44:15.840705 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:15.840674 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="61be905a-63c4-432b-89d3-a67ac4a286bf" containerName="ta-container" Oct 30 03:44:15.845043 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:15.845025 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" Oct 30 03:44:15.883681 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:15.883642 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc"] Oct 30 03:44:15.983899 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:15.983869 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/11758052-3bf8-4f77-83f0-3e097c6fc90c-ta-internal\") pod \"data-science-collector-targetallocator-64c756b9d-dgqxc\" (UID: \"11758052-3bf8-4f77-83f0-3e097c6fc90c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" Oct 30 03:44:15.984129 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:15.983924 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/11758052-3bf8-4f77-83f0-3e097c6fc90c-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-64c756b9d-dgqxc\" (UID: \"11758052-3bf8-4f77-83f0-3e097c6fc90c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" Oct 30 03:44:15.984129 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:15.984007 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x76z2\" (UniqueName: \"kubernetes.io/projected/11758052-3bf8-4f77-83f0-3e097c6fc90c-kube-api-access-x76z2\") pod \"data-science-collector-targetallocator-64c756b9d-dgqxc\" (UID: \"11758052-3bf8-4f77-83f0-3e097c6fc90c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" Oct 30 03:44:16.084709 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:16.084677 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/11758052-3bf8-4f77-83f0-3e097c6fc90c-ta-internal\") pod \"data-science-collector-targetallocator-64c756b9d-dgqxc\" (UID: \"11758052-3bf8-4f77-83f0-3e097c6fc90c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" Oct 30 03:44:16.085399 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:16.085360 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/11758052-3bf8-4f77-83f0-3e097c6fc90c-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-64c756b9d-dgqxc\" (UID: \"11758052-3bf8-4f77-83f0-3e097c6fc90c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" Oct 30 03:44:16.085534 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:16.085420 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x76z2\" (UniqueName: \"kubernetes.io/projected/11758052-3bf8-4f77-83f0-3e097c6fc90c-kube-api-access-x76z2\") pod \"data-science-collector-targetallocator-64c756b9d-dgqxc\" (UID: \"11758052-3bf8-4f77-83f0-3e097c6fc90c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" Oct 30 03:44:16.085608 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:16.085534 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/11758052-3bf8-4f77-83f0-3e097c6fc90c-ta-internal\") pod \"data-science-collector-targetallocator-64c756b9d-dgqxc\" (UID: \"11758052-3bf8-4f77-83f0-3e097c6fc90c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" Oct 30 03:44:16.087848 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:16.087819 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/11758052-3bf8-4f77-83f0-3e097c6fc90c-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-64c756b9d-dgqxc\" (UID: \"11758052-3bf8-4f77-83f0-3e097c6fc90c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" Oct 30 03:44:16.111525 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:16.111458 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x76z2\" (UniqueName: \"kubernetes.io/projected/11758052-3bf8-4f77-83f0-3e097c6fc90c-kube-api-access-x76z2\") pod \"data-science-collector-targetallocator-64c756b9d-dgqxc\" (UID: \"11758052-3bf8-4f77-83f0-3e097c6fc90c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" Oct 30 03:44:16.159477 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:16.159443 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" Oct 30 03:44:16.451811 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:16.451690 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc"] Oct 30 03:44:16.454691 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:44:16.454667 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11758052_3bf8_4f77_83f0_3e097c6fc90c.slice/crio-a5e0266f2a665d664bb78879ac1d2e61a4f882c355b7ebef8d6e6d04e131c3e9 WatchSource:0}: Error finding container a5e0266f2a665d664bb78879ac1d2e61a4f882c355b7ebef8d6e6d04e131c3e9: Status 404 returned error can't find the container with id a5e0266f2a665d664bb78879ac1d2e61a4f882c355b7ebef8d6e6d04e131c3e9 Oct 30 03:44:16.456873 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:16.456843 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:16.456963 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:16.456899 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:16.572481 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:16.572451 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" event={"ID":"11758052-3bf8-4f77-83f0-3e097c6fc90c","Type":"ContainerStarted","Data":"b2bd281471bd1dad348b42e1c78cabef0f1b4e257a5b98d9e76a63396bc76826"} Oct 30 03:44:16.572691 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:16.572489 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" event={"ID":"11758052-3bf8-4f77-83f0-3e097c6fc90c","Type":"ContainerStarted","Data":"a5e0266f2a665d664bb78879ac1d2e61a4f882c355b7ebef8d6e6d04e131c3e9"} Oct 30 03:44:16.572691 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:16.572506 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" Oct 30 03:44:16.573484 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:16.573458 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" podUID="11758052-3bf8-4f77-83f0-3e097c6fc90c" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/readyz\": dial tcp 10.134.0.50:8080: connect: connection refused" Oct 30 03:44:16.613593 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:16.613546 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" podStartSLOduration=1.6135337440000002 podStartE2EDuration="1.613533744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:16.613028747 +0000 UTC m=+3768.727343720" watchObservedRunningTime="2026-10-30 03:44:16.613533744 +0000 UTC m=+3768.727848719" Oct 30 03:44:17.579483 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:17.579452 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" Oct 30 03:44:17.657884 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:17.657846 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj"] Oct 30 03:44:17.658048 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:17.658024 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" podUID="6aa3b896-b2f3-4e8a-b733-12572a0d88d4" containerName="ta-container" containerID="cri-o://e9d04a66f80e63b2a43783694b56c1e4f7f4e9fbaa9ebfc2e4bd8a17fb5613f5" gracePeriod=30 Oct 30 03:44:17.910445 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:17.910419 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" Oct 30 03:44:18.001860 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:18.001828 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-ta-internal\" (UniqueName: \"kubernetes.io/configmap/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-ta-internal\") pod \"6aa3b896-b2f3-4e8a-b733-12572a0d88d4\" (UID: \"6aa3b896-b2f3-4e8a-b733-12572a0d88d4\") " Oct 30 03:44:18.002023 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:18.001875 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-kube-api-access-ts48v\" (UniqueName: \"kubernetes.io/projected/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-kube-api-access-ts48v\") pod \"6aa3b896-b2f3-4e8a-b733-12572a0d88d4\" (UID: \"6aa3b896-b2f3-4e8a-b733-12572a0d88d4\") " Oct 30 03:44:18.002023 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:18.001904 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-data-science-collector-ta-server-cert\") pod \"6aa3b896-b2f3-4e8a-b733-12572a0d88d4\" (UID: \"6aa3b896-b2f3-4e8a-b733-12572a0d88d4\") " Oct 30 03:44:18.002235 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:18.002209 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-ta-internal" pod "6aa3b896-b2f3-4e8a-b733-12572a0d88d4" (UID: "6aa3b896-b2f3-4e8a-b733-12572a0d88d4"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:18.003947 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:18.003915 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-kube-api-access-ts48v" pod "6aa3b896-b2f3-4e8a-b733-12572a0d88d4" (UID: "6aa3b896-b2f3-4e8a-b733-12572a0d88d4"). InnerVolumeSpecName "kube-api-access-ts48v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:18.003947 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:18.003933 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-data-science-collector-ta-server-cert" pod "6aa3b896-b2f3-4e8a-b733-12572a0d88d4" (UID: "6aa3b896-b2f3-4e8a-b733-12572a0d88d4"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:18.103042 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:18.103007 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:18.103042 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:18.103037 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:18.103042 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:18.103047 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts48v\" (UniqueName: \"kubernetes.io/projected/6aa3b896-b2f3-4e8a-b733-12572a0d88d4-kube-api-access-ts48v\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:18.583936 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:18.583905 5882 generic.go:356] "Generic (PLEG): container finished" podID="6aa3b896-b2f3-4e8a-b733-12572a0d88d4" containerID="e9d04a66f80e63b2a43783694b56c1e4f7f4e9fbaa9ebfc2e4bd8a17fb5613f5" exitCode=0 Oct 30 03:44:18.584375 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:18.583947 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" Oct 30 03:44:18.584375 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:18.583983 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" event={"ID":"6aa3b896-b2f3-4e8a-b733-12572a0d88d4","Type":"ContainerDied","Data":"e9d04a66f80e63b2a43783694b56c1e4f7f4e9fbaa9ebfc2e4bd8a17fb5613f5"} Oct 30 03:44:18.584375 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:18.584023 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj" event={"ID":"6aa3b896-b2f3-4e8a-b733-12572a0d88d4","Type":"ContainerDied","Data":"9ac0ea8a43495e3e584ff0a0da5040ccf4a10dea1289a944c7c96b35084ce210"} Oct 30 03:44:18.584375 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:18.584040 5882 scope.go:122] "RemoveContainer" containerID="e9d04a66f80e63b2a43783694b56c1e4f7f4e9fbaa9ebfc2e4bd8a17fb5613f5" Oct 30 03:44:18.592271 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:18.592253 5882 scope.go:122] "RemoveContainer" containerID="e9d04a66f80e63b2a43783694b56c1e4f7f4e9fbaa9ebfc2e4bd8a17fb5613f5" Oct 30 03:44:18.592542 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:44:18.592526 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9d04a66f80e63b2a43783694b56c1e4f7f4e9fbaa9ebfc2e4bd8a17fb5613f5\": container with ID starting with e9d04a66f80e63b2a43783694b56c1e4f7f4e9fbaa9ebfc2e4bd8a17fb5613f5 not found: ID does not exist" containerID="e9d04a66f80e63b2a43783694b56c1e4f7f4e9fbaa9ebfc2e4bd8a17fb5613f5" Oct 30 03:44:18.592591 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:18.592549 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9d04a66f80e63b2a43783694b56c1e4f7f4e9fbaa9ebfc2e4bd8a17fb5613f5"} err="failed to get container status \"e9d04a66f80e63b2a43783694b56c1e4f7f4e9fbaa9ebfc2e4bd8a17fb5613f5\": rpc error: code = NotFound desc = could not find container \"e9d04a66f80e63b2a43783694b56c1e4f7f4e9fbaa9ebfc2e4bd8a17fb5613f5\": container with ID starting with e9d04a66f80e63b2a43783694b56c1e4f7f4e9fbaa9ebfc2e4bd8a17fb5613f5 not found: ID does not exist" Oct 30 03:44:18.633573 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:18.633547 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj"] Oct 30 03:44:18.645344 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:18.645321 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-76649584c9-t4kkj"] Oct 30 03:44:20.009988 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:20.009955 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa3b896-b2f3-4e8a-b733-12572a0d88d4" path="/var/lib/kubelet/pods/6aa3b896-b2f3-4e8a-b733-12572a0d88d4/volumes" Oct 30 03:44:23.942871 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:44:23.942837 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:44:25.295799 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:25.295762 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2"] Oct 30 03:44:25.296258 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:25.296241 5882 cpu_manager.go:414] "containerMap: removing container" podUID="6aa3b896-b2f3-4e8a-b733-12572a0d88d4" containerName="ta-container" Oct 30 03:44:25.296356 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:25.296259 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa3b896-b2f3-4e8a-b733-12572a0d88d4" containerName="ta-container" Oct 30 03:44:25.296356 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:25.296351 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa3b896-b2f3-4e8a-b733-12572a0d88d4" containerName="ta-container" Oct 30 03:44:25.300882 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:25.300862 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" Oct 30 03:44:25.330976 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:25.330944 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2"] Oct 30 03:44:25.465790 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:25.465746 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9z9n\" (UniqueName: \"kubernetes.io/projected/f1506dbc-39bf-4817-ad08-9d308920092c-kube-api-access-p9z9n\") pod \"data-science-collector-targetallocator-8dcc98df-kkjf2\" (UID: \"f1506dbc-39bf-4817-ad08-9d308920092c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" Oct 30 03:44:25.465790 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:25.465790 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f1506dbc-39bf-4817-ad08-9d308920092c-ta-internal\") pod \"data-science-collector-targetallocator-8dcc98df-kkjf2\" (UID: \"f1506dbc-39bf-4817-ad08-9d308920092c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" Oct 30 03:44:25.465997 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:25.465884 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f1506dbc-39bf-4817-ad08-9d308920092c-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-8dcc98df-kkjf2\" (UID: \"f1506dbc-39bf-4817-ad08-9d308920092c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" Oct 30 03:44:25.566853 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:25.566775 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9z9n\" (UniqueName: \"kubernetes.io/projected/f1506dbc-39bf-4817-ad08-9d308920092c-kube-api-access-p9z9n\") pod \"data-science-collector-targetallocator-8dcc98df-kkjf2\" (UID: \"f1506dbc-39bf-4817-ad08-9d308920092c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" Oct 30 03:44:25.566853 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:25.566813 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f1506dbc-39bf-4817-ad08-9d308920092c-ta-internal\") pod \"data-science-collector-targetallocator-8dcc98df-kkjf2\" (UID: \"f1506dbc-39bf-4817-ad08-9d308920092c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" Oct 30 03:44:25.566853 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:25.566845 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f1506dbc-39bf-4817-ad08-9d308920092c-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-8dcc98df-kkjf2\" (UID: \"f1506dbc-39bf-4817-ad08-9d308920092c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" Oct 30 03:44:25.567433 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:25.567403 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f1506dbc-39bf-4817-ad08-9d308920092c-ta-internal\") pod \"data-science-collector-targetallocator-8dcc98df-kkjf2\" (UID: \"f1506dbc-39bf-4817-ad08-9d308920092c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" Oct 30 03:44:25.569377 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:25.569354 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f1506dbc-39bf-4817-ad08-9d308920092c-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-8dcc98df-kkjf2\" (UID: \"f1506dbc-39bf-4817-ad08-9d308920092c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" Oct 30 03:44:25.589915 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:25.589885 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9z9n\" (UniqueName: \"kubernetes.io/projected/f1506dbc-39bf-4817-ad08-9d308920092c-kube-api-access-p9z9n\") pod \"data-science-collector-targetallocator-8dcc98df-kkjf2\" (UID: \"f1506dbc-39bf-4817-ad08-9d308920092c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" Oct 30 03:44:25.612632 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:25.612592 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" Oct 30 03:44:25.903009 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:25.902963 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2"] Oct 30 03:44:25.905887 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:44:25.905858 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1506dbc_39bf_4817_ad08_9d308920092c.slice/crio-4d78f395d9f0fa648166eae7c5997cbdbcdb34f3f9ad99097dc156e131308caf WatchSource:0}: Error finding container 4d78f395d9f0fa648166eae7c5997cbdbcdb34f3f9ad99097dc156e131308caf: Status 404 returned error can't find the container with id 4d78f395d9f0fa648166eae7c5997cbdbcdb34f3f9ad99097dc156e131308caf Oct 30 03:44:25.908170 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:25.908139 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:25.908257 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:25.908222 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:26.624926 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:26.624892 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" event={"ID":"f1506dbc-39bf-4817-ad08-9d308920092c","Type":"ContainerStarted","Data":"f6748322d593497349e1b5f82d8a61db683d0a908a95b8c29b43edfa957dcde2"} Oct 30 03:44:26.625346 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:26.624933 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" event={"ID":"f1506dbc-39bf-4817-ad08-9d308920092c","Type":"ContainerStarted","Data":"4d78f395d9f0fa648166eae7c5997cbdbcdb34f3f9ad99097dc156e131308caf"} Oct 30 03:44:26.625346 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:26.625070 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" Oct 30 03:44:26.626960 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:26.626940 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" Oct 30 03:44:26.680067 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:26.679975 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" podStartSLOduration=1.679962191 podStartE2EDuration="1.679962191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:26.678523557 +0000 UTC m=+3778.792838535" watchObservedRunningTime="2026-10-30 03:44:26.679962191 +0000 UTC m=+3778.794277162" Oct 30 03:44:26.760764 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:26.760721 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc"] Oct 30 03:44:26.760943 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:26.760920 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" podUID="11758052-3bf8-4f77-83f0-3e097c6fc90c" containerName="ta-container" containerID="cri-o://b2bd281471bd1dad348b42e1c78cabef0f1b4e257a5b98d9e76a63396bc76826" gracePeriod=30 Oct 30 03:44:27.008305 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:27.008276 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" Oct 30 03:44:27.081439 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:27.081400 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11758052-3bf8-4f77-83f0-3e097c6fc90c-kube-api-access-x76z2\" (UniqueName: \"kubernetes.io/projected/11758052-3bf8-4f77-83f0-3e097c6fc90c-kube-api-access-x76z2\") pod \"11758052-3bf8-4f77-83f0-3e097c6fc90c\" (UID: \"11758052-3bf8-4f77-83f0-3e097c6fc90c\") " Oct 30 03:44:27.081597 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:27.081489 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11758052-3bf8-4f77-83f0-3e097c6fc90c-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/11758052-3bf8-4f77-83f0-3e097c6fc90c-data-science-collector-ta-server-cert\") pod \"11758052-3bf8-4f77-83f0-3e097c6fc90c\" (UID: \"11758052-3bf8-4f77-83f0-3e097c6fc90c\") " Oct 30 03:44:27.081597 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:27.081530 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11758052-3bf8-4f77-83f0-3e097c6fc90c-ta-internal\" (UniqueName: \"kubernetes.io/configmap/11758052-3bf8-4f77-83f0-3e097c6fc90c-ta-internal\") pod \"11758052-3bf8-4f77-83f0-3e097c6fc90c\" (UID: \"11758052-3bf8-4f77-83f0-3e097c6fc90c\") " Oct 30 03:44:27.081955 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:27.081936 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11758052-3bf8-4f77-83f0-3e097c6fc90c-ta-internal" pod "11758052-3bf8-4f77-83f0-3e097c6fc90c" (UID: "11758052-3bf8-4f77-83f0-3e097c6fc90c"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:27.083561 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:27.083538 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11758052-3bf8-4f77-83f0-3e097c6fc90c-data-science-collector-ta-server-cert" pod "11758052-3bf8-4f77-83f0-3e097c6fc90c" (UID: "11758052-3bf8-4f77-83f0-3e097c6fc90c"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:27.083670 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:27.083559 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11758052-3bf8-4f77-83f0-3e097c6fc90c-kube-api-access-x76z2" pod "11758052-3bf8-4f77-83f0-3e097c6fc90c" (UID: "11758052-3bf8-4f77-83f0-3e097c6fc90c"). InnerVolumeSpecName "kube-api-access-x76z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:27.182339 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:27.182253 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x76z2\" (UniqueName: \"kubernetes.io/projected/11758052-3bf8-4f77-83f0-3e097c6fc90c-kube-api-access-x76z2\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:27.182339 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:27.182293 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/11758052-3bf8-4f77-83f0-3e097c6fc90c-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:27.182339 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:27.182306 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/11758052-3bf8-4f77-83f0-3e097c6fc90c-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:27.635996 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:27.635957 5882 generic.go:356] "Generic (PLEG): container finished" podID="11758052-3bf8-4f77-83f0-3e097c6fc90c" containerID="b2bd281471bd1dad348b42e1c78cabef0f1b4e257a5b98d9e76a63396bc76826" exitCode=0 Oct 30 03:44:27.636423 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:27.636013 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" Oct 30 03:44:27.636423 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:27.636039 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" event={"ID":"11758052-3bf8-4f77-83f0-3e097c6fc90c","Type":"ContainerDied","Data":"b2bd281471bd1dad348b42e1c78cabef0f1b4e257a5b98d9e76a63396bc76826"} Oct 30 03:44:27.636423 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:27.636085 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc" event={"ID":"11758052-3bf8-4f77-83f0-3e097c6fc90c","Type":"ContainerDied","Data":"a5e0266f2a665d664bb78879ac1d2e61a4f882c355b7ebef8d6e6d04e131c3e9"} Oct 30 03:44:27.636423 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:27.636106 5882 scope.go:122] "RemoveContainer" containerID="b2bd281471bd1dad348b42e1c78cabef0f1b4e257a5b98d9e76a63396bc76826" Oct 30 03:44:27.644628 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:27.644596 5882 scope.go:122] "RemoveContainer" containerID="b2bd281471bd1dad348b42e1c78cabef0f1b4e257a5b98d9e76a63396bc76826" Oct 30 03:44:27.644941 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:44:27.644918 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2bd281471bd1dad348b42e1c78cabef0f1b4e257a5b98d9e76a63396bc76826\": container with ID starting with b2bd281471bd1dad348b42e1c78cabef0f1b4e257a5b98d9e76a63396bc76826 not found: ID does not exist" containerID="b2bd281471bd1dad348b42e1c78cabef0f1b4e257a5b98d9e76a63396bc76826" Oct 30 03:44:27.645006 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:27.644952 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2bd281471bd1dad348b42e1c78cabef0f1b4e257a5b98d9e76a63396bc76826"} err="failed to get container status \"b2bd281471bd1dad348b42e1c78cabef0f1b4e257a5b98d9e76a63396bc76826\": rpc error: code = NotFound desc = could not find container \"b2bd281471bd1dad348b42e1c78cabef0f1b4e257a5b98d9e76a63396bc76826\": container with ID starting with b2bd281471bd1dad348b42e1c78cabef0f1b4e257a5b98d9e76a63396bc76826 not found: ID does not exist" Oct 30 03:44:27.684722 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:27.684695 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc"] Oct 30 03:44:27.695979 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:27.695951 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-64c756b9d-dgqxc"] Oct 30 03:44:28.009899 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:28.009867 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11758052-3bf8-4f77-83f0-3e097c6fc90c" path="/var/lib/kubelet/pods/11758052-3bf8-4f77-83f0-3e097c6fc90c/volumes" Oct 30 03:44:35.011678 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:35.011646 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc"] Oct 30 03:44:35.012092 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:35.012005 5882 cpu_manager.go:414] "containerMap: removing container" podUID="11758052-3bf8-4f77-83f0-3e097c6fc90c" containerName="ta-container" Oct 30 03:44:35.012092 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:35.012019 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11758052-3bf8-4f77-83f0-3e097c6fc90c" containerName="ta-container" Oct 30 03:44:35.012194 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:35.012145 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="11758052-3bf8-4f77-83f0-3e097c6fc90c" containerName="ta-container" Oct 30 03:44:35.015203 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:35.015184 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" Oct 30 03:44:35.034265 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:35.034241 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc"] Oct 30 03:44:35.043566 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:35.043540 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7f59b48d-1c75-4695-a135-966d3e65fb2a-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7749cf65b4-n8scc\" (UID: \"7f59b48d-1c75-4695-a135-966d3e65fb2a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" Oct 30 03:44:35.043731 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:35.043598 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7f59b48d-1c75-4695-a135-966d3e65fb2a-ta-internal\") pod \"data-science-collector-targetallocator-7749cf65b4-n8scc\" (UID: \"7f59b48d-1c75-4695-a135-966d3e65fb2a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" Oct 30 03:44:35.043731 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:35.043675 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qmq4\" (UniqueName: \"kubernetes.io/projected/7f59b48d-1c75-4695-a135-966d3e65fb2a-kube-api-access-8qmq4\") pod \"data-science-collector-targetallocator-7749cf65b4-n8scc\" (UID: \"7f59b48d-1c75-4695-a135-966d3e65fb2a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" Oct 30 03:44:35.144600 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:35.144559 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7f59b48d-1c75-4695-a135-966d3e65fb2a-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7749cf65b4-n8scc\" (UID: \"7f59b48d-1c75-4695-a135-966d3e65fb2a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" Oct 30 03:44:35.144814 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:35.144650 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7f59b48d-1c75-4695-a135-966d3e65fb2a-ta-internal\") pod \"data-science-collector-targetallocator-7749cf65b4-n8scc\" (UID: \"7f59b48d-1c75-4695-a135-966d3e65fb2a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" Oct 30 03:44:35.144814 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:35.144669 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qmq4\" (UniqueName: \"kubernetes.io/projected/7f59b48d-1c75-4695-a135-966d3e65fb2a-kube-api-access-8qmq4\") pod \"data-science-collector-targetallocator-7749cf65b4-n8scc\" (UID: \"7f59b48d-1c75-4695-a135-966d3e65fb2a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" Oct 30 03:44:35.145332 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:35.145309 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7f59b48d-1c75-4695-a135-966d3e65fb2a-ta-internal\") pod \"data-science-collector-targetallocator-7749cf65b4-n8scc\" (UID: \"7f59b48d-1c75-4695-a135-966d3e65fb2a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" Oct 30 03:44:35.147075 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:35.147048 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7f59b48d-1c75-4695-a135-966d3e65fb2a-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7749cf65b4-n8scc\" (UID: \"7f59b48d-1c75-4695-a135-966d3e65fb2a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" Oct 30 03:44:35.171288 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:35.171255 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qmq4\" (UniqueName: \"kubernetes.io/projected/7f59b48d-1c75-4695-a135-966d3e65fb2a-kube-api-access-8qmq4\") pod \"data-science-collector-targetallocator-7749cf65b4-n8scc\" (UID: \"7f59b48d-1c75-4695-a135-966d3e65fb2a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" Oct 30 03:44:35.326701 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:35.326587 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" Oct 30 03:44:35.621585 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:35.621511 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc"] Oct 30 03:44:35.623980 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:44:35.623951 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f59b48d_1c75_4695_a135_966d3e65fb2a.slice/crio-a9b193156b5bd93f046f86c1c2137155f232e4c3db7ac832551a3b24b21f1b9e WatchSource:0}: Error finding container a9b193156b5bd93f046f86c1c2137155f232e4c3db7ac832551a3b24b21f1b9e: Status 404 returned error can't find the container with id a9b193156b5bd93f046f86c1c2137155f232e4c3db7ac832551a3b24b21f1b9e Oct 30 03:44:35.626114 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:35.626079 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:35.626206 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:35.626140 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:35.675093 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:35.675067 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" event={"ID":"7f59b48d-1c75-4695-a135-966d3e65fb2a","Type":"ContainerStarted","Data":"a9b193156b5bd93f046f86c1c2137155f232e4c3db7ac832551a3b24b21f1b9e"} Oct 30 03:44:36.681079 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:36.681047 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" event={"ID":"7f59b48d-1c75-4695-a135-966d3e65fb2a","Type":"ContainerStarted","Data":"017e0471b8ae075650c4691a834174bc67880316e6fb107a27ea5877e5cb809d"} Oct 30 03:44:36.681539 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:36.681089 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" Oct 30 03:44:36.682814 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:36.682789 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" Oct 30 03:44:36.730750 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:36.730694 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" podStartSLOduration=2.730678244 podStartE2EDuration="2.730678244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:36.729308167 +0000 UTC m=+3788.843623153" watchObservedRunningTime="2026-10-30 03:44:36.730678244 +0000 UTC m=+3788.844993224" Oct 30 03:44:36.835874 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:36.835836 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2"] Oct 30 03:44:36.836090 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:36.836065 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" podUID="f1506dbc-39bf-4817-ad08-9d308920092c" containerName="ta-container" containerID="cri-o://f6748322d593497349e1b5f82d8a61db683d0a908a95b8c29b43edfa957dcde2" gracePeriod=30 Oct 30 03:44:37.080746 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:37.080721 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" Oct 30 03:44:37.162429 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:37.162400 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1506dbc-39bf-4817-ad08-9d308920092c-kube-api-access-p9z9n\" (UniqueName: \"kubernetes.io/projected/f1506dbc-39bf-4817-ad08-9d308920092c-kube-api-access-p9z9n\") pod \"f1506dbc-39bf-4817-ad08-9d308920092c\" (UID: \"f1506dbc-39bf-4817-ad08-9d308920092c\") " Oct 30 03:44:37.162590 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:37.162466 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1506dbc-39bf-4817-ad08-9d308920092c-ta-internal\" (UniqueName: \"kubernetes.io/configmap/f1506dbc-39bf-4817-ad08-9d308920092c-ta-internal\") pod \"f1506dbc-39bf-4817-ad08-9d308920092c\" (UID: \"f1506dbc-39bf-4817-ad08-9d308920092c\") " Oct 30 03:44:37.162590 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:37.162485 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1506dbc-39bf-4817-ad08-9d308920092c-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f1506dbc-39bf-4817-ad08-9d308920092c-data-science-collector-ta-server-cert\") pod \"f1506dbc-39bf-4817-ad08-9d308920092c\" (UID: \"f1506dbc-39bf-4817-ad08-9d308920092c\") " Oct 30 03:44:37.162854 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:37.162829 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1506dbc-39bf-4817-ad08-9d308920092c-ta-internal" pod "f1506dbc-39bf-4817-ad08-9d308920092c" (UID: "f1506dbc-39bf-4817-ad08-9d308920092c"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:37.164507 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:37.164482 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1506dbc-39bf-4817-ad08-9d308920092c-kube-api-access-p9z9n" pod "f1506dbc-39bf-4817-ad08-9d308920092c" (UID: "f1506dbc-39bf-4817-ad08-9d308920092c"). InnerVolumeSpecName "kube-api-access-p9z9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:37.164598 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:37.164535 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1506dbc-39bf-4817-ad08-9d308920092c-data-science-collector-ta-server-cert" pod "f1506dbc-39bf-4817-ad08-9d308920092c" (UID: "f1506dbc-39bf-4817-ad08-9d308920092c"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:37.263558 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:37.263526 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f1506dbc-39bf-4817-ad08-9d308920092c-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:37.263558 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:37.263556 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f1506dbc-39bf-4817-ad08-9d308920092c-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:37.263765 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:37.263566 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9z9n\" (UniqueName: \"kubernetes.io/projected/f1506dbc-39bf-4817-ad08-9d308920092c-kube-api-access-p9z9n\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:37.686954 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:37.686860 5882 generic.go:356] "Generic (PLEG): container finished" podID="f1506dbc-39bf-4817-ad08-9d308920092c" containerID="f6748322d593497349e1b5f82d8a61db683d0a908a95b8c29b43edfa957dcde2" exitCode=0 Oct 30 03:44:37.686954 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:37.686912 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" Oct 30 03:44:37.687470 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:37.686946 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" event={"ID":"f1506dbc-39bf-4817-ad08-9d308920092c","Type":"ContainerDied","Data":"f6748322d593497349e1b5f82d8a61db683d0a908a95b8c29b43edfa957dcde2"} Oct 30 03:44:37.687470 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:37.686986 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2" event={"ID":"f1506dbc-39bf-4817-ad08-9d308920092c","Type":"ContainerDied","Data":"4d78f395d9f0fa648166eae7c5997cbdbcdb34f3f9ad99097dc156e131308caf"} Oct 30 03:44:37.687470 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:37.687007 5882 scope.go:122] "RemoveContainer" containerID="f6748322d593497349e1b5f82d8a61db683d0a908a95b8c29b43edfa957dcde2" Oct 30 03:44:37.695823 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:37.695806 5882 scope.go:122] "RemoveContainer" containerID="f6748322d593497349e1b5f82d8a61db683d0a908a95b8c29b43edfa957dcde2" Oct 30 03:44:37.696090 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:44:37.696075 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6748322d593497349e1b5f82d8a61db683d0a908a95b8c29b43edfa957dcde2\": container with ID starting with f6748322d593497349e1b5f82d8a61db683d0a908a95b8c29b43edfa957dcde2 not found: ID does not exist" containerID="f6748322d593497349e1b5f82d8a61db683d0a908a95b8c29b43edfa957dcde2" Oct 30 03:44:37.696136 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:37.696097 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6748322d593497349e1b5f82d8a61db683d0a908a95b8c29b43edfa957dcde2"} err="failed to get container status \"f6748322d593497349e1b5f82d8a61db683d0a908a95b8c29b43edfa957dcde2\": rpc error: code = NotFound desc = could not find container \"f6748322d593497349e1b5f82d8a61db683d0a908a95b8c29b43edfa957dcde2\": container with ID starting with f6748322d593497349e1b5f82d8a61db683d0a908a95b8c29b43edfa957dcde2 not found: ID does not exist" Oct 30 03:44:37.736570 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:37.736546 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2"] Oct 30 03:44:37.742933 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:37.742916 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-8dcc98df-kkjf2"] Oct 30 03:44:38.008896 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:38.008865 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1506dbc-39bf-4817-ad08-9d308920092c" path="/var/lib/kubelet/pods/f1506dbc-39bf-4817-ad08-9d308920092c/volumes" Oct 30 03:44:44.180481 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:44.180400 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr"] Oct 30 03:44:44.180919 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:44.180903 5882 cpu_manager.go:414] "containerMap: removing container" podUID="f1506dbc-39bf-4817-ad08-9d308920092c" containerName="ta-container" Oct 30 03:44:44.180963 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:44.180922 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1506dbc-39bf-4817-ad08-9d308920092c" containerName="ta-container" Oct 30 03:44:44.181046 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:44.181036 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1506dbc-39bf-4817-ad08-9d308920092c" containerName="ta-container" Oct 30 03:44:44.185022 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:44.185003 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" Oct 30 03:44:44.211348 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:44.211319 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr"] Oct 30 03:44:44.225824 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:44.225793 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/fdad6c63-d420-4cec-b238-743b712c86e7-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6dfbcdf4b7-l95nr\" (UID: \"fdad6c63-d420-4cec-b238-743b712c86e7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" Oct 30 03:44:44.225998 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:44.225861 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/fdad6c63-d420-4cec-b238-743b712c86e7-ta-internal\") pod \"data-science-collector-targetallocator-6dfbcdf4b7-l95nr\" (UID: \"fdad6c63-d420-4cec-b238-743b712c86e7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" Oct 30 03:44:44.225998 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:44.225879 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zvb8\" (UniqueName: \"kubernetes.io/projected/fdad6c63-d420-4cec-b238-743b712c86e7-kube-api-access-8zvb8\") pod \"data-science-collector-targetallocator-6dfbcdf4b7-l95nr\" (UID: \"fdad6c63-d420-4cec-b238-743b712c86e7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" Oct 30 03:44:44.326403 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:44.326354 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/fdad6c63-d420-4cec-b238-743b712c86e7-ta-internal\") pod \"data-science-collector-targetallocator-6dfbcdf4b7-l95nr\" (UID: \"fdad6c63-d420-4cec-b238-743b712c86e7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" Oct 30 03:44:44.326598 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:44.326404 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zvb8\" (UniqueName: \"kubernetes.io/projected/fdad6c63-d420-4cec-b238-743b712c86e7-kube-api-access-8zvb8\") pod \"data-science-collector-targetallocator-6dfbcdf4b7-l95nr\" (UID: \"fdad6c63-d420-4cec-b238-743b712c86e7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" Oct 30 03:44:44.326598 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:44.326524 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/fdad6c63-d420-4cec-b238-743b712c86e7-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6dfbcdf4b7-l95nr\" (UID: \"fdad6c63-d420-4cec-b238-743b712c86e7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" Oct 30 03:44:44.327093 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:44.327065 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/fdad6c63-d420-4cec-b238-743b712c86e7-ta-internal\") pod \"data-science-collector-targetallocator-6dfbcdf4b7-l95nr\" (UID: \"fdad6c63-d420-4cec-b238-743b712c86e7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" Oct 30 03:44:44.329023 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:44.329004 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/fdad6c63-d420-4cec-b238-743b712c86e7-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6dfbcdf4b7-l95nr\" (UID: \"fdad6c63-d420-4cec-b238-743b712c86e7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" Oct 30 03:44:44.352762 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:44.352730 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zvb8\" (UniqueName: \"kubernetes.io/projected/fdad6c63-d420-4cec-b238-743b712c86e7-kube-api-access-8zvb8\") pod \"data-science-collector-targetallocator-6dfbcdf4b7-l95nr\" (UID: \"fdad6c63-d420-4cec-b238-743b712c86e7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" Oct 30 03:44:44.496652 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:44.496599 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" Oct 30 03:44:44.785458 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:44.785431 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr"] Oct 30 03:44:44.788366 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:44:44.788338 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdad6c63_d420_4cec_b238_743b712c86e7.slice/crio-05f43e87841b3b1cc43d86adc4d77a4fd980edfad16798a09eaeb34fe59c1234 WatchSource:0}: Error finding container 05f43e87841b3b1cc43d86adc4d77a4fd980edfad16798a09eaeb34fe59c1234: Status 404 returned error can't find the container with id 05f43e87841b3b1cc43d86adc4d77a4fd980edfad16798a09eaeb34fe59c1234 Oct 30 03:44:44.790438 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:44.790404 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:44.790527 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:44.790461 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:45.724501 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:45.724456 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" event={"ID":"fdad6c63-d420-4cec-b238-743b712c86e7","Type":"ContainerStarted","Data":"8cdee7e945ad80120f40e4d4eebf94d4431c9301f6dcfa32b2c5fb266800bf0d"} Oct 30 03:44:45.724961 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:45.724510 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" event={"ID":"fdad6c63-d420-4cec-b238-743b712c86e7","Type":"ContainerStarted","Data":"05f43e87841b3b1cc43d86adc4d77a4fd980edfad16798a09eaeb34fe59c1234"} Oct 30 03:44:45.724961 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:45.724648 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" Oct 30 03:44:45.726534 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:45.726515 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" Oct 30 03:44:45.768393 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:45.768339 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" podStartSLOduration=1.76832193 podStartE2EDuration="1.76832193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:45.766643918 +0000 UTC m=+3797.880958904" watchObservedRunningTime="2026-10-30 03:44:45.76832193 +0000 UTC m=+3797.882636902" Oct 30 03:44:45.846443 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:45.846409 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc"] Oct 30 03:44:45.846702 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:45.846675 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" podUID="7f59b48d-1c75-4695-a135-966d3e65fb2a" containerName="ta-container" containerID="cri-o://017e0471b8ae075650c4691a834174bc67880316e6fb107a27ea5877e5cb809d" gracePeriod=30 Oct 30 03:44:46.091042 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:46.091019 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" Oct 30 03:44:46.142413 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:46.142377 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f59b48d-1c75-4695-a135-966d3e65fb2a-ta-internal\" (UniqueName: \"kubernetes.io/configmap/7f59b48d-1c75-4695-a135-966d3e65fb2a-ta-internal\") pod \"7f59b48d-1c75-4695-a135-966d3e65fb2a\" (UID: \"7f59b48d-1c75-4695-a135-966d3e65fb2a\") " Oct 30 03:44:46.142563 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:46.142496 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f59b48d-1c75-4695-a135-966d3e65fb2a-kube-api-access-8qmq4\" (UniqueName: \"kubernetes.io/projected/7f59b48d-1c75-4695-a135-966d3e65fb2a-kube-api-access-8qmq4\") pod \"7f59b48d-1c75-4695-a135-966d3e65fb2a\" (UID: \"7f59b48d-1c75-4695-a135-966d3e65fb2a\") " Oct 30 03:44:46.142664 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:46.142593 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f59b48d-1c75-4695-a135-966d3e65fb2a-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7f59b48d-1c75-4695-a135-966d3e65fb2a-data-science-collector-ta-server-cert\") pod \"7f59b48d-1c75-4695-a135-966d3e65fb2a\" (UID: \"7f59b48d-1c75-4695-a135-966d3e65fb2a\") " Oct 30 03:44:46.142837 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:46.142809 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f59b48d-1c75-4695-a135-966d3e65fb2a-ta-internal" pod "7f59b48d-1c75-4695-a135-966d3e65fb2a" (UID: "7f59b48d-1c75-4695-a135-966d3e65fb2a"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:46.144641 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:46.144593 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f59b48d-1c75-4695-a135-966d3e65fb2a-kube-api-access-8qmq4" pod "7f59b48d-1c75-4695-a135-966d3e65fb2a" (UID: "7f59b48d-1c75-4695-a135-966d3e65fb2a"). InnerVolumeSpecName "kube-api-access-8qmq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:46.144780 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:46.144763 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f59b48d-1c75-4695-a135-966d3e65fb2a-data-science-collector-ta-server-cert" pod "7f59b48d-1c75-4695-a135-966d3e65fb2a" (UID: "7f59b48d-1c75-4695-a135-966d3e65fb2a"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:46.243305 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:46.243266 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qmq4\" (UniqueName: \"kubernetes.io/projected/7f59b48d-1c75-4695-a135-966d3e65fb2a-kube-api-access-8qmq4\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:46.243305 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:46.243298 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7f59b48d-1c75-4695-a135-966d3e65fb2a-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:46.243305 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:46.243311 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7f59b48d-1c75-4695-a135-966d3e65fb2a-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:46.730079 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:46.730038 5882 generic.go:356] "Generic (PLEG): container finished" podID="7f59b48d-1c75-4695-a135-966d3e65fb2a" containerID="017e0471b8ae075650c4691a834174bc67880316e6fb107a27ea5877e5cb809d" exitCode=0 Oct 30 03:44:46.730511 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:46.730091 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" Oct 30 03:44:46.730511 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:46.730093 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" event={"ID":"7f59b48d-1c75-4695-a135-966d3e65fb2a","Type":"ContainerDied","Data":"017e0471b8ae075650c4691a834174bc67880316e6fb107a27ea5877e5cb809d"} Oct 30 03:44:46.730511 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:46.730137 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc" event={"ID":"7f59b48d-1c75-4695-a135-966d3e65fb2a","Type":"ContainerDied","Data":"a9b193156b5bd93f046f86c1c2137155f232e4c3db7ac832551a3b24b21f1b9e"} Oct 30 03:44:46.730511 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:46.730156 5882 scope.go:122] "RemoveContainer" containerID="017e0471b8ae075650c4691a834174bc67880316e6fb107a27ea5877e5cb809d" Oct 30 03:44:46.739370 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:46.739352 5882 scope.go:122] "RemoveContainer" containerID="017e0471b8ae075650c4691a834174bc67880316e6fb107a27ea5877e5cb809d" Oct 30 03:44:46.739643 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:44:46.739600 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"017e0471b8ae075650c4691a834174bc67880316e6fb107a27ea5877e5cb809d\": container with ID starting with 017e0471b8ae075650c4691a834174bc67880316e6fb107a27ea5877e5cb809d not found: ID does not exist" containerID="017e0471b8ae075650c4691a834174bc67880316e6fb107a27ea5877e5cb809d" Oct 30 03:44:46.739713 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:46.739649 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"017e0471b8ae075650c4691a834174bc67880316e6fb107a27ea5877e5cb809d"} err="failed to get container status \"017e0471b8ae075650c4691a834174bc67880316e6fb107a27ea5877e5cb809d\": rpc error: code = NotFound desc = could not find container \"017e0471b8ae075650c4691a834174bc67880316e6fb107a27ea5877e5cb809d\": container with ID starting with 017e0471b8ae075650c4691a834174bc67880316e6fb107a27ea5877e5cb809d not found: ID does not exist" Oct 30 03:44:46.780044 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:46.780013 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc"] Oct 30 03:44:46.791347 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:46.791320 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7749cf65b4-n8scc"] Oct 30 03:44:48.009280 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:48.009243 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f59b48d-1c75-4695-a135-966d3e65fb2a" path="/var/lib/kubelet/pods/7f59b48d-1c75-4695-a135-966d3e65fb2a/volumes" Oct 30 03:44:52.770045 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:52.770009 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l"] Oct 30 03:44:52.770692 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:52.770564 5882 cpu_manager.go:414] "containerMap: removing container" podUID="7f59b48d-1c75-4695-a135-966d3e65fb2a" containerName="ta-container" Oct 30 03:44:52.770692 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:52.770584 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f59b48d-1c75-4695-a135-966d3e65fb2a" containerName="ta-container" Oct 30 03:44:52.770823 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:52.770707 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f59b48d-1c75-4695-a135-966d3e65fb2a" containerName="ta-container" Oct 30 03:44:52.774735 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:52.774714 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" Oct 30 03:44:52.804437 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:52.804409 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l"] Oct 30 03:44:52.904583 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:52.904551 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5d89dc86c4-rms2l\" (UID: \"f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" Oct 30 03:44:52.904754 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:52.904588 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txfld\" (UniqueName: \"kubernetes.io/projected/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-kube-api-access-txfld\") pod \"data-science-collector-targetallocator-5d89dc86c4-rms2l\" (UID: \"f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" Oct 30 03:44:52.904754 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:52.904692 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-ta-internal\") pod \"data-science-collector-targetallocator-5d89dc86c4-rms2l\" (UID: \"f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" Oct 30 03:44:53.005505 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:53.005469 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5d89dc86c4-rms2l\" (UID: \"f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" Oct 30 03:44:53.005680 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:53.005514 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txfld\" (UniqueName: \"kubernetes.io/projected/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-kube-api-access-txfld\") pod \"data-science-collector-targetallocator-5d89dc86c4-rms2l\" (UID: \"f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" Oct 30 03:44:53.005680 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:53.005572 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-ta-internal\") pod \"data-science-collector-targetallocator-5d89dc86c4-rms2l\" (UID: \"f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" Oct 30 03:44:53.006308 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:53.006291 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-ta-internal\") pod \"data-science-collector-targetallocator-5d89dc86c4-rms2l\" (UID: \"f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" Oct 30 03:44:53.008126 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:53.008106 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5d89dc86c4-rms2l\" (UID: \"f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" Oct 30 03:44:53.028568 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:53.028492 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txfld\" (UniqueName: \"kubernetes.io/projected/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-kube-api-access-txfld\") pod \"data-science-collector-targetallocator-5d89dc86c4-rms2l\" (UID: \"f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" Oct 30 03:44:53.086117 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:53.086081 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" Oct 30 03:44:53.375137 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:53.375108 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l"] Oct 30 03:44:53.377768 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:44:53.377739 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf57b39ef_0c0c_4d76_a2f9_5e4febc1d5c3.slice/crio-4f6493aababcf44f00efc5a9c984b49a31a6a647afcfb2f4c1ebe0e2db50d3ee WatchSource:0}: Error finding container 4f6493aababcf44f00efc5a9c984b49a31a6a647afcfb2f4c1ebe0e2db50d3ee: Status 404 returned error can't find the container with id 4f6493aababcf44f00efc5a9c984b49a31a6a647afcfb2f4c1ebe0e2db50d3ee Oct 30 03:44:53.380039 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:53.379997 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:53.380124 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:53.380079 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:53.762189 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:53.762158 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" event={"ID":"f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3","Type":"ContainerStarted","Data":"1790dadfad9d3f4f1705046963be4d403103895e6a4f530a5eacad9681225c2e"} Oct 30 03:44:53.762189 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:53.762196 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" event={"ID":"f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3","Type":"ContainerStarted","Data":"4f6493aababcf44f00efc5a9c984b49a31a6a647afcfb2f4c1ebe0e2db50d3ee"} Oct 30 03:44:53.762413 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:53.762315 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" Oct 30 03:44:53.763342 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:53.763314 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" podUID="f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.54:8080/readyz\": dial tcp 10.134.0.54:8080: connect: connection refused" Oct 30 03:44:53.801999 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:53.801930 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" podStartSLOduration=1.801913759 podStartE2EDuration="1.801913759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:53.801090825 +0000 UTC m=+3805.915405800" watchObservedRunningTime="2026-10-30 03:44:53.801913759 +0000 UTC m=+3805.916228734" Oct 30 03:44:53.943527 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:44:53.943490 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:44:54.768319 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:54.768287 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" Oct 30 03:44:54.839221 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:54.839191 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr"] Oct 30 03:44:54.839738 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:54.839351 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" podUID="fdad6c63-d420-4cec-b238-743b712c86e7" containerName="ta-container" containerID="cri-o://8cdee7e945ad80120f40e4d4eebf94d4431c9301f6dcfa32b2c5fb266800bf0d" gracePeriod=30 Oct 30 03:44:55.079699 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:55.079674 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" Oct 30 03:44:55.124901 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:55.124867 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdad6c63-d420-4cec-b238-743b712c86e7-ta-internal\" (UniqueName: \"kubernetes.io/configmap/fdad6c63-d420-4cec-b238-743b712c86e7-ta-internal\") pod \"fdad6c63-d420-4cec-b238-743b712c86e7\" (UID: \"fdad6c63-d420-4cec-b238-743b712c86e7\") " Oct 30 03:44:55.125058 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:55.124989 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdad6c63-d420-4cec-b238-743b712c86e7-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/fdad6c63-d420-4cec-b238-743b712c86e7-data-science-collector-ta-server-cert\") pod \"fdad6c63-d420-4cec-b238-743b712c86e7\" (UID: \"fdad6c63-d420-4cec-b238-743b712c86e7\") " Oct 30 03:44:55.125058 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:55.125049 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdad6c63-d420-4cec-b238-743b712c86e7-kube-api-access-8zvb8\" (UniqueName: \"kubernetes.io/projected/fdad6c63-d420-4cec-b238-743b712c86e7-kube-api-access-8zvb8\") pod \"fdad6c63-d420-4cec-b238-743b712c86e7\" (UID: \"fdad6c63-d420-4cec-b238-743b712c86e7\") " Oct 30 03:44:55.125269 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:55.125244 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdad6c63-d420-4cec-b238-743b712c86e7-ta-internal" pod "fdad6c63-d420-4cec-b238-743b712c86e7" (UID: "fdad6c63-d420-4cec-b238-743b712c86e7"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:55.125355 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:55.125339 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/fdad6c63-d420-4cec-b238-743b712c86e7-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:55.127185 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:55.127157 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdad6c63-d420-4cec-b238-743b712c86e7-data-science-collector-ta-server-cert" pod "fdad6c63-d420-4cec-b238-743b712c86e7" (UID: "fdad6c63-d420-4cec-b238-743b712c86e7"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:55.127291 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:55.127182 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdad6c63-d420-4cec-b238-743b712c86e7-kube-api-access-8zvb8" pod "fdad6c63-d420-4cec-b238-743b712c86e7" (UID: "fdad6c63-d420-4cec-b238-743b712c86e7"). InnerVolumeSpecName "kube-api-access-8zvb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:55.225998 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:55.225964 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zvb8\" (UniqueName: \"kubernetes.io/projected/fdad6c63-d420-4cec-b238-743b712c86e7-kube-api-access-8zvb8\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:55.225998 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:55.225993 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/fdad6c63-d420-4cec-b238-743b712c86e7-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:44:55.772931 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:55.772894 5882 generic.go:356] "Generic (PLEG): container finished" podID="fdad6c63-d420-4cec-b238-743b712c86e7" containerID="8cdee7e945ad80120f40e4d4eebf94d4431c9301f6dcfa32b2c5fb266800bf0d" exitCode=0 Oct 30 03:44:55.773113 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:55.772943 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" Oct 30 03:44:55.773113 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:55.772974 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" event={"ID":"fdad6c63-d420-4cec-b238-743b712c86e7","Type":"ContainerDied","Data":"8cdee7e945ad80120f40e4d4eebf94d4431c9301f6dcfa32b2c5fb266800bf0d"} Oct 30 03:44:55.773113 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:55.773015 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr" event={"ID":"fdad6c63-d420-4cec-b238-743b712c86e7","Type":"ContainerDied","Data":"05f43e87841b3b1cc43d86adc4d77a4fd980edfad16798a09eaeb34fe59c1234"} Oct 30 03:44:55.773113 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:55.773038 5882 scope.go:122] "RemoveContainer" containerID="8cdee7e945ad80120f40e4d4eebf94d4431c9301f6dcfa32b2c5fb266800bf0d" Oct 30 03:44:55.781777 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:55.781730 5882 scope.go:122] "RemoveContainer" containerID="8cdee7e945ad80120f40e4d4eebf94d4431c9301f6dcfa32b2c5fb266800bf0d" Oct 30 03:44:55.782058 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:44:55.782037 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cdee7e945ad80120f40e4d4eebf94d4431c9301f6dcfa32b2c5fb266800bf0d\": container with ID starting with 8cdee7e945ad80120f40e4d4eebf94d4431c9301f6dcfa32b2c5fb266800bf0d not found: ID does not exist" containerID="8cdee7e945ad80120f40e4d4eebf94d4431c9301f6dcfa32b2c5fb266800bf0d" Oct 30 03:44:55.782121 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:55.782067 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cdee7e945ad80120f40e4d4eebf94d4431c9301f6dcfa32b2c5fb266800bf0d"} err="failed to get container status \"8cdee7e945ad80120f40e4d4eebf94d4431c9301f6dcfa32b2c5fb266800bf0d\": rpc error: code = NotFound desc = could not find container \"8cdee7e945ad80120f40e4d4eebf94d4431c9301f6dcfa32b2c5fb266800bf0d\": container with ID starting with 8cdee7e945ad80120f40e4d4eebf94d4431c9301f6dcfa32b2c5fb266800bf0d not found: ID does not exist" Oct 30 03:44:55.822054 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:55.822022 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr"] Oct 30 03:44:55.833145 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:55.833118 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6dfbcdf4b7-l95nr"] Oct 30 03:44:56.009825 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:44:56.009797 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdad6c63-d420-4cec-b238-743b712c86e7" path="/var/lib/kubelet/pods/fdad6c63-d420-4cec-b238-743b712c86e7/volumes" Oct 30 03:45:01.068802 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.068763 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-2"] Oct 30 03:45:01.069243 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.069163 5882 cpu_manager.go:414] "containerMap: removing container" podUID="fdad6c63-d420-4cec-b238-743b712c86e7" containerName="ta-container" Oct 30 03:45:01.069243 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.069176 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdad6c63-d420-4cec-b238-743b712c86e7" containerName="ta-container" Oct 30 03:45:01.069243 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.069242 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdad6c63-d420-4cec-b238-743b712c86e7" containerName="ta-container" Oct 30 03:45:01.073658 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.073636 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 30 03:45:01.099553 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.099499 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 03:45:01.099717 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.099593 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\"" Oct 30 03:45:01.099796 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.099711 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 03:45:01.099796 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.099719 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:45:01.099796 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.099736 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 03:45:01.112461 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.112432 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-2"] Oct 30 03:45:01.178893 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.178853 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-monitoring-tls-certs\") pod \"data-science-collector-collector-2\" (UID: \"2c991a01-a730-44a5-a117-287c80b7a97b\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 30 03:45:01.178893 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.178901 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rktc\" (UniqueName: \"kubernetes.io/projected/2c991a01-a730-44a5-a117-287c80b7a97b-kube-api-access-6rktc\") pod \"data-science-collector-collector-2\" (UID: \"2c991a01-a730-44a5-a117-287c80b7a97b\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 30 03:45:01.179134 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.179042 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-tls-certs\") pod \"data-science-collector-collector-2\" (UID: \"2c991a01-a730-44a5-a117-287c80b7a97b\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 30 03:45:01.179134 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.179082 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-2\" (UID: \"2c991a01-a730-44a5-a117-287c80b7a97b\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 30 03:45:01.179134 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.179105 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2c991a01-a730-44a5-a117-287c80b7a97b-otc-internal\") pod \"data-science-collector-collector-2\" (UID: \"2c991a01-a730-44a5-a117-287c80b7a97b\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 30 03:45:01.279765 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.279725 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-2\" (UID: \"2c991a01-a730-44a5-a117-287c80b7a97b\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 30 03:45:01.279765 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.279768 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2c991a01-a730-44a5-a117-287c80b7a97b-otc-internal\") pod \"data-science-collector-collector-2\" (UID: \"2c991a01-a730-44a5-a117-287c80b7a97b\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 30 03:45:01.279985 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.279824 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-monitoring-tls-certs\") pod \"data-science-collector-collector-2\" (UID: \"2c991a01-a730-44a5-a117-287c80b7a97b\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 30 03:45:01.279985 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.279851 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rktc\" (UniqueName: \"kubernetes.io/projected/2c991a01-a730-44a5-a117-287c80b7a97b-kube-api-access-6rktc\") pod \"data-science-collector-collector-2\" (UID: \"2c991a01-a730-44a5-a117-287c80b7a97b\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 30 03:45:01.279985 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.279943 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-tls-certs\") pod \"data-science-collector-collector-2\" (UID: \"2c991a01-a730-44a5-a117-287c80b7a97b\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 30 03:45:01.280714 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.280685 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2c991a01-a730-44a5-a117-287c80b7a97b-otc-internal\") pod \"data-science-collector-collector-2\" (UID: \"2c991a01-a730-44a5-a117-287c80b7a97b\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 30 03:45:01.282412 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.282384 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-tls-certs\") pod \"data-science-collector-collector-2\" (UID: \"2c991a01-a730-44a5-a117-287c80b7a97b\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 30 03:45:01.282503 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.282388 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-monitoring-tls-certs\") pod \"data-science-collector-collector-2\" (UID: \"2c991a01-a730-44a5-a117-287c80b7a97b\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 30 03:45:01.282503 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.282453 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-2\" (UID: \"2c991a01-a730-44a5-a117-287c80b7a97b\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 30 03:45:01.308109 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.308081 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rktc\" (UniqueName: \"kubernetes.io/projected/2c991a01-a730-44a5-a117-287c80b7a97b-kube-api-access-6rktc\") pod \"data-science-collector-collector-2\" (UID: \"2c991a01-a730-44a5-a117-287c80b7a97b\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 30 03:45:01.340895 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.340829 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4"] Oct 30 03:45:01.344612 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.344587 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" Oct 30 03:45:01.364655 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.364610 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4"] Oct 30 03:45:01.386241 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.386210 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 30 03:45:01.481751 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.481725 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjkt8\" (UniqueName: \"kubernetes.io/projected/093d858f-363d-4a0c-8246-a297d65476ea-kube-api-access-bjkt8\") pod \"data-science-collector-targetallocator-7556764457-c5hg4\" (UID: \"093d858f-363d-4a0c-8246-a297d65476ea\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" Oct 30 03:45:01.481906 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.481781 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/093d858f-363d-4a0c-8246-a297d65476ea-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7556764457-c5hg4\" (UID: \"093d858f-363d-4a0c-8246-a297d65476ea\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" Oct 30 03:45:01.481906 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.481838 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/093d858f-363d-4a0c-8246-a297d65476ea-ta-internal\") pod \"data-science-collector-targetallocator-7556764457-c5hg4\" (UID: \"093d858f-363d-4a0c-8246-a297d65476ea\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" Oct 30 03:45:01.583055 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.583025 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/093d858f-363d-4a0c-8246-a297d65476ea-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7556764457-c5hg4\" (UID: \"093d858f-363d-4a0c-8246-a297d65476ea\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" Oct 30 03:45:01.583195 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.583067 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/093d858f-363d-4a0c-8246-a297d65476ea-ta-internal\") pod \"data-science-collector-targetallocator-7556764457-c5hg4\" (UID: \"093d858f-363d-4a0c-8246-a297d65476ea\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" Oct 30 03:45:01.583195 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.583118 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjkt8\" (UniqueName: \"kubernetes.io/projected/093d858f-363d-4a0c-8246-a297d65476ea-kube-api-access-bjkt8\") pod \"data-science-collector-targetallocator-7556764457-c5hg4\" (UID: \"093d858f-363d-4a0c-8246-a297d65476ea\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" Oct 30 03:45:01.583755 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.583733 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/093d858f-363d-4a0c-8246-a297d65476ea-ta-internal\") pod \"data-science-collector-targetallocator-7556764457-c5hg4\" (UID: \"093d858f-363d-4a0c-8246-a297d65476ea\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" Oct 30 03:45:01.585207 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.585184 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/093d858f-363d-4a0c-8246-a297d65476ea-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7556764457-c5hg4\" (UID: \"093d858f-363d-4a0c-8246-a297d65476ea\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" Oct 30 03:45:01.606783 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.606730 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjkt8\" (UniqueName: \"kubernetes.io/projected/093d858f-363d-4a0c-8246-a297d65476ea-kube-api-access-bjkt8\") pod \"data-science-collector-targetallocator-7556764457-c5hg4\" (UID: \"093d858f-363d-4a0c-8246-a297d65476ea\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" Oct 30 03:45:01.656572 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.656542 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" Oct 30 03:45:01.687103 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.687071 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-2"] Oct 30 03:45:01.689676 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:45:01.689643 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c991a01_a730_44a5_a117_287c80b7a97b.slice/crio-14986b35164fcbf35899a8d88c59995994880b34f1a0b886cc7ca17710439674 WatchSource:0}: Error finding container 14986b35164fcbf35899a8d88c59995994880b34f1a0b886cc7ca17710439674: Status 404 returned error can't find the container with id 14986b35164fcbf35899a8d88c59995994880b34f1a0b886cc7ca17710439674 Oct 30 03:45:01.801748 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.801716 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-2" event={"ID":"2c991a01-a730-44a5-a117-287c80b7a97b","Type":"ContainerStarted","Data":"14986b35164fcbf35899a8d88c59995994880b34f1a0b886cc7ca17710439674"} Oct 30 03:45:01.949443 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.949368 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4"] Oct 30 03:45:01.952814 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:45:01.952784 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod093d858f_363d_4a0c_8246_a297d65476ea.slice/crio-4c2c3cf07d89ff336d3f81d07ddc71b79a5d07d0a6c8be9a1f3e6a921e4147f5 WatchSource:0}: Error finding container 4c2c3cf07d89ff336d3f81d07ddc71b79a5d07d0a6c8be9a1f3e6a921e4147f5: Status 404 returned error can't find the container with id 4c2c3cf07d89ff336d3f81d07ddc71b79a5d07d0a6c8be9a1f3e6a921e4147f5 Oct 30 03:45:01.954968 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.954938 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:45:01.955032 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:01.954993 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:45:02.808099 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:02.808067 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" event={"ID":"093d858f-363d-4a0c-8246-a297d65476ea","Type":"ContainerStarted","Data":"7a8a009a6149b07e65a93c1d1bcfe8d01204c1c371f9a2e0f186cc7e56c791a1"} Oct 30 03:45:02.808099 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:02.808106 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" event={"ID":"093d858f-363d-4a0c-8246-a297d65476ea","Type":"ContainerStarted","Data":"4c2c3cf07d89ff336d3f81d07ddc71b79a5d07d0a6c8be9a1f3e6a921e4147f5"} Oct 30 03:45:02.808652 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:02.808208 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" Oct 30 03:45:02.809858 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:02.809838 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" Oct 30 03:45:02.849964 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:02.849899 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" podStartSLOduration=1.84988383 podStartE2EDuration="1.84988383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:02.849032354 +0000 UTC m=+3814.963347329" watchObservedRunningTime="2026-10-30 03:45:02.84988383 +0000 UTC m=+3814.964198806" Oct 30 03:45:02.919933 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:02.919901 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l"] Oct 30 03:45:02.920274 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:02.920218 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" podUID="f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3" containerName="ta-container" containerID="cri-o://1790dadfad9d3f4f1705046963be4d403103895e6a4f530a5eacad9681225c2e" gracePeriod=30 Oct 30 03:45:03.180220 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:03.180196 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" Oct 30 03:45:03.302141 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:03.302111 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-data-science-collector-ta-server-cert\") pod \"f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3\" (UID: \"f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3\") " Oct 30 03:45:03.302299 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:03.302158 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-kube-api-access-txfld\" (UniqueName: \"kubernetes.io/projected/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-kube-api-access-txfld\") pod \"f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3\" (UID: \"f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3\") " Oct 30 03:45:03.302299 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:03.302202 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-ta-internal\" (UniqueName: \"kubernetes.io/configmap/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-ta-internal\") pod \"f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3\" (UID: \"f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3\") " Oct 30 03:45:03.302660 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:03.302604 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-ta-internal" pod "f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3" (UID: "f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:03.304351 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:03.304320 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-data-science-collector-ta-server-cert" pod "f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3" (UID: "f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:45:03.304461 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:03.304398 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-kube-api-access-txfld" pod "f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3" (UID: "f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3"). InnerVolumeSpecName "kube-api-access-txfld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:03.403268 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:03.403185 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:45:03.403268 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:03.403217 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:45:03.403268 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:03.403229 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txfld\" (UniqueName: \"kubernetes.io/projected/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3-kube-api-access-txfld\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:45:03.814865 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:03.814832 5882 generic.go:356] "Generic (PLEG): container finished" podID="f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3" containerID="1790dadfad9d3f4f1705046963be4d403103895e6a4f530a5eacad9681225c2e" exitCode=0 Oct 30 03:45:03.815338 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:03.814879 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" Oct 30 03:45:03.815338 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:03.814920 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" event={"ID":"f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3","Type":"ContainerDied","Data":"1790dadfad9d3f4f1705046963be4d403103895e6a4f530a5eacad9681225c2e"} Oct 30 03:45:03.815338 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:03.814964 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l" event={"ID":"f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3","Type":"ContainerDied","Data":"4f6493aababcf44f00efc5a9c984b49a31a6a647afcfb2f4c1ebe0e2db50d3ee"} Oct 30 03:45:03.815338 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:03.814984 5882 scope.go:122] "RemoveContainer" containerID="1790dadfad9d3f4f1705046963be4d403103895e6a4f530a5eacad9681225c2e" Oct 30 03:45:03.827192 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:03.827166 5882 scope.go:122] "RemoveContainer" containerID="1790dadfad9d3f4f1705046963be4d403103895e6a4f530a5eacad9681225c2e" Oct 30 03:45:03.827528 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:45:03.827501 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1790dadfad9d3f4f1705046963be4d403103895e6a4f530a5eacad9681225c2e\": container with ID starting with 1790dadfad9d3f4f1705046963be4d403103895e6a4f530a5eacad9681225c2e not found: ID does not exist" containerID="1790dadfad9d3f4f1705046963be4d403103895e6a4f530a5eacad9681225c2e" Oct 30 03:45:03.827631 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:03.827540 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1790dadfad9d3f4f1705046963be4d403103895e6a4f530a5eacad9681225c2e"} err="failed to get container status \"1790dadfad9d3f4f1705046963be4d403103895e6a4f530a5eacad9681225c2e\": rpc error: code = NotFound desc = could not find container \"1790dadfad9d3f4f1705046963be4d403103895e6a4f530a5eacad9681225c2e\": container with ID starting with 1790dadfad9d3f4f1705046963be4d403103895e6a4f530a5eacad9681225c2e not found: ID does not exist" Oct 30 03:45:03.863044 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:03.863015 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l"] Oct 30 03:45:03.874739 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:03.874711 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5d89dc86c4-rms2l"] Oct 30 03:45:04.010892 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:04.010848 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3" path="/var/lib/kubelet/pods/f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3/volumes" Oct 30 03:45:06.543186 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:06.543156 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:45:06.543450 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:06.543218 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:45:06.831155 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:06.831076 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-2" event={"ID":"2c991a01-a730-44a5-a117-287c80b7a97b","Type":"ContainerStarted","Data":"7d10bcc60a60c36c749822ba498f1bb407d12208de723d4617dd7297ae611836"} Oct 30 03:45:06.874948 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:06.874881 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-2" podStartSLOduration=1.024342478 podStartE2EDuration="5.874862209s" totalImagesPullingTime="4.850519731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:06.872727745 +0000 UTC m=+3818.987042721" watchObservedRunningTime="2026-10-30 03:45:06.874862209 +0000 UTC m=+3818.989177184" Oct 30 03:45:09.807603 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:09.807562 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-2"] Oct 30 03:45:09.808238 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:09.807782 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-2" podUID="2c991a01-a730-44a5-a117-287c80b7a97b" containerName="otc-container" containerID="cri-o://7d10bcc60a60c36c749822ba498f1bb407d12208de723d4617dd7297ae611836" gracePeriod=30 Oct 30 03:45:10.047735 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.047711 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 30 03:45:10.062879 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.062804 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt"] Oct 30 03:45:10.063227 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.063207 5882 cpu_manager.go:414] "containerMap: removing container" podUID="2c991a01-a730-44a5-a117-287c80b7a97b" containerName="otc-container" Oct 30 03:45:10.063227 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.063227 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c991a01-a730-44a5-a117-287c80b7a97b" containerName="otc-container" Oct 30 03:45:10.063351 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.063245 5882 cpu_manager.go:414] "containerMap: removing container" podUID="f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3" containerName="ta-container" Oct 30 03:45:10.063351 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.063251 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3" containerName="ta-container" Oct 30 03:45:10.063419 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.063352 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c991a01-a730-44a5-a117-287c80b7a97b" containerName="otc-container" Oct 30 03:45:10.063419 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.063363 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f57b39ef-0c0c-4d76-a2f9-5e4febc1d5c3" containerName="ta-container" Oct 30 03:45:10.066728 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.066712 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" Oct 30 03:45:10.087051 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.087025 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt"] Oct 30 03:45:10.161505 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.161473 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-monitoring-tls-certs\") pod \"2c991a01-a730-44a5-a117-287c80b7a97b\" (UID: \"2c991a01-a730-44a5-a117-287c80b7a97b\") " Oct 30 03:45:10.161700 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.161512 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-tls-certs\") pod \"2c991a01-a730-44a5-a117-287c80b7a97b\" (UID: \"2c991a01-a730-44a5-a117-287c80b7a97b\") " Oct 30 03:45:10.161700 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.161532 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c991a01-a730-44a5-a117-287c80b7a97b-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2c991a01-a730-44a5-a117-287c80b7a97b-otc-internal\") pod \"2c991a01-a730-44a5-a117-287c80b7a97b\" (UID: \"2c991a01-a730-44a5-a117-287c80b7a97b\") " Oct 30 03:45:10.161700 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.161569 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c991a01-a730-44a5-a117-287c80b7a97b-kube-api-access-6rktc\" (UniqueName: \"kubernetes.io/projected/2c991a01-a730-44a5-a117-287c80b7a97b-kube-api-access-6rktc\") pod \"2c991a01-a730-44a5-a117-287c80b7a97b\" (UID: \"2c991a01-a730-44a5-a117-287c80b7a97b\") " Oct 30 03:45:10.161700 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.161597 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-data-science-collector-ta-client-cert\") pod \"2c991a01-a730-44a5-a117-287c80b7a97b\" (UID: \"2c991a01-a730-44a5-a117-287c80b7a97b\") " Oct 30 03:45:10.161933 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.161874 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f667e723-641f-4538-b35e-d9727a982e1a-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-78bf55ddf6-mh5zt\" (UID: \"f667e723-641f-4538-b35e-d9727a982e1a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" Oct 30 03:45:10.161999 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.161931 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c991a01-a730-44a5-a117-287c80b7a97b-otc-internal" pod "2c991a01-a730-44a5-a117-287c80b7a97b" (UID: "2c991a01-a730-44a5-a117-287c80b7a97b"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:10.161999 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.161960 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f667e723-641f-4538-b35e-d9727a982e1a-ta-internal\") pod \"data-science-collector-targetallocator-78bf55ddf6-mh5zt\" (UID: \"f667e723-641f-4538-b35e-d9727a982e1a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" Oct 30 03:45:10.162115 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.162050 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmvrk\" (UniqueName: \"kubernetes.io/projected/f667e723-641f-4538-b35e-d9727a982e1a-kube-api-access-pmvrk\") pod \"data-science-collector-targetallocator-78bf55ddf6-mh5zt\" (UID: \"f667e723-641f-4538-b35e-d9727a982e1a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" Oct 30 03:45:10.162194 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.162174 5882 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2c991a01-a730-44a5-a117-287c80b7a97b-otc-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:45:10.163930 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.163903 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-monitoring-tls-certs" pod "2c991a01-a730-44a5-a117-287c80b7a97b" (UID: "2c991a01-a730-44a5-a117-287c80b7a97b"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:45:10.164035 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.163945 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-data-science-collector-ta-client-cert" pod "2c991a01-a730-44a5-a117-287c80b7a97b" (UID: "2c991a01-a730-44a5-a117-287c80b7a97b"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:45:10.164194 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.164170 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-tls-certs" pod "2c991a01-a730-44a5-a117-287c80b7a97b" (UID: "2c991a01-a730-44a5-a117-287c80b7a97b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:45:10.164296 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.164193 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c991a01-a730-44a5-a117-287c80b7a97b-kube-api-access-6rktc" pod "2c991a01-a730-44a5-a117-287c80b7a97b" (UID: "2c991a01-a730-44a5-a117-287c80b7a97b"). InnerVolumeSpecName "kube-api-access-6rktc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:10.263210 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.263173 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f667e723-641f-4538-b35e-d9727a982e1a-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-78bf55ddf6-mh5zt\" (UID: \"f667e723-641f-4538-b35e-d9727a982e1a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" Oct 30 03:45:10.263372 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.263220 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f667e723-641f-4538-b35e-d9727a982e1a-ta-internal\") pod \"data-science-collector-targetallocator-78bf55ddf6-mh5zt\" (UID: \"f667e723-641f-4538-b35e-d9727a982e1a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" Oct 30 03:45:10.263372 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.263258 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmvrk\" (UniqueName: \"kubernetes.io/projected/f667e723-641f-4538-b35e-d9727a982e1a-kube-api-access-pmvrk\") pod \"data-science-collector-targetallocator-78bf55ddf6-mh5zt\" (UID: \"f667e723-641f-4538-b35e-d9727a982e1a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" Oct 30 03:45:10.263372 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.263290 5882 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-monitoring-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:45:10.263372 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.263301 5882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:45:10.263372 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.263310 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rktc\" (UniqueName: \"kubernetes.io/projected/2c991a01-a730-44a5-a117-287c80b7a97b-kube-api-access-6rktc\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:45:10.263372 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.263323 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2c991a01-a730-44a5-a117-287c80b7a97b-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:45:10.263981 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.263956 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f667e723-641f-4538-b35e-d9727a982e1a-ta-internal\") pod \"data-science-collector-targetallocator-78bf55ddf6-mh5zt\" (UID: \"f667e723-641f-4538-b35e-d9727a982e1a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" Oct 30 03:45:10.265569 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.265550 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f667e723-641f-4538-b35e-d9727a982e1a-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-78bf55ddf6-mh5zt\" (UID: \"f667e723-641f-4538-b35e-d9727a982e1a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" Oct 30 03:45:10.285810 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.285787 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmvrk\" (UniqueName: \"kubernetes.io/projected/f667e723-641f-4538-b35e-d9727a982e1a-kube-api-access-pmvrk\") pod \"data-science-collector-targetallocator-78bf55ddf6-mh5zt\" (UID: \"f667e723-641f-4538-b35e-d9727a982e1a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" Oct 30 03:45:10.377911 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.377845 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" Oct 30 03:45:10.669056 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.668894 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt"] Oct 30 03:45:10.671825 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:45:10.671796 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf667e723_641f_4538_b35e_d9727a982e1a.slice/crio-a0f3770bb751423b931b8d38a6348ecdc44fe24980e19a9a9b43b5ae4a95a3a8 WatchSource:0}: Error finding container a0f3770bb751423b931b8d38a6348ecdc44fe24980e19a9a9b43b5ae4a95a3a8: Status 404 returned error can't find the container with id a0f3770bb751423b931b8d38a6348ecdc44fe24980e19a9a9b43b5ae4a95a3a8 Oct 30 03:45:10.673904 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.673876 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:45:10.673960 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.673927 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:45:10.850641 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.850583 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" event={"ID":"f667e723-641f-4538-b35e-d9727a982e1a","Type":"ContainerStarted","Data":"fd9e3938cc301f1dada072dfe15e9ef7d6e725b8d8457b62ed26b53f7ea4f8b1"} Oct 30 03:45:10.850641 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.850641 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" event={"ID":"f667e723-641f-4538-b35e-d9727a982e1a","Type":"ContainerStarted","Data":"a0f3770bb751423b931b8d38a6348ecdc44fe24980e19a9a9b43b5ae4a95a3a8"} Oct 30 03:45:10.851187 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.850665 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" Oct 30 03:45:10.851864 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.851832 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" podUID="f667e723-641f-4538-b35e-d9727a982e1a" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.57:8080/readyz\": dial tcp 10.134.0.57:8080: connect: connection refused" Oct 30 03:45:10.852449 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.852427 5882 generic.go:356] "Generic (PLEG): container finished" podID="2c991a01-a730-44a5-a117-287c80b7a97b" containerID="7d10bcc60a60c36c749822ba498f1bb407d12208de723d4617dd7297ae611836" exitCode=0 Oct 30 03:45:10.852528 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.852469 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 30 03:45:10.852528 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.852495 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-2" event={"ID":"2c991a01-a730-44a5-a117-287c80b7a97b","Type":"ContainerDied","Data":"7d10bcc60a60c36c749822ba498f1bb407d12208de723d4617dd7297ae611836"} Oct 30 03:45:10.852592 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.852529 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-2" event={"ID":"2c991a01-a730-44a5-a117-287c80b7a97b","Type":"ContainerDied","Data":"14986b35164fcbf35899a8d88c59995994880b34f1a0b886cc7ca17710439674"} Oct 30 03:45:10.852592 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.852551 5882 scope.go:122] "RemoveContainer" containerID="7d10bcc60a60c36c749822ba498f1bb407d12208de723d4617dd7297ae611836" Oct 30 03:45:10.862748 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.862722 5882 scope.go:122] "RemoveContainer" containerID="7d10bcc60a60c36c749822ba498f1bb407d12208de723d4617dd7297ae611836" Oct 30 03:45:10.863005 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:45:10.862982 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d10bcc60a60c36c749822ba498f1bb407d12208de723d4617dd7297ae611836\": container with ID starting with 7d10bcc60a60c36c749822ba498f1bb407d12208de723d4617dd7297ae611836 not found: ID does not exist" containerID="7d10bcc60a60c36c749822ba498f1bb407d12208de723d4617dd7297ae611836" Oct 30 03:45:10.863075 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.863016 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d10bcc60a60c36c749822ba498f1bb407d12208de723d4617dd7297ae611836"} err="failed to get container status \"7d10bcc60a60c36c749822ba498f1bb407d12208de723d4617dd7297ae611836\": rpc error: code = NotFound desc = could not find container \"7d10bcc60a60c36c749822ba498f1bb407d12208de723d4617dd7297ae611836\": container with ID starting with 7d10bcc60a60c36c749822ba498f1bb407d12208de723d4617dd7297ae611836 not found: ID does not exist" Oct 30 03:45:10.890181 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.890137 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" podStartSLOduration=0.890125046 podStartE2EDuration="890.125046ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:10.889314636 +0000 UTC m=+3823.003629623" watchObservedRunningTime="2026-10-30 03:45:10.890125046 +0000 UTC m=+3823.004440023" Oct 30 03:45:10.923158 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.923097 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-2"] Oct 30 03:45:10.927803 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:10.927782 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-2"] Oct 30 03:45:11.859844 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:11.859814 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" Oct 30 03:45:11.929292 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:11.929262 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4"] Oct 30 03:45:11.929580 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:11.929542 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" podUID="093d858f-363d-4a0c-8246-a297d65476ea" containerName="ta-container" containerID="cri-o://7a8a009a6149b07e65a93c1d1bcfe8d01204c1c371f9a2e0f186cc7e56c791a1" gracePeriod=30 Oct 30 03:45:12.009811 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:12.009784 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c991a01-a730-44a5-a117-287c80b7a97b" path="/var/lib/kubelet/pods/2c991a01-a730-44a5-a117-287c80b7a97b/volumes" Oct 30 03:45:12.173681 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:12.173653 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" Oct 30 03:45:12.279672 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:12.279635 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/093d858f-363d-4a0c-8246-a297d65476ea-ta-internal\" (UniqueName: \"kubernetes.io/configmap/093d858f-363d-4a0c-8246-a297d65476ea-ta-internal\") pod \"093d858f-363d-4a0c-8246-a297d65476ea\" (UID: \"093d858f-363d-4a0c-8246-a297d65476ea\") " Oct 30 03:45:12.279873 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:12.279693 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093d858f-363d-4a0c-8246-a297d65476ea-kube-api-access-bjkt8\" (UniqueName: \"kubernetes.io/projected/093d858f-363d-4a0c-8246-a297d65476ea-kube-api-access-bjkt8\") pod \"093d858f-363d-4a0c-8246-a297d65476ea\" (UID: \"093d858f-363d-4a0c-8246-a297d65476ea\") " Oct 30 03:45:12.279873 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:12.279717 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/093d858f-363d-4a0c-8246-a297d65476ea-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/093d858f-363d-4a0c-8246-a297d65476ea-data-science-collector-ta-server-cert\") pod \"093d858f-363d-4a0c-8246-a297d65476ea\" (UID: \"093d858f-363d-4a0c-8246-a297d65476ea\") " Oct 30 03:45:12.280065 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:12.280036 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093d858f-363d-4a0c-8246-a297d65476ea-ta-internal" pod "093d858f-363d-4a0c-8246-a297d65476ea" (UID: "093d858f-363d-4a0c-8246-a297d65476ea"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:12.281809 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:12.281782 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093d858f-363d-4a0c-8246-a297d65476ea-kube-api-access-bjkt8" pod "093d858f-363d-4a0c-8246-a297d65476ea" (UID: "093d858f-363d-4a0c-8246-a297d65476ea"). InnerVolumeSpecName "kube-api-access-bjkt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:12.281890 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:12.281838 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/093d858f-363d-4a0c-8246-a297d65476ea-data-science-collector-ta-server-cert" pod "093d858f-363d-4a0c-8246-a297d65476ea" (UID: "093d858f-363d-4a0c-8246-a297d65476ea"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:45:12.381106 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:12.381072 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/093d858f-363d-4a0c-8246-a297d65476ea-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:45:12.381106 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:12.381102 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjkt8\" (UniqueName: \"kubernetes.io/projected/093d858f-363d-4a0c-8246-a297d65476ea-kube-api-access-bjkt8\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:45:12.381106 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:12.381113 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/093d858f-363d-4a0c-8246-a297d65476ea-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:45:12.863943 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:12.863911 5882 generic.go:356] "Generic (PLEG): container finished" podID="093d858f-363d-4a0c-8246-a297d65476ea" containerID="7a8a009a6149b07e65a93c1d1bcfe8d01204c1c371f9a2e0f186cc7e56c791a1" exitCode=0 Oct 30 03:45:12.864366 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:12.863960 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" Oct 30 03:45:12.864366 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:12.864000 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" event={"ID":"093d858f-363d-4a0c-8246-a297d65476ea","Type":"ContainerDied","Data":"7a8a009a6149b07e65a93c1d1bcfe8d01204c1c371f9a2e0f186cc7e56c791a1"} Oct 30 03:45:12.864366 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:12.864046 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4" event={"ID":"093d858f-363d-4a0c-8246-a297d65476ea","Type":"ContainerDied","Data":"4c2c3cf07d89ff336d3f81d07ddc71b79a5d07d0a6c8be9a1f3e6a921e4147f5"} Oct 30 03:45:12.864366 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:12.864066 5882 scope.go:122] "RemoveContainer" containerID="7a8a009a6149b07e65a93c1d1bcfe8d01204c1c371f9a2e0f186cc7e56c791a1" Oct 30 03:45:12.873066 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:12.873049 5882 scope.go:122] "RemoveContainer" containerID="7a8a009a6149b07e65a93c1d1bcfe8d01204c1c371f9a2e0f186cc7e56c791a1" Oct 30 03:45:12.873333 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:45:12.873315 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a8a009a6149b07e65a93c1d1bcfe8d01204c1c371f9a2e0f186cc7e56c791a1\": container with ID starting with 7a8a009a6149b07e65a93c1d1bcfe8d01204c1c371f9a2e0f186cc7e56c791a1 not found: ID does not exist" containerID="7a8a009a6149b07e65a93c1d1bcfe8d01204c1c371f9a2e0f186cc7e56c791a1" Oct 30 03:45:12.873375 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:12.873341 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8a009a6149b07e65a93c1d1bcfe8d01204c1c371f9a2e0f186cc7e56c791a1"} err="failed to get container status \"7a8a009a6149b07e65a93c1d1bcfe8d01204c1c371f9a2e0f186cc7e56c791a1\": rpc error: code = NotFound desc = could not find container \"7a8a009a6149b07e65a93c1d1bcfe8d01204c1c371f9a2e0f186cc7e56c791a1\": container with ID starting with 7a8a009a6149b07e65a93c1d1bcfe8d01204c1c371f9a2e0f186cc7e56c791a1 not found: ID does not exist" Oct 30 03:45:12.913038 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:12.913009 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4"] Oct 30 03:45:12.924171 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:12.924139 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7556764457-c5hg4"] Oct 30 03:45:14.009995 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:14.009965 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="093d858f-363d-4a0c-8246-a297d65476ea" path="/var/lib/kubelet/pods/093d858f-363d-4a0c-8246-a297d65476ea/volumes" Oct 30 03:45:23.941637 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:45:23.941586 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:45:27.147936 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.147894 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq"] Oct 30 03:45:27.148332 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.148313 5882 cpu_manager.go:414] "containerMap: removing container" podUID="093d858f-363d-4a0c-8246-a297d65476ea" containerName="ta-container" Oct 30 03:45:27.148332 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.148330 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093d858f-363d-4a0c-8246-a297d65476ea" containerName="ta-container" Oct 30 03:45:27.148442 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.148429 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="093d858f-363d-4a0c-8246-a297d65476ea" containerName="ta-container" Oct 30 03:45:27.152802 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.152778 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" Oct 30 03:45:27.181609 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.181574 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq"] Oct 30 03:45:27.311951 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.311917 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnpxk\" (UniqueName: \"kubernetes.io/projected/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-kube-api-access-gnpxk\") pod \"data-science-collector-targetallocator-b6f46c55d-dn9fq\" (UID: \"a72fd6fa-4ac5-4e51-b13c-d2ef6246f027\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" Oct 30 03:45:27.312123 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.311975 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-ta-internal\") pod \"data-science-collector-targetallocator-b6f46c55d-dn9fq\" (UID: \"a72fd6fa-4ac5-4e51-b13c-d2ef6246f027\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" Oct 30 03:45:27.312123 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.312054 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-b6f46c55d-dn9fq\" (UID: \"a72fd6fa-4ac5-4e51-b13c-d2ef6246f027\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" Oct 30 03:45:27.412717 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.412637 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-ta-internal\") pod \"data-science-collector-targetallocator-b6f46c55d-dn9fq\" (UID: \"a72fd6fa-4ac5-4e51-b13c-d2ef6246f027\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" Oct 30 03:45:27.412717 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.412683 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-b6f46c55d-dn9fq\" (UID: \"a72fd6fa-4ac5-4e51-b13c-d2ef6246f027\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" Oct 30 03:45:27.412892 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.412737 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnpxk\" (UniqueName: \"kubernetes.io/projected/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-kube-api-access-gnpxk\") pod \"data-science-collector-targetallocator-b6f46c55d-dn9fq\" (UID: \"a72fd6fa-4ac5-4e51-b13c-d2ef6246f027\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" Oct 30 03:45:27.413298 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.413276 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-ta-internal\") pod \"data-science-collector-targetallocator-b6f46c55d-dn9fq\" (UID: \"a72fd6fa-4ac5-4e51-b13c-d2ef6246f027\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" Oct 30 03:45:27.415258 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.415235 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-b6f46c55d-dn9fq\" (UID: \"a72fd6fa-4ac5-4e51-b13c-d2ef6246f027\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" Oct 30 03:45:27.435733 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.435705 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnpxk\" (UniqueName: \"kubernetes.io/projected/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-kube-api-access-gnpxk\") pod \"data-science-collector-targetallocator-b6f46c55d-dn9fq\" (UID: \"a72fd6fa-4ac5-4e51-b13c-d2ef6246f027\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" Oct 30 03:45:27.464557 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.464528 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" Oct 30 03:45:27.753230 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.753203 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq"] Oct 30 03:45:27.756248 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:45:27.756218 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda72fd6fa_4ac5_4e51_b13c_d2ef6246f027.slice/crio-a03dc815b43cdbb313b9591556d1c11a48fa2c046db49a72997fa3a574a4a476 WatchSource:0}: Error finding container a03dc815b43cdbb313b9591556d1c11a48fa2c046db49a72997fa3a574a4a476: Status 404 returned error can't find the container with id a03dc815b43cdbb313b9591556d1c11a48fa2c046db49a72997fa3a574a4a476 Oct 30 03:45:27.758720 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.758688 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:45:27.758819 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.758753 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:45:27.944524 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.944490 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" event={"ID":"a72fd6fa-4ac5-4e51-b13c-d2ef6246f027","Type":"ContainerStarted","Data":"f853c7844b211a270ead7373d97a938981c69a1bd6b4338cf2af1a1931159139"} Oct 30 03:45:27.944524 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.944531 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" event={"ID":"a72fd6fa-4ac5-4e51-b13c-d2ef6246f027","Type":"ContainerStarted","Data":"a03dc815b43cdbb313b9591556d1c11a48fa2c046db49a72997fa3a574a4a476"} Oct 30 03:45:27.944785 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.944725 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" Oct 30 03:45:27.945686 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.945663 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" podUID="a72fd6fa-4ac5-4e51-b13c-d2ef6246f027" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.58:8080/readyz\": dial tcp 10.134.0.58:8080: connect: connection refused" Oct 30 03:45:27.985520 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:27.985430 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" podStartSLOduration=0.985417381 podStartE2EDuration="985.417381ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:27.984241531 +0000 UTC m=+3840.098556506" watchObservedRunningTime="2026-10-30 03:45:27.985417381 +0000 UTC m=+3840.099732356" Oct 30 03:45:28.950096 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:28.950012 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" Oct 30 03:45:29.035523 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:29.035481 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt"] Oct 30 03:45:29.035948 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:29.035830 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" podUID="f667e723-641f-4538-b35e-d9727a982e1a" containerName="ta-container" containerID="cri-o://fd9e3938cc301f1dada072dfe15e9ef7d6e725b8d8457b62ed26b53f7ea4f8b1" gracePeriod=30 Oct 30 03:45:29.277950 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:29.277920 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" Oct 30 03:45:29.430481 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:29.430446 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f667e723-641f-4538-b35e-d9727a982e1a-kube-api-access-pmvrk\" (UniqueName: \"kubernetes.io/projected/f667e723-641f-4538-b35e-d9727a982e1a-kube-api-access-pmvrk\") pod \"f667e723-641f-4538-b35e-d9727a982e1a\" (UID: \"f667e723-641f-4538-b35e-d9727a982e1a\") " Oct 30 03:45:29.430650 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:29.430496 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f667e723-641f-4538-b35e-d9727a982e1a-ta-internal\" (UniqueName: \"kubernetes.io/configmap/f667e723-641f-4538-b35e-d9727a982e1a-ta-internal\") pod \"f667e723-641f-4538-b35e-d9727a982e1a\" (UID: \"f667e723-641f-4538-b35e-d9727a982e1a\") " Oct 30 03:45:29.430650 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:29.430534 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f667e723-641f-4538-b35e-d9727a982e1a-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f667e723-641f-4538-b35e-d9727a982e1a-data-science-collector-ta-server-cert\") pod \"f667e723-641f-4538-b35e-d9727a982e1a\" (UID: \"f667e723-641f-4538-b35e-d9727a982e1a\") " Oct 30 03:45:29.430897 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:29.430867 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f667e723-641f-4538-b35e-d9727a982e1a-ta-internal" pod "f667e723-641f-4538-b35e-d9727a982e1a" (UID: "f667e723-641f-4538-b35e-d9727a982e1a"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:29.432707 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:29.432682 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f667e723-641f-4538-b35e-d9727a982e1a-data-science-collector-ta-server-cert" pod "f667e723-641f-4538-b35e-d9727a982e1a" (UID: "f667e723-641f-4538-b35e-d9727a982e1a"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:45:29.432801 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:29.432715 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f667e723-641f-4538-b35e-d9727a982e1a-kube-api-access-pmvrk" pod "f667e723-641f-4538-b35e-d9727a982e1a" (UID: "f667e723-641f-4538-b35e-d9727a982e1a"). InnerVolumeSpecName "kube-api-access-pmvrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:29.531426 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:29.531397 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmvrk\" (UniqueName: \"kubernetes.io/projected/f667e723-641f-4538-b35e-d9727a982e1a-kube-api-access-pmvrk\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:45:29.531426 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:29.531423 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f667e723-641f-4538-b35e-d9727a982e1a-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:45:29.531604 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:29.531433 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f667e723-641f-4538-b35e-d9727a982e1a-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:45:29.954532 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:29.954445 5882 generic.go:356] "Generic (PLEG): container finished" podID="f667e723-641f-4538-b35e-d9727a982e1a" containerID="fd9e3938cc301f1dada072dfe15e9ef7d6e725b8d8457b62ed26b53f7ea4f8b1" exitCode=0 Oct 30 03:45:29.954532 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:29.954493 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" Oct 30 03:45:29.955019 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:29.954533 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" event={"ID":"f667e723-641f-4538-b35e-d9727a982e1a","Type":"ContainerDied","Data":"fd9e3938cc301f1dada072dfe15e9ef7d6e725b8d8457b62ed26b53f7ea4f8b1"} Oct 30 03:45:29.955019 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:29.954580 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt" event={"ID":"f667e723-641f-4538-b35e-d9727a982e1a","Type":"ContainerDied","Data":"a0f3770bb751423b931b8d38a6348ecdc44fe24980e19a9a9b43b5ae4a95a3a8"} Oct 30 03:45:29.955019 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:29.954600 5882 scope.go:122] "RemoveContainer" containerID="fd9e3938cc301f1dada072dfe15e9ef7d6e725b8d8457b62ed26b53f7ea4f8b1" Oct 30 03:45:29.963786 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:29.963770 5882 scope.go:122] "RemoveContainer" containerID="fd9e3938cc301f1dada072dfe15e9ef7d6e725b8d8457b62ed26b53f7ea4f8b1" Oct 30 03:45:29.964031 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:45:29.964013 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9e3938cc301f1dada072dfe15e9ef7d6e725b8d8457b62ed26b53f7ea4f8b1\": container with ID starting with fd9e3938cc301f1dada072dfe15e9ef7d6e725b8d8457b62ed26b53f7ea4f8b1 not found: ID does not exist" containerID="fd9e3938cc301f1dada072dfe15e9ef7d6e725b8d8457b62ed26b53f7ea4f8b1" Oct 30 03:45:29.964066 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:29.964040 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9e3938cc301f1dada072dfe15e9ef7d6e725b8d8457b62ed26b53f7ea4f8b1"} err="failed to get container status \"fd9e3938cc301f1dada072dfe15e9ef7d6e725b8d8457b62ed26b53f7ea4f8b1\": rpc error: code = NotFound desc = could not find container \"fd9e3938cc301f1dada072dfe15e9ef7d6e725b8d8457b62ed26b53f7ea4f8b1\": container with ID starting with fd9e3938cc301f1dada072dfe15e9ef7d6e725b8d8457b62ed26b53f7ea4f8b1 not found: ID does not exist" Oct 30 03:45:30.002476 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:30.002446 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt"] Oct 30 03:45:30.014051 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:30.014026 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-78bf55ddf6-mh5zt"] Oct 30 03:45:32.010100 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:32.010066 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f667e723-641f-4538-b35e-d9727a982e1a" path="/var/lib/kubelet/pods/f667e723-641f-4538-b35e-d9727a982e1a/volumes" Oct 30 03:45:34.272785 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.272756 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9"] Oct 30 03:45:34.273192 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.273104 5882 cpu_manager.go:414] "containerMap: removing container" podUID="f667e723-641f-4538-b35e-d9727a982e1a" containerName="ta-container" Oct 30 03:45:34.273192 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.273115 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f667e723-641f-4538-b35e-d9727a982e1a" containerName="ta-container" Oct 30 03:45:34.273280 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.273199 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f667e723-641f-4538-b35e-d9727a982e1a" containerName="ta-container" Oct 30 03:45:34.276206 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.276186 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" Oct 30 03:45:34.302719 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.302694 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9"] Oct 30 03:45:34.373850 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.373819 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3679fb4c-772b-46a5-ab2e-e5105a819b6f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-744cffff95-699q9\" (UID: \"3679fb4c-772b-46a5-ab2e-e5105a819b6f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" Oct 30 03:45:34.373990 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.373930 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/3679fb4c-772b-46a5-ab2e-e5105a819b6f-ta-internal\") pod \"data-science-collector-targetallocator-744cffff95-699q9\" (UID: \"3679fb4c-772b-46a5-ab2e-e5105a819b6f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" Oct 30 03:45:34.373990 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.373968 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v29q9\" (UniqueName: \"kubernetes.io/projected/3679fb4c-772b-46a5-ab2e-e5105a819b6f-kube-api-access-v29q9\") pod \"data-science-collector-targetallocator-744cffff95-699q9\" (UID: \"3679fb4c-772b-46a5-ab2e-e5105a819b6f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" Oct 30 03:45:34.474631 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.474582 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v29q9\" (UniqueName: \"kubernetes.io/projected/3679fb4c-772b-46a5-ab2e-e5105a819b6f-kube-api-access-v29q9\") pod \"data-science-collector-targetallocator-744cffff95-699q9\" (UID: \"3679fb4c-772b-46a5-ab2e-e5105a819b6f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" Oct 30 03:45:34.474790 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.474649 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3679fb4c-772b-46a5-ab2e-e5105a819b6f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-744cffff95-699q9\" (UID: \"3679fb4c-772b-46a5-ab2e-e5105a819b6f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" Oct 30 03:45:34.474790 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.474705 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/3679fb4c-772b-46a5-ab2e-e5105a819b6f-ta-internal\") pod \"data-science-collector-targetallocator-744cffff95-699q9\" (UID: \"3679fb4c-772b-46a5-ab2e-e5105a819b6f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" Oct 30 03:45:34.475323 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.475302 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/3679fb4c-772b-46a5-ab2e-e5105a819b6f-ta-internal\") pod \"data-science-collector-targetallocator-744cffff95-699q9\" (UID: \"3679fb4c-772b-46a5-ab2e-e5105a819b6f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" Oct 30 03:45:34.477285 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.477262 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3679fb4c-772b-46a5-ab2e-e5105a819b6f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-744cffff95-699q9\" (UID: \"3679fb4c-772b-46a5-ab2e-e5105a819b6f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" Oct 30 03:45:34.497400 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.497380 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v29q9\" (UniqueName: \"kubernetes.io/projected/3679fb4c-772b-46a5-ab2e-e5105a819b6f-kube-api-access-v29q9\") pod \"data-science-collector-targetallocator-744cffff95-699q9\" (UID: \"3679fb4c-772b-46a5-ab2e-e5105a819b6f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" Oct 30 03:45:34.587505 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.587427 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" Oct 30 03:45:34.885785 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.885760 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9"] Oct 30 03:45:34.888405 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:45:34.888378 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3679fb4c_772b_46a5_ab2e_e5105a819b6f.slice/crio-39f248936e08fa351c1f2ca4b190ab4a0976232c2c6fb011516e7e0241518b24 WatchSource:0}: Error finding container 39f248936e08fa351c1f2ca4b190ab4a0976232c2c6fb011516e7e0241518b24: Status 404 returned error can't find the container with id 39f248936e08fa351c1f2ca4b190ab4a0976232c2c6fb011516e7e0241518b24 Oct 30 03:45:34.891056 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.891010 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:45:34.891146 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.891092 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:45:34.983552 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.983512 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" event={"ID":"3679fb4c-772b-46a5-ab2e-e5105a819b6f","Type":"ContainerStarted","Data":"1c4f73ad07328347be0b77086db7d5a652377ae42716503036d0efb9137da5ac"} Oct 30 03:45:34.983552 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.983557 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" event={"ID":"3679fb4c-772b-46a5-ab2e-e5105a819b6f","Type":"ContainerStarted","Data":"39f248936e08fa351c1f2ca4b190ab4a0976232c2c6fb011516e7e0241518b24"} Oct 30 03:45:34.983799 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.983778 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" Oct 30 03:45:34.985156 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:34.985128 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" podUID="3679fb4c-772b-46a5-ab2e-e5105a819b6f" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.59:8080/readyz\": dial tcp 10.134.0.59:8080: connect: connection refused" Oct 30 03:45:35.024770 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:35.024713 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" podStartSLOduration=1.0246996290000001 podStartE2EDuration="1.024699629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:35.023699153 +0000 UTC m=+3847.138014128" watchObservedRunningTime="2026-10-30 03:45:35.024699629 +0000 UTC m=+3847.139014604" Oct 30 03:45:35.988658 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:35.988627 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" Oct 30 03:45:36.058431 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:36.058397 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq"] Oct 30 03:45:36.058750 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:36.058717 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" podUID="a72fd6fa-4ac5-4e51-b13c-d2ef6246f027" containerName="ta-container" containerID="cri-o://f853c7844b211a270ead7373d97a938981c69a1bd6b4338cf2af1a1931159139" gracePeriod=30 Oct 30 03:45:36.297090 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:36.297066 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" Oct 30 03:45:36.393325 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:36.393287 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-data-science-collector-ta-server-cert\") pod \"a72fd6fa-4ac5-4e51-b13c-d2ef6246f027\" (UID: \"a72fd6fa-4ac5-4e51-b13c-d2ef6246f027\") " Oct 30 03:45:36.393466 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:36.393346 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-kube-api-access-gnpxk\" (UniqueName: \"kubernetes.io/projected/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-kube-api-access-gnpxk\") pod \"a72fd6fa-4ac5-4e51-b13c-d2ef6246f027\" (UID: \"a72fd6fa-4ac5-4e51-b13c-d2ef6246f027\") " Oct 30 03:45:36.393466 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:36.393363 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-ta-internal\" (UniqueName: \"kubernetes.io/configmap/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-ta-internal\") pod \"a72fd6fa-4ac5-4e51-b13c-d2ef6246f027\" (UID: \"a72fd6fa-4ac5-4e51-b13c-d2ef6246f027\") " Oct 30 03:45:36.393772 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:36.393751 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-ta-internal" pod "a72fd6fa-4ac5-4e51-b13c-d2ef6246f027" (UID: "a72fd6fa-4ac5-4e51-b13c-d2ef6246f027"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:36.395371 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:36.395346 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-kube-api-access-gnpxk" pod "a72fd6fa-4ac5-4e51-b13c-d2ef6246f027" (UID: "a72fd6fa-4ac5-4e51-b13c-d2ef6246f027"). InnerVolumeSpecName "kube-api-access-gnpxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:36.395457 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:36.395396 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-data-science-collector-ta-server-cert" pod "a72fd6fa-4ac5-4e51-b13c-d2ef6246f027" (UID: "a72fd6fa-4ac5-4e51-b13c-d2ef6246f027"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:45:36.494923 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:36.494892 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:45:36.494923 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:36.494922 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnpxk\" (UniqueName: \"kubernetes.io/projected/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-kube-api-access-gnpxk\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:45:36.495082 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:36.494933 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:45:36.993226 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:36.993191 5882 generic.go:356] "Generic (PLEG): container finished" podID="a72fd6fa-4ac5-4e51-b13c-d2ef6246f027" containerID="f853c7844b211a270ead7373d97a938981c69a1bd6b4338cf2af1a1931159139" exitCode=0 Oct 30 03:45:36.993728 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:36.993241 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" Oct 30 03:45:36.993728 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:36.993279 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" event={"ID":"a72fd6fa-4ac5-4e51-b13c-d2ef6246f027","Type":"ContainerDied","Data":"f853c7844b211a270ead7373d97a938981c69a1bd6b4338cf2af1a1931159139"} Oct 30 03:45:36.993728 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:36.993321 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq" event={"ID":"a72fd6fa-4ac5-4e51-b13c-d2ef6246f027","Type":"ContainerDied","Data":"a03dc815b43cdbb313b9591556d1c11a48fa2c046db49a72997fa3a574a4a476"} Oct 30 03:45:36.993728 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:36.993337 5882 scope.go:122] "RemoveContainer" containerID="f853c7844b211a270ead7373d97a938981c69a1bd6b4338cf2af1a1931159139" Oct 30 03:45:37.002494 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:37.002479 5882 scope.go:122] "RemoveContainer" containerID="f853c7844b211a270ead7373d97a938981c69a1bd6b4338cf2af1a1931159139" Oct 30 03:45:37.002778 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:45:37.002760 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f853c7844b211a270ead7373d97a938981c69a1bd6b4338cf2af1a1931159139\": container with ID starting with f853c7844b211a270ead7373d97a938981c69a1bd6b4338cf2af1a1931159139 not found: ID does not exist" containerID="f853c7844b211a270ead7373d97a938981c69a1bd6b4338cf2af1a1931159139" Oct 30 03:45:37.002830 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:37.002785 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f853c7844b211a270ead7373d97a938981c69a1bd6b4338cf2af1a1931159139"} err="failed to get container status \"f853c7844b211a270ead7373d97a938981c69a1bd6b4338cf2af1a1931159139\": rpc error: code = NotFound desc = could not find container \"f853c7844b211a270ead7373d97a938981c69a1bd6b4338cf2af1a1931159139\": container with ID starting with f853c7844b211a270ead7373d97a938981c69a1bd6b4338cf2af1a1931159139 not found: ID does not exist" Oct 30 03:45:37.041503 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:37.041473 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq"] Oct 30 03:45:37.052996 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:37.052972 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-dn9fq"] Oct 30 03:45:38.009044 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:45:38.009011 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a72fd6fa-4ac5-4e51-b13c-d2ef6246f027" path="/var/lib/kubelet/pods/a72fd6fa-4ac5-4e51-b13c-d2ef6246f027/volumes" Oct 30 03:45:53.942409 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:45:53.942373 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:46:23.310031 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:23.309945 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9"] Oct 30 03:46:23.310427 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:23.310337 5882 cpu_manager.go:414] "containerMap: removing container" podUID="a72fd6fa-4ac5-4e51-b13c-d2ef6246f027" containerName="ta-container" Oct 30 03:46:23.310427 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:23.310348 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a72fd6fa-4ac5-4e51-b13c-d2ef6246f027" containerName="ta-container" Oct 30 03:46:23.310427 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:23.310413 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a72fd6fa-4ac5-4e51-b13c-d2ef6246f027" containerName="ta-container" Oct 30 03:46:23.312995 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:23.312974 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" Oct 30 03:46:23.341329 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:23.341306 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9"] Oct 30 03:46:23.422972 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:23.422942 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-ta-internal\") pod \"data-science-collector-targetallocator-df9b59866-lhxf9\" (UID: \"4dfd0c0b-7334-40f8-b67d-16da1250c7dc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" Oct 30 03:46:23.423131 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:23.423000 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgzrx\" (UniqueName: \"kubernetes.io/projected/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-kube-api-access-lgzrx\") pod \"data-science-collector-targetallocator-df9b59866-lhxf9\" (UID: \"4dfd0c0b-7334-40f8-b67d-16da1250c7dc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" Oct 30 03:46:23.423186 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:23.423124 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-df9b59866-lhxf9\" (UID: \"4dfd0c0b-7334-40f8-b67d-16da1250c7dc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" Oct 30 03:46:23.524324 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:23.524288 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-df9b59866-lhxf9\" (UID: \"4dfd0c0b-7334-40f8-b67d-16da1250c7dc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" Oct 30 03:46:23.524481 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:23.524329 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-ta-internal\") pod \"data-science-collector-targetallocator-df9b59866-lhxf9\" (UID: \"4dfd0c0b-7334-40f8-b67d-16da1250c7dc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" Oct 30 03:46:23.524481 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:23.524368 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgzrx\" (UniqueName: \"kubernetes.io/projected/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-kube-api-access-lgzrx\") pod \"data-science-collector-targetallocator-df9b59866-lhxf9\" (UID: \"4dfd0c0b-7334-40f8-b67d-16da1250c7dc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" Oct 30 03:46:23.525053 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:23.525028 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-ta-internal\") pod \"data-science-collector-targetallocator-df9b59866-lhxf9\" (UID: \"4dfd0c0b-7334-40f8-b67d-16da1250c7dc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" Oct 30 03:46:23.526857 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:23.526830 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-df9b59866-lhxf9\" (UID: \"4dfd0c0b-7334-40f8-b67d-16da1250c7dc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" Oct 30 03:46:23.546997 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:23.546971 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgzrx\" (UniqueName: \"kubernetes.io/projected/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-kube-api-access-lgzrx\") pod \"data-science-collector-targetallocator-df9b59866-lhxf9\" (UID: \"4dfd0c0b-7334-40f8-b67d-16da1250c7dc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" Oct 30 03:46:23.624016 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:23.623947 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" Oct 30 03:46:23.911544 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:23.911470 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9"] Oct 30 03:46:23.915281 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:46:23.915253 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4dfd0c0b_7334_40f8_b67d_16da1250c7dc.slice/crio-ded42e09ad6dac34c64c2401d850c7e8339ab6954d02e02cb01f87c620cd7359 WatchSource:0}: Error finding container ded42e09ad6dac34c64c2401d850c7e8339ab6954d02e02cb01f87c620cd7359: Status 404 returned error can't find the container with id ded42e09ad6dac34c64c2401d850c7e8339ab6954d02e02cb01f87c620cd7359 Oct 30 03:46:23.921059 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:23.920202 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:46:23.921059 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:23.920286 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:46:23.944365 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:46:23.944334 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:46:24.200236 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:24.200148 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" event={"ID":"4dfd0c0b-7334-40f8-b67d-16da1250c7dc","Type":"ContainerStarted","Data":"3fbb8663f0d1a490d18cad5bdba309be4009c412bb7c3538687ebf289b22aa10"} Oct 30 03:46:24.200236 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:24.200188 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" event={"ID":"4dfd0c0b-7334-40f8-b67d-16da1250c7dc","Type":"ContainerStarted","Data":"ded42e09ad6dac34c64c2401d850c7e8339ab6954d02e02cb01f87c620cd7359"} Oct 30 03:46:24.200453 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:24.200360 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" Oct 30 03:46:24.201604 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:24.201580 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" podUID="4dfd0c0b-7334-40f8-b67d-16da1250c7dc" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.60:8080/readyz\": dial tcp 10.134.0.60:8080: connect: connection refused" Oct 30 03:46:24.240809 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:24.240764 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" podStartSLOduration=1.240750784 podStartE2EDuration="1.240750784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:24.239794312 +0000 UTC m=+3896.354109287" watchObservedRunningTime="2026-10-30 03:46:24.240750784 +0000 UTC m=+3896.355065752" Oct 30 03:46:25.205551 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:25.205520 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" Oct 30 03:46:25.278340 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:25.277640 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9"] Oct 30 03:46:25.278340 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:25.277926 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" podUID="3679fb4c-772b-46a5-ab2e-e5105a819b6f" containerName="ta-container" containerID="cri-o://1c4f73ad07328347be0b77086db7d5a652377ae42716503036d0efb9137da5ac" gracePeriod=30 Oct 30 03:46:25.528848 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:25.528820 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" Oct 30 03:46:25.640472 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:25.640444 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3679fb4c-772b-46a5-ab2e-e5105a819b6f-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3679fb4c-772b-46a5-ab2e-e5105a819b6f-data-science-collector-ta-server-cert\") pod \"3679fb4c-772b-46a5-ab2e-e5105a819b6f\" (UID: \"3679fb4c-772b-46a5-ab2e-e5105a819b6f\") " Oct 30 03:46:25.640691 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:25.640545 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3679fb4c-772b-46a5-ab2e-e5105a819b6f-kube-api-access-v29q9\" (UniqueName: \"kubernetes.io/projected/3679fb4c-772b-46a5-ab2e-e5105a819b6f-kube-api-access-v29q9\") pod \"3679fb4c-772b-46a5-ab2e-e5105a819b6f\" (UID: \"3679fb4c-772b-46a5-ab2e-e5105a819b6f\") " Oct 30 03:46:25.640691 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:25.640585 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3679fb4c-772b-46a5-ab2e-e5105a819b6f-ta-internal\" (UniqueName: \"kubernetes.io/configmap/3679fb4c-772b-46a5-ab2e-e5105a819b6f-ta-internal\") pod \"3679fb4c-772b-46a5-ab2e-e5105a819b6f\" (UID: \"3679fb4c-772b-46a5-ab2e-e5105a819b6f\") " Oct 30 03:46:25.641027 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:25.640998 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3679fb4c-772b-46a5-ab2e-e5105a819b6f-ta-internal" pod "3679fb4c-772b-46a5-ab2e-e5105a819b6f" (UID: "3679fb4c-772b-46a5-ab2e-e5105a819b6f"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:46:25.642576 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:25.642546 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3679fb4c-772b-46a5-ab2e-e5105a819b6f-data-science-collector-ta-server-cert" pod "3679fb4c-772b-46a5-ab2e-e5105a819b6f" (UID: "3679fb4c-772b-46a5-ab2e-e5105a819b6f"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:46:25.642576 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:25.642567 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3679fb4c-772b-46a5-ab2e-e5105a819b6f-kube-api-access-v29q9" pod "3679fb4c-772b-46a5-ab2e-e5105a819b6f" (UID: "3679fb4c-772b-46a5-ab2e-e5105a819b6f"). InnerVolumeSpecName "kube-api-access-v29q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:46:25.741833 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:25.741793 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/3679fb4c-772b-46a5-ab2e-e5105a819b6f-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:46:25.741833 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:25.741831 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3679fb4c-772b-46a5-ab2e-e5105a819b6f-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:46:25.741833 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:25.741842 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v29q9\" (UniqueName: \"kubernetes.io/projected/3679fb4c-772b-46a5-ab2e-e5105a819b6f-kube-api-access-v29q9\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:46:26.210064 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:26.210033 5882 generic.go:356] "Generic (PLEG): container finished" podID="3679fb4c-772b-46a5-ab2e-e5105a819b6f" containerID="1c4f73ad07328347be0b77086db7d5a652377ae42716503036d0efb9137da5ac" exitCode=0 Oct 30 03:46:26.210523 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:26.210076 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" Oct 30 03:46:26.210523 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:26.210124 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" event={"ID":"3679fb4c-772b-46a5-ab2e-e5105a819b6f","Type":"ContainerDied","Data":"1c4f73ad07328347be0b77086db7d5a652377ae42716503036d0efb9137da5ac"} Oct 30 03:46:26.210523 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:26.210167 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9" event={"ID":"3679fb4c-772b-46a5-ab2e-e5105a819b6f","Type":"ContainerDied","Data":"39f248936e08fa351c1f2ca4b190ab4a0976232c2c6fb011516e7e0241518b24"} Oct 30 03:46:26.210523 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:26.210186 5882 scope.go:122] "RemoveContainer" containerID="1c4f73ad07328347be0b77086db7d5a652377ae42716503036d0efb9137da5ac" Oct 30 03:46:26.218635 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:26.218603 5882 scope.go:122] "RemoveContainer" containerID="1c4f73ad07328347be0b77086db7d5a652377ae42716503036d0efb9137da5ac" Oct 30 03:46:26.218885 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:46:26.218869 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c4f73ad07328347be0b77086db7d5a652377ae42716503036d0efb9137da5ac\": container with ID starting with 1c4f73ad07328347be0b77086db7d5a652377ae42716503036d0efb9137da5ac not found: ID does not exist" containerID="1c4f73ad07328347be0b77086db7d5a652377ae42716503036d0efb9137da5ac" Oct 30 03:46:26.218951 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:26.218902 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c4f73ad07328347be0b77086db7d5a652377ae42716503036d0efb9137da5ac"} err="failed to get container status \"1c4f73ad07328347be0b77086db7d5a652377ae42716503036d0efb9137da5ac\": rpc error: code = NotFound desc = could not find container \"1c4f73ad07328347be0b77086db7d5a652377ae42716503036d0efb9137da5ac\": container with ID starting with 1c4f73ad07328347be0b77086db7d5a652377ae42716503036d0efb9137da5ac not found: ID does not exist" Oct 30 03:46:26.257784 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:26.257751 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9"] Oct 30 03:46:26.269520 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:26.269457 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-699q9"] Oct 30 03:46:28.009965 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:28.009930 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3679fb4c-772b-46a5-ab2e-e5105a819b6f" path="/var/lib/kubelet/pods/3679fb4c-772b-46a5-ab2e-e5105a819b6f/volumes" Oct 30 03:46:28.372822 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:28.372663 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:46:28.387423 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:28.384444 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:46:29.517762 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:29.517728 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t"] Oct 30 03:46:29.518198 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:29.518129 5882 cpu_manager.go:414] "containerMap: removing container" podUID="3679fb4c-772b-46a5-ab2e-e5105a819b6f" containerName="ta-container" Oct 30 03:46:29.518198 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:29.518141 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3679fb4c-772b-46a5-ab2e-e5105a819b6f" containerName="ta-container" Oct 30 03:46:29.518285 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:29.518232 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3679fb4c-772b-46a5-ab2e-e5105a819b6f" containerName="ta-container" Oct 30 03:46:29.522742 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:29.522722 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" Oct 30 03:46:29.552814 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:29.552790 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t"] Oct 30 03:46:29.575513 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:29.575487 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-b6f46c55d-s4t2t\" (UID: \"c82bed6c-71a3-40ec-a3c9-79cce41d0ea7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" Oct 30 03:46:29.575650 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:29.575530 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-ta-internal\") pod \"data-science-collector-targetallocator-b6f46c55d-s4t2t\" (UID: \"c82bed6c-71a3-40ec-a3c9-79cce41d0ea7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" Oct 30 03:46:29.575650 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:29.575631 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j82fw\" (UniqueName: \"kubernetes.io/projected/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-kube-api-access-j82fw\") pod \"data-science-collector-targetallocator-b6f46c55d-s4t2t\" (UID: \"c82bed6c-71a3-40ec-a3c9-79cce41d0ea7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" Oct 30 03:46:29.676850 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:29.676817 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-b6f46c55d-s4t2t\" (UID: \"c82bed6c-71a3-40ec-a3c9-79cce41d0ea7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" Oct 30 03:46:29.676996 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:29.676863 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-ta-internal\") pod \"data-science-collector-targetallocator-b6f46c55d-s4t2t\" (UID: \"c82bed6c-71a3-40ec-a3c9-79cce41d0ea7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" Oct 30 03:46:29.676996 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:29.676913 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j82fw\" (UniqueName: \"kubernetes.io/projected/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-kube-api-access-j82fw\") pod \"data-science-collector-targetallocator-b6f46c55d-s4t2t\" (UID: \"c82bed6c-71a3-40ec-a3c9-79cce41d0ea7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" Oct 30 03:46:29.677410 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:29.677392 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-ta-internal\") pod \"data-science-collector-targetallocator-b6f46c55d-s4t2t\" (UID: \"c82bed6c-71a3-40ec-a3c9-79cce41d0ea7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" Oct 30 03:46:29.679430 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:29.679409 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-b6f46c55d-s4t2t\" (UID: \"c82bed6c-71a3-40ec-a3c9-79cce41d0ea7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" Oct 30 03:46:29.699572 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:29.699546 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j82fw\" (UniqueName: \"kubernetes.io/projected/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-kube-api-access-j82fw\") pod \"data-science-collector-targetallocator-b6f46c55d-s4t2t\" (UID: \"c82bed6c-71a3-40ec-a3c9-79cce41d0ea7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" Oct 30 03:46:29.834014 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:29.833930 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" Oct 30 03:46:30.124750 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:30.124680 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t"] Oct 30 03:46:30.128005 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:46:30.127972 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc82bed6c_71a3_40ec_a3c9_79cce41d0ea7.slice/crio-f894df67f1f6e84cc9b3b2a0ba5800cf3ef6e0c396843908350444f356b41794 WatchSource:0}: Error finding container f894df67f1f6e84cc9b3b2a0ba5800cf3ef6e0c396843908350444f356b41794: Status 404 returned error can't find the container with id f894df67f1f6e84cc9b3b2a0ba5800cf3ef6e0c396843908350444f356b41794 Oct 30 03:46:30.130344 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:30.130304 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:46:30.130417 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:30.130381 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:46:30.230869 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:30.230840 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" event={"ID":"c82bed6c-71a3-40ec-a3c9-79cce41d0ea7","Type":"ContainerStarted","Data":"552485cdbc1a532c151a4d1f2415af99dc38e43b5a6b67289273d42652eefc65"} Oct 30 03:46:30.231036 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:30.230878 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" event={"ID":"c82bed6c-71a3-40ec-a3c9-79cce41d0ea7","Type":"ContainerStarted","Data":"f894df67f1f6e84cc9b3b2a0ba5800cf3ef6e0c396843908350444f356b41794"} Oct 30 03:46:30.231154 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:30.231128 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" Oct 30 03:46:30.231916 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:30.231894 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" podUID="c82bed6c-71a3-40ec-a3c9-79cce41d0ea7" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.61:8080/readyz\": dial tcp 10.134.0.61:8080: connect: connection refused" Oct 30 03:46:30.271759 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:30.271708 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" podStartSLOduration=1.271695947 podStartE2EDuration="1.271695947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:30.27018425 +0000 UTC m=+3902.384499225" watchObservedRunningTime="2026-10-30 03:46:30.271695947 +0000 UTC m=+3902.386010921" Oct 30 03:46:31.236814 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:31.236783 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" Oct 30 03:46:31.321454 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:31.321424 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9"] Oct 30 03:46:31.321651 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:31.321629 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" podUID="4dfd0c0b-7334-40f8-b67d-16da1250c7dc" containerName="ta-container" containerID="cri-o://3fbb8663f0d1a490d18cad5bdba309be4009c412bb7c3538687ebf289b22aa10" gracePeriod=30 Oct 30 03:46:31.566288 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:31.566260 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" Oct 30 03:46:31.696097 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:31.696064 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-data-science-collector-ta-server-cert\") pod \"4dfd0c0b-7334-40f8-b67d-16da1250c7dc\" (UID: \"4dfd0c0b-7334-40f8-b67d-16da1250c7dc\") " Oct 30 03:46:31.696097 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:31.696103 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-ta-internal\" (UniqueName: \"kubernetes.io/configmap/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-ta-internal\") pod \"4dfd0c0b-7334-40f8-b67d-16da1250c7dc\" (UID: \"4dfd0c0b-7334-40f8-b67d-16da1250c7dc\") " Oct 30 03:46:31.696310 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:31.696179 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-kube-api-access-lgzrx\" (UniqueName: \"kubernetes.io/projected/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-kube-api-access-lgzrx\") pod \"4dfd0c0b-7334-40f8-b67d-16da1250c7dc\" (UID: \"4dfd0c0b-7334-40f8-b67d-16da1250c7dc\") " Oct 30 03:46:31.696518 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:31.696492 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-ta-internal" pod "4dfd0c0b-7334-40f8-b67d-16da1250c7dc" (UID: "4dfd0c0b-7334-40f8-b67d-16da1250c7dc"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:46:31.698096 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:31.698064 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-data-science-collector-ta-server-cert" pod "4dfd0c0b-7334-40f8-b67d-16da1250c7dc" (UID: "4dfd0c0b-7334-40f8-b67d-16da1250c7dc"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:46:31.698208 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:31.698192 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-kube-api-access-lgzrx" pod "4dfd0c0b-7334-40f8-b67d-16da1250c7dc" (UID: "4dfd0c0b-7334-40f8-b67d-16da1250c7dc"). InnerVolumeSpecName "kube-api-access-lgzrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:46:31.797411 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:31.797348 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:46:31.797411 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:31.797373 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:46:31.797411 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:31.797385 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgzrx\" (UniqueName: \"kubernetes.io/projected/4dfd0c0b-7334-40f8-b67d-16da1250c7dc-kube-api-access-lgzrx\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:46:32.241141 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:32.241059 5882 generic.go:356] "Generic (PLEG): container finished" podID="4dfd0c0b-7334-40f8-b67d-16da1250c7dc" containerID="3fbb8663f0d1a490d18cad5bdba309be4009c412bb7c3538687ebf289b22aa10" exitCode=0 Oct 30 03:46:32.241141 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:32.241103 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" Oct 30 03:46:32.241588 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:32.241150 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" event={"ID":"4dfd0c0b-7334-40f8-b67d-16da1250c7dc","Type":"ContainerDied","Data":"3fbb8663f0d1a490d18cad5bdba309be4009c412bb7c3538687ebf289b22aa10"} Oct 30 03:46:32.241588 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:32.241190 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9" event={"ID":"4dfd0c0b-7334-40f8-b67d-16da1250c7dc","Type":"ContainerDied","Data":"ded42e09ad6dac34c64c2401d850c7e8339ab6954d02e02cb01f87c620cd7359"} Oct 30 03:46:32.241588 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:32.241207 5882 scope.go:122] "RemoveContainer" containerID="3fbb8663f0d1a490d18cad5bdba309be4009c412bb7c3538687ebf289b22aa10" Oct 30 03:46:32.249709 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:32.249687 5882 scope.go:122] "RemoveContainer" containerID="3fbb8663f0d1a490d18cad5bdba309be4009c412bb7c3538687ebf289b22aa10" Oct 30 03:46:32.249964 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:46:32.249940 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fbb8663f0d1a490d18cad5bdba309be4009c412bb7c3538687ebf289b22aa10\": container with ID starting with 3fbb8663f0d1a490d18cad5bdba309be4009c412bb7c3538687ebf289b22aa10 not found: ID does not exist" containerID="3fbb8663f0d1a490d18cad5bdba309be4009c412bb7c3538687ebf289b22aa10" Oct 30 03:46:32.250017 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:32.249973 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fbb8663f0d1a490d18cad5bdba309be4009c412bb7c3538687ebf289b22aa10"} err="failed to get container status \"3fbb8663f0d1a490d18cad5bdba309be4009c412bb7c3538687ebf289b22aa10\": rpc error: code = NotFound desc = could not find container \"3fbb8663f0d1a490d18cad5bdba309be4009c412bb7c3538687ebf289b22aa10\": container with ID starting with 3fbb8663f0d1a490d18cad5bdba309be4009c412bb7c3538687ebf289b22aa10 not found: ID does not exist" Oct 30 03:46:32.289829 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:32.289801 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9"] Oct 30 03:46:32.301196 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:32.301174 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-lhxf9"] Oct 30 03:46:34.010197 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:34.010167 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dfd0c0b-7334-40f8-b67d-16da1250c7dc" path="/var/lib/kubelet/pods/4dfd0c0b-7334-40f8-b67d-16da1250c7dc/volumes" Oct 30 03:46:36.196102 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:36.196070 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h"] Oct 30 03:46:36.196701 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:36.196680 5882 cpu_manager.go:414] "containerMap: removing container" podUID="4dfd0c0b-7334-40f8-b67d-16da1250c7dc" containerName="ta-container" Oct 30 03:46:36.196808 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:36.196702 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dfd0c0b-7334-40f8-b67d-16da1250c7dc" containerName="ta-container" Oct 30 03:46:36.196861 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:36.196809 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dfd0c0b-7334-40f8-b67d-16da1250c7dc" containerName="ta-container" Oct 30 03:46:36.199872 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:36.199852 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" Oct 30 03:46:36.226317 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:36.226281 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h"] Oct 30 03:46:36.336075 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:36.336042 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/527bc623-e6ed-41e9-92ac-8ffea6366fc2-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-744cffff95-zjc5h\" (UID: \"527bc623-e6ed-41e9-92ac-8ffea6366fc2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" Oct 30 03:46:36.336240 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:36.336098 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljxpr\" (UniqueName: \"kubernetes.io/projected/527bc623-e6ed-41e9-92ac-8ffea6366fc2-kube-api-access-ljxpr\") pod \"data-science-collector-targetallocator-744cffff95-zjc5h\" (UID: \"527bc623-e6ed-41e9-92ac-8ffea6366fc2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" Oct 30 03:46:36.336240 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:36.336217 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/527bc623-e6ed-41e9-92ac-8ffea6366fc2-ta-internal\") pod \"data-science-collector-targetallocator-744cffff95-zjc5h\" (UID: \"527bc623-e6ed-41e9-92ac-8ffea6366fc2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" Oct 30 03:46:36.436924 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:36.436885 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/527bc623-e6ed-41e9-92ac-8ffea6366fc2-ta-internal\") pod \"data-science-collector-targetallocator-744cffff95-zjc5h\" (UID: \"527bc623-e6ed-41e9-92ac-8ffea6366fc2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" Oct 30 03:46:36.437104 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:36.436957 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/527bc623-e6ed-41e9-92ac-8ffea6366fc2-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-744cffff95-zjc5h\" (UID: \"527bc623-e6ed-41e9-92ac-8ffea6366fc2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" Oct 30 03:46:36.437104 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:36.437003 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljxpr\" (UniqueName: \"kubernetes.io/projected/527bc623-e6ed-41e9-92ac-8ffea6366fc2-kube-api-access-ljxpr\") pod \"data-science-collector-targetallocator-744cffff95-zjc5h\" (UID: \"527bc623-e6ed-41e9-92ac-8ffea6366fc2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" Oct 30 03:46:36.437530 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:36.437509 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/527bc623-e6ed-41e9-92ac-8ffea6366fc2-ta-internal\") pod \"data-science-collector-targetallocator-744cffff95-zjc5h\" (UID: \"527bc623-e6ed-41e9-92ac-8ffea6366fc2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" Oct 30 03:46:36.439444 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:36.439420 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/527bc623-e6ed-41e9-92ac-8ffea6366fc2-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-744cffff95-zjc5h\" (UID: \"527bc623-e6ed-41e9-92ac-8ffea6366fc2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" Oct 30 03:46:36.459564 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:36.459508 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljxpr\" (UniqueName: \"kubernetes.io/projected/527bc623-e6ed-41e9-92ac-8ffea6366fc2-kube-api-access-ljxpr\") pod \"data-science-collector-targetallocator-744cffff95-zjc5h\" (UID: \"527bc623-e6ed-41e9-92ac-8ffea6366fc2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" Oct 30 03:46:36.510598 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:36.510570 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" Oct 30 03:46:36.800054 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:36.800027 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h"] Oct 30 03:46:36.802871 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:46:36.802844 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod527bc623_e6ed_41e9_92ac_8ffea6366fc2.slice/crio-3079ca81d77ff728bf4cb41731ebefe0e081cd0e46ae6ee22382e8f133595264 WatchSource:0}: Error finding container 3079ca81d77ff728bf4cb41731ebefe0e081cd0e46ae6ee22382e8f133595264: Status 404 returned error can't find the container with id 3079ca81d77ff728bf4cb41731ebefe0e081cd0e46ae6ee22382e8f133595264 Oct 30 03:46:36.804913 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:36.804883 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:46:36.805003 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:36.804938 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:46:37.265087 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:37.265043 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" event={"ID":"527bc623-e6ed-41e9-92ac-8ffea6366fc2","Type":"ContainerStarted","Data":"c646e3e633d1d7e664f19437cb7063b9f679fdeda8e981224d515e71f09d363a"} Oct 30 03:46:37.265087 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:37.265089 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" event={"ID":"527bc623-e6ed-41e9-92ac-8ffea6366fc2","Type":"ContainerStarted","Data":"3079ca81d77ff728bf4cb41731ebefe0e081cd0e46ae6ee22382e8f133595264"} Oct 30 03:46:37.265531 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:37.265107 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" Oct 30 03:46:37.266902 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:37.266873 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" Oct 30 03:46:37.306640 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:37.306581 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" podStartSLOduration=1.306567386 podStartE2EDuration="1.306567386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:37.304555726 +0000 UTC m=+3909.418870704" watchObservedRunningTime="2026-10-30 03:46:37.306567386 +0000 UTC m=+3909.420882361" Oct 30 03:46:37.379711 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:37.379680 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t"] Oct 30 03:46:37.379925 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:37.379889 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" podUID="c82bed6c-71a3-40ec-a3c9-79cce41d0ea7" containerName="ta-container" containerID="cri-o://552485cdbc1a532c151a4d1f2415af99dc38e43b5a6b67289273d42652eefc65" gracePeriod=30 Oct 30 03:46:37.619934 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:37.619910 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" Oct 30 03:46:37.749588 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:37.749553 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-kube-api-access-j82fw\" (UniqueName: \"kubernetes.io/projected/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-kube-api-access-j82fw\") pod \"c82bed6c-71a3-40ec-a3c9-79cce41d0ea7\" (UID: \"c82bed6c-71a3-40ec-a3c9-79cce41d0ea7\") " Oct 30 03:46:37.749783 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:37.749667 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-ta-internal\" (UniqueName: \"kubernetes.io/configmap/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-ta-internal\") pod \"c82bed6c-71a3-40ec-a3c9-79cce41d0ea7\" (UID: \"c82bed6c-71a3-40ec-a3c9-79cce41d0ea7\") " Oct 30 03:46:37.749783 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:37.749744 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-data-science-collector-ta-server-cert\") pod \"c82bed6c-71a3-40ec-a3c9-79cce41d0ea7\" (UID: \"c82bed6c-71a3-40ec-a3c9-79cce41d0ea7\") " Oct 30 03:46:37.750015 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:37.749989 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-ta-internal" pod "c82bed6c-71a3-40ec-a3c9-79cce41d0ea7" (UID: "c82bed6c-71a3-40ec-a3c9-79cce41d0ea7"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:46:37.751724 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:37.751702 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-kube-api-access-j82fw" pod "c82bed6c-71a3-40ec-a3c9-79cce41d0ea7" (UID: "c82bed6c-71a3-40ec-a3c9-79cce41d0ea7"). InnerVolumeSpecName "kube-api-access-j82fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:46:37.751781 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:37.751726 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-data-science-collector-ta-server-cert" pod "c82bed6c-71a3-40ec-a3c9-79cce41d0ea7" (UID: "c82bed6c-71a3-40ec-a3c9-79cce41d0ea7"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:46:37.851402 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:37.851320 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:46:37.851402 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:37.851351 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j82fw\" (UniqueName: \"kubernetes.io/projected/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-kube-api-access-j82fw\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:46:37.851402 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:37.851362 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:46:38.271204 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:38.271172 5882 generic.go:356] "Generic (PLEG): container finished" podID="c82bed6c-71a3-40ec-a3c9-79cce41d0ea7" containerID="552485cdbc1a532c151a4d1f2415af99dc38e43b5a6b67289273d42652eefc65" exitCode=0 Oct 30 03:46:38.271667 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:38.271215 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" Oct 30 03:46:38.271667 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:38.271254 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" event={"ID":"c82bed6c-71a3-40ec-a3c9-79cce41d0ea7","Type":"ContainerDied","Data":"552485cdbc1a532c151a4d1f2415af99dc38e43b5a6b67289273d42652eefc65"} Oct 30 03:46:38.271667 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:38.271296 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t" event={"ID":"c82bed6c-71a3-40ec-a3c9-79cce41d0ea7","Type":"ContainerDied","Data":"f894df67f1f6e84cc9b3b2a0ba5800cf3ef6e0c396843908350444f356b41794"} Oct 30 03:46:38.271667 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:38.271312 5882 scope.go:122] "RemoveContainer" containerID="552485cdbc1a532c151a4d1f2415af99dc38e43b5a6b67289273d42652eefc65" Oct 30 03:46:38.279559 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:38.279541 5882 scope.go:122] "RemoveContainer" containerID="552485cdbc1a532c151a4d1f2415af99dc38e43b5a6b67289273d42652eefc65" Oct 30 03:46:38.279854 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:46:38.279832 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552485cdbc1a532c151a4d1f2415af99dc38e43b5a6b67289273d42652eefc65\": container with ID starting with 552485cdbc1a532c151a4d1f2415af99dc38e43b5a6b67289273d42652eefc65 not found: ID does not exist" containerID="552485cdbc1a532c151a4d1f2415af99dc38e43b5a6b67289273d42652eefc65" Oct 30 03:46:38.279951 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:38.279862 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552485cdbc1a532c151a4d1f2415af99dc38e43b5a6b67289273d42652eefc65"} err="failed to get container status \"552485cdbc1a532c151a4d1f2415af99dc38e43b5a6b67289273d42652eefc65\": rpc error: code = NotFound desc = could not find container \"552485cdbc1a532c151a4d1f2415af99dc38e43b5a6b67289273d42652eefc65\": container with ID starting with 552485cdbc1a532c151a4d1f2415af99dc38e43b5a6b67289273d42652eefc65 not found: ID does not exist" Oct 30 03:46:38.319644 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:38.319608 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t"] Oct 30 03:46:38.331174 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:38.331142 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-s4t2t"] Oct 30 03:46:39.350768 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:39.350734 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg"] Oct 30 03:46:39.351164 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:39.351153 5882 cpu_manager.go:414] "containerMap: removing container" podUID="c82bed6c-71a3-40ec-a3c9-79cce41d0ea7" containerName="ta-container" Oct 30 03:46:39.351205 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:39.351165 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c82bed6c-71a3-40ec-a3c9-79cce41d0ea7" containerName="ta-container" Oct 30 03:46:39.351255 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:39.351231 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c82bed6c-71a3-40ec-a3c9-79cce41d0ea7" containerName="ta-container" Oct 30 03:46:39.355745 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:39.355720 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" Oct 30 03:46:39.388667 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:39.388639 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg"] Oct 30 03:46:39.467129 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:39.467100 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/178c0ce3-fa51-4487-abeb-78c6496df192-ta-internal\") pod \"data-science-collector-targetallocator-6d48bd4dcc-jppfg\" (UID: \"178c0ce3-fa51-4487-abeb-78c6496df192\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" Oct 30 03:46:39.467309 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:39.467139 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/178c0ce3-fa51-4487-abeb-78c6496df192-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6d48bd4dcc-jppfg\" (UID: \"178c0ce3-fa51-4487-abeb-78c6496df192\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" Oct 30 03:46:39.467309 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:39.467168 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2hcv\" (UniqueName: \"kubernetes.io/projected/178c0ce3-fa51-4487-abeb-78c6496df192-kube-api-access-w2hcv\") pod \"data-science-collector-targetallocator-6d48bd4dcc-jppfg\" (UID: \"178c0ce3-fa51-4487-abeb-78c6496df192\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" Oct 30 03:46:39.568015 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:39.567981 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/178c0ce3-fa51-4487-abeb-78c6496df192-ta-internal\") pod \"data-science-collector-targetallocator-6d48bd4dcc-jppfg\" (UID: \"178c0ce3-fa51-4487-abeb-78c6496df192\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" Oct 30 03:46:39.568015 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:39.568018 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/178c0ce3-fa51-4487-abeb-78c6496df192-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6d48bd4dcc-jppfg\" (UID: \"178c0ce3-fa51-4487-abeb-78c6496df192\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" Oct 30 03:46:39.568250 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:39.568038 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2hcv\" (UniqueName: \"kubernetes.io/projected/178c0ce3-fa51-4487-abeb-78c6496df192-kube-api-access-w2hcv\") pod \"data-science-collector-targetallocator-6d48bd4dcc-jppfg\" (UID: \"178c0ce3-fa51-4487-abeb-78c6496df192\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" Oct 30 03:46:39.568747 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:39.568721 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/178c0ce3-fa51-4487-abeb-78c6496df192-ta-internal\") pod \"data-science-collector-targetallocator-6d48bd4dcc-jppfg\" (UID: \"178c0ce3-fa51-4487-abeb-78c6496df192\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" Oct 30 03:46:39.570426 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:39.570408 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/178c0ce3-fa51-4487-abeb-78c6496df192-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6d48bd4dcc-jppfg\" (UID: \"178c0ce3-fa51-4487-abeb-78c6496df192\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" Oct 30 03:46:39.591282 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:39.591250 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2hcv\" (UniqueName: \"kubernetes.io/projected/178c0ce3-fa51-4487-abeb-78c6496df192-kube-api-access-w2hcv\") pod \"data-science-collector-targetallocator-6d48bd4dcc-jppfg\" (UID: \"178c0ce3-fa51-4487-abeb-78c6496df192\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" Oct 30 03:46:39.667286 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:39.667195 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" Oct 30 03:46:39.958465 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:39.958382 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg"] Oct 30 03:46:39.961492 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:46:39.961462 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod178c0ce3_fa51_4487_abeb_78c6496df192.slice/crio-b0739d05a5f7b33f8248a8045f42239582a6ce86daa013dcee24b56f691863a0 WatchSource:0}: Error finding container b0739d05a5f7b33f8248a8045f42239582a6ce86daa013dcee24b56f691863a0: Status 404 returned error can't find the container with id b0739d05a5f7b33f8248a8045f42239582a6ce86daa013dcee24b56f691863a0 Oct 30 03:46:39.963666 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:39.963636 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:46:39.963765 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:39.963703 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:46:40.009537 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:40.009471 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c82bed6c-71a3-40ec-a3c9-79cce41d0ea7" path="/var/lib/kubelet/pods/c82bed6c-71a3-40ec-a3c9-79cce41d0ea7/volumes" Oct 30 03:46:40.283609 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:40.283577 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" event={"ID":"178c0ce3-fa51-4487-abeb-78c6496df192","Type":"ContainerStarted","Data":"a2f3bac04be1812909c114d6bf0588b95c41472b6b08d13d6e1acadec22f458b"} Oct 30 03:46:40.283803 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:40.283643 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" event={"ID":"178c0ce3-fa51-4487-abeb-78c6496df192","Type":"ContainerStarted","Data":"b0739d05a5f7b33f8248a8045f42239582a6ce86daa013dcee24b56f691863a0"} Oct 30 03:46:40.283803 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:40.283779 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" Oct 30 03:46:40.284656 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:40.284612 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" podUID="178c0ce3-fa51-4487-abeb-78c6496df192" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.63:8080/readyz\": dial tcp 10.134.0.63:8080: connect: connection refused" Oct 30 03:46:40.328828 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:40.328783 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" podStartSLOduration=1.328771263 podStartE2EDuration="1.328771263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:40.328342668 +0000 UTC m=+3912.442657643" watchObservedRunningTime="2026-10-30 03:46:40.328771263 +0000 UTC m=+3912.443086237" Oct 30 03:46:41.288842 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:41.288810 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" Oct 30 03:46:41.360592 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:41.360562 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h"] Oct 30 03:46:41.360771 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:41.360741 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" podUID="527bc623-e6ed-41e9-92ac-8ffea6366fc2" containerName="ta-container" containerID="cri-o://c646e3e633d1d7e664f19437cb7063b9f679fdeda8e981224d515e71f09d363a" gracePeriod=30 Oct 30 03:46:41.601164 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:41.601143 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" Oct 30 03:46:41.689708 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:41.689673 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/527bc623-e6ed-41e9-92ac-8ffea6366fc2-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/527bc623-e6ed-41e9-92ac-8ffea6366fc2-data-science-collector-ta-server-cert\") pod \"527bc623-e6ed-41e9-92ac-8ffea6366fc2\" (UID: \"527bc623-e6ed-41e9-92ac-8ffea6366fc2\") " Oct 30 03:46:41.689873 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:41.689723 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/527bc623-e6ed-41e9-92ac-8ffea6366fc2-kube-api-access-ljxpr\" (UniqueName: \"kubernetes.io/projected/527bc623-e6ed-41e9-92ac-8ffea6366fc2-kube-api-access-ljxpr\") pod \"527bc623-e6ed-41e9-92ac-8ffea6366fc2\" (UID: \"527bc623-e6ed-41e9-92ac-8ffea6366fc2\") " Oct 30 03:46:41.689873 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:41.689801 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/527bc623-e6ed-41e9-92ac-8ffea6366fc2-ta-internal\" (UniqueName: \"kubernetes.io/configmap/527bc623-e6ed-41e9-92ac-8ffea6366fc2-ta-internal\") pod \"527bc623-e6ed-41e9-92ac-8ffea6366fc2\" (UID: \"527bc623-e6ed-41e9-92ac-8ffea6366fc2\") " Oct 30 03:46:41.690175 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:41.690153 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/527bc623-e6ed-41e9-92ac-8ffea6366fc2-ta-internal" pod "527bc623-e6ed-41e9-92ac-8ffea6366fc2" (UID: "527bc623-e6ed-41e9-92ac-8ffea6366fc2"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:46:41.691797 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:41.691775 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527bc623-e6ed-41e9-92ac-8ffea6366fc2-kube-api-access-ljxpr" pod "527bc623-e6ed-41e9-92ac-8ffea6366fc2" (UID: "527bc623-e6ed-41e9-92ac-8ffea6366fc2"). InnerVolumeSpecName "kube-api-access-ljxpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:46:41.691855 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:41.691779 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/527bc623-e6ed-41e9-92ac-8ffea6366fc2-data-science-collector-ta-server-cert" pod "527bc623-e6ed-41e9-92ac-8ffea6366fc2" (UID: "527bc623-e6ed-41e9-92ac-8ffea6366fc2"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:46:41.790963 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:41.790930 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljxpr\" (UniqueName: \"kubernetes.io/projected/527bc623-e6ed-41e9-92ac-8ffea6366fc2-kube-api-access-ljxpr\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:46:41.790963 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:41.790959 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/527bc623-e6ed-41e9-92ac-8ffea6366fc2-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:46:41.790963 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:41.790969 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/527bc623-e6ed-41e9-92ac-8ffea6366fc2-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:46:42.293418 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:42.293330 5882 generic.go:356] "Generic (PLEG): container finished" podID="527bc623-e6ed-41e9-92ac-8ffea6366fc2" containerID="c646e3e633d1d7e664f19437cb7063b9f679fdeda8e981224d515e71f09d363a" exitCode=0 Oct 30 03:46:42.293418 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:42.293385 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" Oct 30 03:46:42.293922 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:42.293380 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" event={"ID":"527bc623-e6ed-41e9-92ac-8ffea6366fc2","Type":"ContainerDied","Data":"c646e3e633d1d7e664f19437cb7063b9f679fdeda8e981224d515e71f09d363a"} Oct 30 03:46:42.293922 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:42.293495 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h" event={"ID":"527bc623-e6ed-41e9-92ac-8ffea6366fc2","Type":"ContainerDied","Data":"3079ca81d77ff728bf4cb41731ebefe0e081cd0e46ae6ee22382e8f133595264"} Oct 30 03:46:42.293922 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:42.293512 5882 scope.go:122] "RemoveContainer" containerID="c646e3e633d1d7e664f19437cb7063b9f679fdeda8e981224d515e71f09d363a" Oct 30 03:46:42.301912 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:42.301891 5882 scope.go:122] "RemoveContainer" containerID="c646e3e633d1d7e664f19437cb7063b9f679fdeda8e981224d515e71f09d363a" Oct 30 03:46:42.302197 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:46:42.302178 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c646e3e633d1d7e664f19437cb7063b9f679fdeda8e981224d515e71f09d363a\": container with ID starting with c646e3e633d1d7e664f19437cb7063b9f679fdeda8e981224d515e71f09d363a not found: ID does not exist" containerID="c646e3e633d1d7e664f19437cb7063b9f679fdeda8e981224d515e71f09d363a" Oct 30 03:46:42.302237 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:42.302205 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c646e3e633d1d7e664f19437cb7063b9f679fdeda8e981224d515e71f09d363a"} err="failed to get container status \"c646e3e633d1d7e664f19437cb7063b9f679fdeda8e981224d515e71f09d363a\": rpc error: code = NotFound desc = could not find container \"c646e3e633d1d7e664f19437cb7063b9f679fdeda8e981224d515e71f09d363a\": container with ID starting with c646e3e633d1d7e664f19437cb7063b9f679fdeda8e981224d515e71f09d363a not found: ID does not exist" Oct 30 03:46:42.341556 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:42.341517 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h"] Oct 30 03:46:42.353168 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:42.353143 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-zjc5h"] Oct 30 03:46:44.009873 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:46:44.009838 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="527bc623-e6ed-41e9-92ac-8ffea6366fc2" path="/var/lib/kubelet/pods/527bc623-e6ed-41e9-92ac-8ffea6366fc2/volumes" Oct 30 03:46:53.943062 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:46:53.943026 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:47:19.108168 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:47:19.108130 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["tests-monitoring-otlp-identity/otlp-identity-spoof-probe"] Oct 30 03:47:19.108821 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:47:19.108779 5882 cpu_manager.go:414] "containerMap: removing container" podUID="527bc623-e6ed-41e9-92ac-8ffea6366fc2" containerName="ta-container" Oct 30 03:47:19.108821 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:47:19.108804 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="527bc623-e6ed-41e9-92ac-8ffea6366fc2" containerName="ta-container" Oct 30 03:47:19.108957 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:47:19.108899 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="527bc623-e6ed-41e9-92ac-8ffea6366fc2" containerName="ta-container" Oct 30 03:47:19.112181 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:47:19.112162 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" Oct 30 03:47:19.127940 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:47:19.127915 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-otlp-identity/otlp-identity-spoof-probe"] Oct 30 03:47:19.138166 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:47:19.138144 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"tests-monitoring-otlp-identity\"/\"default-dockercfg-8ptcq\"" Oct 30 03:47:19.145220 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:47:19.145202 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" Oct 30 03:47:19.437866 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:47:19.437833 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-otlp-identity/otlp-identity-spoof-probe"] Oct 30 03:47:19.440706 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:47:19.440676 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddee2dcfa_ebfc_4f81_9ea7_7ffc32acbcec.slice/crio-adad18a2580e049ffc761e4491678ae9b81442403e665dc88e0c4a902f150e7f WatchSource:0}: Error finding container adad18a2580e049ffc761e4491678ae9b81442403e665dc88e0c4a902f150e7f: Status 404 returned error can't find the container with id adad18a2580e049ffc761e4491678ae9b81442403e665dc88e0c4a902f150e7f Oct 30 03:47:19.457805 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:47:19.457775 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" event={"ID":"dee2dcfa-ebfc-4f81-9ea7-7ffc32acbcec","Type":"ContainerStarted","Data":"adad18a2580e049ffc761e4491678ae9b81442403e665dc88e0c4a902f150e7f"} Oct 30 03:47:20.464057 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:47:20.464022 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" event={"ID":"dee2dcfa-ebfc-4f81-9ea7-7ffc32acbcec","Type":"ContainerStarted","Data":"92ce748fac6bd7dec39afdfe046c98eb48051446bc77f6cd743691b543db2a15"} Oct 30 03:47:20.501123 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:47:20.501077 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" podStartSLOduration=1.5010633439999999 podStartE2EDuration="1.501063344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:20.500544938 +0000 UTC m=+3952.614859912" watchObservedRunningTime="2026-10-30 03:47:20.501063344 +0000 UTC m=+3952.615378362" Oct 30 03:47:23.942246 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:47:23.942208 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:47:53.942164 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:47:53.942122 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:48:23.942903 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:48:23.942858 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:48:53.942756 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:48:53.942700 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:49:23.943003 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:49:23.942919 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:49:28.564008 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:49:28.563974 5882 scope.go:122] "RemoveContainer" containerID="bb7e40a8e240789c028e607fd4c02fd2ff5839b1b91e8be46b38fc0faab3896b" Oct 30 03:49:53.944925 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:49:53.944874 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:50:01.631124 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:01.631087 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-otlp-identity/otlp-identity-spoof-probe"] Oct 30 03:50:01.631781 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:01.631309 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" podUID="dee2dcfa-ebfc-4f81-9ea7-7ffc32acbcec" containerName="otlp-spoof-probe" containerID="cri-o://92ce748fac6bd7dec39afdfe046c98eb48051446bc77f6cd743691b543db2a15" gracePeriod=30 Oct 30 03:50:23.942701 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:50:23.942659 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:50:31.770998 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:31.770971 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" Oct 30 03:50:32.288101 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:32.288021 5882 generic.go:356] "Generic (PLEG): container finished" podID="dee2dcfa-ebfc-4f81-9ea7-7ffc32acbcec" containerID="92ce748fac6bd7dec39afdfe046c98eb48051446bc77f6cd743691b543db2a15" exitCode=137 Oct 30 03:50:32.288101 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:32.288063 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" Oct 30 03:50:32.288298 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:32.288104 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" event={"ID":"dee2dcfa-ebfc-4f81-9ea7-7ffc32acbcec","Type":"ContainerDied","Data":"92ce748fac6bd7dec39afdfe046c98eb48051446bc77f6cd743691b543db2a15"} Oct 30 03:50:32.288298 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:32.288149 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" event={"ID":"dee2dcfa-ebfc-4f81-9ea7-7ffc32acbcec","Type":"ContainerDied","Data":"adad18a2580e049ffc761e4491678ae9b81442403e665dc88e0c4a902f150e7f"} Oct 30 03:50:32.288298 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:32.288170 5882 scope.go:122] "RemoveContainer" containerID="92ce748fac6bd7dec39afdfe046c98eb48051446bc77f6cd743691b543db2a15" Oct 30 03:50:32.296229 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:32.296210 5882 scope.go:122] "RemoveContainer" containerID="92ce748fac6bd7dec39afdfe046c98eb48051446bc77f6cd743691b543db2a15" Oct 30 03:50:32.296518 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:50:32.296489 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ce748fac6bd7dec39afdfe046c98eb48051446bc77f6cd743691b543db2a15\": container with ID starting with 92ce748fac6bd7dec39afdfe046c98eb48051446bc77f6cd743691b543db2a15 not found: ID does not exist" containerID="92ce748fac6bd7dec39afdfe046c98eb48051446bc77f6cd743691b543db2a15" Oct 30 03:50:32.296583 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:32.296532 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ce748fac6bd7dec39afdfe046c98eb48051446bc77f6cd743691b543db2a15"} err="failed to get container status \"92ce748fac6bd7dec39afdfe046c98eb48051446bc77f6cd743691b543db2a15\": rpc error: code = NotFound desc = could not find container \"92ce748fac6bd7dec39afdfe046c98eb48051446bc77f6cd743691b543db2a15\": container with ID starting with 92ce748fac6bd7dec39afdfe046c98eb48051446bc77f6cd743691b543db2a15 not found: ID does not exist" Oct 30 03:50:32.336256 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:32.336231 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-otlp-identity/otlp-identity-spoof-probe"] Oct 30 03:50:32.347715 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:32.347689 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["tests-monitoring-otlp-identity/otlp-identity-spoof-probe"] Oct 30 03:50:34.009766 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:34.009734 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dee2dcfa-ebfc-4f81-9ea7-7ffc32acbcec" path="/var/lib/kubelet/pods/dee2dcfa-ebfc-4f81-9ea7-7ffc32acbcec/volumes" Oct 30 03:50:36.519279 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:36.519247 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m"] Oct 30 03:50:36.519699 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:36.519609 5882 cpu_manager.go:414] "containerMap: removing container" podUID="dee2dcfa-ebfc-4f81-9ea7-7ffc32acbcec" containerName="otlp-spoof-probe" Oct 30 03:50:36.519699 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:36.519676 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dee2dcfa-ebfc-4f81-9ea7-7ffc32acbcec" containerName="otlp-spoof-probe" Oct 30 03:50:36.519810 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:36.519798 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="dee2dcfa-ebfc-4f81-9ea7-7ffc32acbcec" containerName="otlp-spoof-probe" Oct 30 03:50:36.522867 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:36.522849 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" Oct 30 03:50:36.552468 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:36.552441 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m"] Oct 30 03:50:36.632355 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:36.632324 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csrcr\" (UniqueName: \"kubernetes.io/projected/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-kube-api-access-csrcr\") pod \"data-science-collector-targetallocator-744cffff95-5rt9m\" (UID: \"2bd1487c-4b5e-4019-aa35-45e5e4dea2ed\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" Oct 30 03:50:36.632490 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:36.632362 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-ta-internal\") pod \"data-science-collector-targetallocator-744cffff95-5rt9m\" (UID: \"2bd1487c-4b5e-4019-aa35-45e5e4dea2ed\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" Oct 30 03:50:36.632490 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:36.632390 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-744cffff95-5rt9m\" (UID: \"2bd1487c-4b5e-4019-aa35-45e5e4dea2ed\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" Oct 30 03:50:36.733139 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:36.733106 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-744cffff95-5rt9m\" (UID: \"2bd1487c-4b5e-4019-aa35-45e5e4dea2ed\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" Oct 30 03:50:36.733290 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:36.733205 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csrcr\" (UniqueName: \"kubernetes.io/projected/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-kube-api-access-csrcr\") pod \"data-science-collector-targetallocator-744cffff95-5rt9m\" (UID: \"2bd1487c-4b5e-4019-aa35-45e5e4dea2ed\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" Oct 30 03:50:36.733290 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:36.733238 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-ta-internal\") pod \"data-science-collector-targetallocator-744cffff95-5rt9m\" (UID: \"2bd1487c-4b5e-4019-aa35-45e5e4dea2ed\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" Oct 30 03:50:36.734014 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:36.733983 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-ta-internal\") pod \"data-science-collector-targetallocator-744cffff95-5rt9m\" (UID: \"2bd1487c-4b5e-4019-aa35-45e5e4dea2ed\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" Oct 30 03:50:36.735504 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:36.735485 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-744cffff95-5rt9m\" (UID: \"2bd1487c-4b5e-4019-aa35-45e5e4dea2ed\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" Oct 30 03:50:36.756082 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:36.756058 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csrcr\" (UniqueName: \"kubernetes.io/projected/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-kube-api-access-csrcr\") pod \"data-science-collector-targetallocator-744cffff95-5rt9m\" (UID: \"2bd1487c-4b5e-4019-aa35-45e5e4dea2ed\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" Oct 30 03:50:36.834229 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:36.834154 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" Oct 30 03:50:37.126298 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:37.126032 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m"] Oct 30 03:50:37.128811 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:50:37.128778 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bd1487c_4b5e_4019_aa35_45e5e4dea2ed.slice/crio-686a348e17fd21be67bc0c2b948c6b80ecc672532a64fd1389501cb1b8dd1e73 WatchSource:0}: Error finding container 686a348e17fd21be67bc0c2b948c6b80ecc672532a64fd1389501cb1b8dd1e73: Status 404 returned error can't find the container with id 686a348e17fd21be67bc0c2b948c6b80ecc672532a64fd1389501cb1b8dd1e73 Oct 30 03:50:37.131329 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:37.131293 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:50:37.131437 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:37.131351 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:50:37.312005 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:37.311963 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" event={"ID":"2bd1487c-4b5e-4019-aa35-45e5e4dea2ed","Type":"ContainerStarted","Data":"22247f43343145f3fe8a74746402f69968410abf6d73302fe1d4db4df8b3c830"} Oct 30 03:50:37.312206 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:37.312015 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" event={"ID":"2bd1487c-4b5e-4019-aa35-45e5e4dea2ed","Type":"ContainerStarted","Data":"686a348e17fd21be67bc0c2b948c6b80ecc672532a64fd1389501cb1b8dd1e73"} Oct 30 03:50:37.312206 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:37.312109 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" Oct 30 03:50:37.313084 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:37.313058 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" podUID="2bd1487c-4b5e-4019-aa35-45e5e4dea2ed" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.65:8080/readyz\": dial tcp 10.134.0.65:8080: connect: connection refused" Oct 30 03:50:37.352666 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:37.352607 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" podStartSLOduration=1.352594334 podStartE2EDuration="1.352594334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:50:37.35145668 +0000 UTC m=+4149.465771669" watchObservedRunningTime="2026-10-30 03:50:37.352594334 +0000 UTC m=+4149.466909308" Oct 30 03:50:38.317290 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:38.317263 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" Oct 30 03:50:38.386333 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:38.386300 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg"] Oct 30 03:50:38.386729 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:38.386700 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" podUID="178c0ce3-fa51-4487-abeb-78c6496df192" containerName="ta-container" containerID="cri-o://a2f3bac04be1812909c114d6bf0588b95c41472b6b08d13d6e1acadec22f458b" gracePeriod=30 Oct 30 03:50:38.629599 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:38.629572 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" Oct 30 03:50:38.751427 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:38.751393 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/178c0ce3-fa51-4487-abeb-78c6496df192-kube-api-access-w2hcv\" (UniqueName: \"kubernetes.io/projected/178c0ce3-fa51-4487-abeb-78c6496df192-kube-api-access-w2hcv\") pod \"178c0ce3-fa51-4487-abeb-78c6496df192\" (UID: \"178c0ce3-fa51-4487-abeb-78c6496df192\") " Oct 30 03:50:38.751596 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:38.751500 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/178c0ce3-fa51-4487-abeb-78c6496df192-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/178c0ce3-fa51-4487-abeb-78c6496df192-data-science-collector-ta-server-cert\") pod \"178c0ce3-fa51-4487-abeb-78c6496df192\" (UID: \"178c0ce3-fa51-4487-abeb-78c6496df192\") " Oct 30 03:50:38.751596 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:38.751524 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/178c0ce3-fa51-4487-abeb-78c6496df192-ta-internal\" (UniqueName: \"kubernetes.io/configmap/178c0ce3-fa51-4487-abeb-78c6496df192-ta-internal\") pod \"178c0ce3-fa51-4487-abeb-78c6496df192\" (UID: \"178c0ce3-fa51-4487-abeb-78c6496df192\") " Oct 30 03:50:38.751944 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:38.751916 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/178c0ce3-fa51-4487-abeb-78c6496df192-ta-internal" pod "178c0ce3-fa51-4487-abeb-78c6496df192" (UID: "178c0ce3-fa51-4487-abeb-78c6496df192"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:50:38.753517 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:38.753487 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178c0ce3-fa51-4487-abeb-78c6496df192-kube-api-access-w2hcv" pod "178c0ce3-fa51-4487-abeb-78c6496df192" (UID: "178c0ce3-fa51-4487-abeb-78c6496df192"). InnerVolumeSpecName "kube-api-access-w2hcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:50:38.753517 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:38.753495 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/178c0ce3-fa51-4487-abeb-78c6496df192-data-science-collector-ta-server-cert" pod "178c0ce3-fa51-4487-abeb-78c6496df192" (UID: "178c0ce3-fa51-4487-abeb-78c6496df192"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:50:38.853117 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:38.853075 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2hcv\" (UniqueName: \"kubernetes.io/projected/178c0ce3-fa51-4487-abeb-78c6496df192-kube-api-access-w2hcv\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:50:38.853117 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:38.853112 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/178c0ce3-fa51-4487-abeb-78c6496df192-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:50:38.853321 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:38.853128 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/178c0ce3-fa51-4487-abeb-78c6496df192-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:50:39.321666 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:39.321608 5882 generic.go:356] "Generic (PLEG): container finished" podID="178c0ce3-fa51-4487-abeb-78c6496df192" containerID="a2f3bac04be1812909c114d6bf0588b95c41472b6b08d13d6e1acadec22f458b" exitCode=0 Oct 30 03:50:39.322044 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:39.321694 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" Oct 30 03:50:39.322044 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:39.321715 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" event={"ID":"178c0ce3-fa51-4487-abeb-78c6496df192","Type":"ContainerDied","Data":"a2f3bac04be1812909c114d6bf0588b95c41472b6b08d13d6e1acadec22f458b"} Oct 30 03:50:39.322044 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:39.321760 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg" event={"ID":"178c0ce3-fa51-4487-abeb-78c6496df192","Type":"ContainerDied","Data":"b0739d05a5f7b33f8248a8045f42239582a6ce86daa013dcee24b56f691863a0"} Oct 30 03:50:39.322044 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:39.321777 5882 scope.go:122] "RemoveContainer" containerID="a2f3bac04be1812909c114d6bf0588b95c41472b6b08d13d6e1acadec22f458b" Oct 30 03:50:39.330117 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:39.330099 5882 scope.go:122] "RemoveContainer" containerID="a2f3bac04be1812909c114d6bf0588b95c41472b6b08d13d6e1acadec22f458b" Oct 30 03:50:39.330377 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:50:39.330361 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f3bac04be1812909c114d6bf0588b95c41472b6b08d13d6e1acadec22f458b\": container with ID starting with a2f3bac04be1812909c114d6bf0588b95c41472b6b08d13d6e1acadec22f458b not found: ID does not exist" containerID="a2f3bac04be1812909c114d6bf0588b95c41472b6b08d13d6e1acadec22f458b" Oct 30 03:50:39.330426 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:39.330385 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f3bac04be1812909c114d6bf0588b95c41472b6b08d13d6e1acadec22f458b"} err="failed to get container status \"a2f3bac04be1812909c114d6bf0588b95c41472b6b08d13d6e1acadec22f458b\": rpc error: code = NotFound desc = could not find container \"a2f3bac04be1812909c114d6bf0588b95c41472b6b08d13d6e1acadec22f458b\": container with ID starting with a2f3bac04be1812909c114d6bf0588b95c41472b6b08d13d6e1acadec22f458b not found: ID does not exist" Oct 30 03:50:39.370134 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:39.370104 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg"] Oct 30 03:50:39.381427 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:39.381402 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6d48bd4dcc-jppfg"] Oct 30 03:50:40.009768 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:40.009737 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="178c0ce3-fa51-4487-abeb-78c6496df192" path="/var/lib/kubelet/pods/178c0ce3-fa51-4487-abeb-78c6496df192/volumes" Oct 30 03:50:42.538679 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:42.538641 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4"] Oct 30 03:50:42.539076 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:42.539046 5882 cpu_manager.go:414] "containerMap: removing container" podUID="178c0ce3-fa51-4487-abeb-78c6496df192" containerName="ta-container" Oct 30 03:50:42.539076 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:42.539062 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="178c0ce3-fa51-4487-abeb-78c6496df192" containerName="ta-container" Oct 30 03:50:42.539163 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:42.539154 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="178c0ce3-fa51-4487-abeb-78c6496df192" containerName="ta-container" Oct 30 03:50:42.543839 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:42.543819 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" Oct 30 03:50:42.561357 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:42.561329 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4"] Oct 30 03:50:42.588327 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:42.588301 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/02b1b79d-9199-4492-bd48-c15d39082277-ta-internal\") pod \"data-science-collector-targetallocator-df9b59866-kbzp4\" (UID: \"02b1b79d-9199-4492-bd48-c15d39082277\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" Oct 30 03:50:42.588464 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:42.588349 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c44tt\" (UniqueName: \"kubernetes.io/projected/02b1b79d-9199-4492-bd48-c15d39082277-kube-api-access-c44tt\") pod \"data-science-collector-targetallocator-df9b59866-kbzp4\" (UID: \"02b1b79d-9199-4492-bd48-c15d39082277\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" Oct 30 03:50:42.588464 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:42.588371 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/02b1b79d-9199-4492-bd48-c15d39082277-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-df9b59866-kbzp4\" (UID: \"02b1b79d-9199-4492-bd48-c15d39082277\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" Oct 30 03:50:42.689310 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:42.689283 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/02b1b79d-9199-4492-bd48-c15d39082277-ta-internal\") pod \"data-science-collector-targetallocator-df9b59866-kbzp4\" (UID: \"02b1b79d-9199-4492-bd48-c15d39082277\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" Oct 30 03:50:42.689461 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:42.689330 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c44tt\" (UniqueName: \"kubernetes.io/projected/02b1b79d-9199-4492-bd48-c15d39082277-kube-api-access-c44tt\") pod \"data-science-collector-targetallocator-df9b59866-kbzp4\" (UID: \"02b1b79d-9199-4492-bd48-c15d39082277\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" Oct 30 03:50:42.689461 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:42.689353 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/02b1b79d-9199-4492-bd48-c15d39082277-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-df9b59866-kbzp4\" (UID: \"02b1b79d-9199-4492-bd48-c15d39082277\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" Oct 30 03:50:42.690005 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:42.689982 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/02b1b79d-9199-4492-bd48-c15d39082277-ta-internal\") pod \"data-science-collector-targetallocator-df9b59866-kbzp4\" (UID: \"02b1b79d-9199-4492-bd48-c15d39082277\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" Oct 30 03:50:42.691866 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:42.691845 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/02b1b79d-9199-4492-bd48-c15d39082277-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-df9b59866-kbzp4\" (UID: \"02b1b79d-9199-4492-bd48-c15d39082277\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" Oct 30 03:50:42.711820 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:42.711793 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c44tt\" (UniqueName: \"kubernetes.io/projected/02b1b79d-9199-4492-bd48-c15d39082277-kube-api-access-c44tt\") pod \"data-science-collector-targetallocator-df9b59866-kbzp4\" (UID: \"02b1b79d-9199-4492-bd48-c15d39082277\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" Oct 30 03:50:42.854768 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:42.854695 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" Oct 30 03:50:43.147939 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:43.147857 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4"] Oct 30 03:50:43.151008 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:50:43.150979 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02b1b79d_9199_4492_bd48_c15d39082277.slice/crio-8a30e52c3b1004dafa5bd54f1bea7556c9364c18e7f63b86bfe9bdd1aaab3147 WatchSource:0}: Error finding container 8a30e52c3b1004dafa5bd54f1bea7556c9364c18e7f63b86bfe9bdd1aaab3147: Status 404 returned error can't find the container with id 8a30e52c3b1004dafa5bd54f1bea7556c9364c18e7f63b86bfe9bdd1aaab3147 Oct 30 03:50:43.153127 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:43.153089 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:50:43.153212 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:43.153160 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:50:43.342421 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:43.342386 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" event={"ID":"02b1b79d-9199-4492-bd48-c15d39082277","Type":"ContainerStarted","Data":"b2e770a515279bdfe6b1a028a0d0da0f9d3056f76985df2f4b20324a128616d1"} Oct 30 03:50:43.342421 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:43.342426 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" event={"ID":"02b1b79d-9199-4492-bd48-c15d39082277","Type":"ContainerStarted","Data":"8a30e52c3b1004dafa5bd54f1bea7556c9364c18e7f63b86bfe9bdd1aaab3147"} Oct 30 03:50:43.342658 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:43.342587 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" Oct 30 03:50:43.343596 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:43.343571 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" podUID="02b1b79d-9199-4492-bd48-c15d39082277" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.66:8080/readyz\": dial tcp 10.134.0.66:8080: connect: connection refused" Oct 30 03:50:43.384695 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:43.384644 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" podStartSLOduration=1.384611424 podStartE2EDuration="1.384611424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:50:43.384566602 +0000 UTC m=+4155.498881577" watchObservedRunningTime="2026-10-30 03:50:43.384611424 +0000 UTC m=+4155.498926400" Oct 30 03:50:44.348246 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:44.348215 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" Oct 30 03:50:44.428407 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:44.428375 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m"] Oct 30 03:50:44.428640 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:44.428582 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" podUID="2bd1487c-4b5e-4019-aa35-45e5e4dea2ed" containerName="ta-container" containerID="cri-o://22247f43343145f3fe8a74746402f69968410abf6d73302fe1d4db4df8b3c830" gracePeriod=30 Oct 30 03:50:44.665348 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:44.665319 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" Oct 30 03:50:44.709814 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:44.709779 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-ta-internal\" (UniqueName: \"kubernetes.io/configmap/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-ta-internal\") pod \"2bd1487c-4b5e-4019-aa35-45e5e4dea2ed\" (UID: \"2bd1487c-4b5e-4019-aa35-45e5e4dea2ed\") " Oct 30 03:50:44.709931 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:44.709854 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-kube-api-access-csrcr\" (UniqueName: \"kubernetes.io/projected/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-kube-api-access-csrcr\") pod \"2bd1487c-4b5e-4019-aa35-45e5e4dea2ed\" (UID: \"2bd1487c-4b5e-4019-aa35-45e5e4dea2ed\") " Oct 30 03:50:44.710003 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:44.709965 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-data-science-collector-ta-server-cert\") pod \"2bd1487c-4b5e-4019-aa35-45e5e4dea2ed\" (UID: \"2bd1487c-4b5e-4019-aa35-45e5e4dea2ed\") " Oct 30 03:50:44.710160 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:44.710133 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-ta-internal" pod "2bd1487c-4b5e-4019-aa35-45e5e4dea2ed" (UID: "2bd1487c-4b5e-4019-aa35-45e5e4dea2ed"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:50:44.712011 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:44.711989 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-kube-api-access-csrcr" pod "2bd1487c-4b5e-4019-aa35-45e5e4dea2ed" (UID: "2bd1487c-4b5e-4019-aa35-45e5e4dea2ed"). InnerVolumeSpecName "kube-api-access-csrcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:50:44.712221 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:44.712199 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-data-science-collector-ta-server-cert" pod "2bd1487c-4b5e-4019-aa35-45e5e4dea2ed" (UID: "2bd1487c-4b5e-4019-aa35-45e5e4dea2ed"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:50:44.810834 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:44.810785 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csrcr\" (UniqueName: \"kubernetes.io/projected/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-kube-api-access-csrcr\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:50:44.810834 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:44.810832 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:50:44.810834 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:44.810848 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:50:45.352028 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:45.351990 5882 generic.go:356] "Generic (PLEG): container finished" podID="2bd1487c-4b5e-4019-aa35-45e5e4dea2ed" containerID="22247f43343145f3fe8a74746402f69968410abf6d73302fe1d4db4df8b3c830" exitCode=0 Oct 30 03:50:45.352528 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:45.352033 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" Oct 30 03:50:45.352528 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:45.352077 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" event={"ID":"2bd1487c-4b5e-4019-aa35-45e5e4dea2ed","Type":"ContainerDied","Data":"22247f43343145f3fe8a74746402f69968410abf6d73302fe1d4db4df8b3c830"} Oct 30 03:50:45.352528 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:45.352129 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m" event={"ID":"2bd1487c-4b5e-4019-aa35-45e5e4dea2ed","Type":"ContainerDied","Data":"686a348e17fd21be67bc0c2b948c6b80ecc672532a64fd1389501cb1b8dd1e73"} Oct 30 03:50:45.352528 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:45.352152 5882 scope.go:122] "RemoveContainer" containerID="22247f43343145f3fe8a74746402f69968410abf6d73302fe1d4db4df8b3c830" Oct 30 03:50:45.361548 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:45.361530 5882 scope.go:122] "RemoveContainer" containerID="22247f43343145f3fe8a74746402f69968410abf6d73302fe1d4db4df8b3c830" Oct 30 03:50:45.361841 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:50:45.361820 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22247f43343145f3fe8a74746402f69968410abf6d73302fe1d4db4df8b3c830\": container with ID starting with 22247f43343145f3fe8a74746402f69968410abf6d73302fe1d4db4df8b3c830 not found: ID does not exist" containerID="22247f43343145f3fe8a74746402f69968410abf6d73302fe1d4db4df8b3c830" Oct 30 03:50:45.361912 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:45.361852 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22247f43343145f3fe8a74746402f69968410abf6d73302fe1d4db4df8b3c830"} err="failed to get container status \"22247f43343145f3fe8a74746402f69968410abf6d73302fe1d4db4df8b3c830\": rpc error: code = NotFound desc = could not find container \"22247f43343145f3fe8a74746402f69968410abf6d73302fe1d4db4df8b3c830\": container with ID starting with 22247f43343145f3fe8a74746402f69968410abf6d73302fe1d4db4df8b3c830 not found: ID does not exist" Oct 30 03:50:45.403796 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:45.403762 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m"] Oct 30 03:50:45.415282 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:45.415251 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-744cffff95-5rt9m"] Oct 30 03:50:46.009167 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:46.009137 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd1487c-4b5e-4019-aa35-45e5e4dea2ed" path="/var/lib/kubelet/pods/2bd1487c-4b5e-4019-aa35-45e5e4dea2ed/volumes" Oct 30 03:50:48.560305 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:48.560273 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb"] Oct 30 03:50:48.560722 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:48.560640 5882 cpu_manager.go:414] "containerMap: removing container" podUID="2bd1487c-4b5e-4019-aa35-45e5e4dea2ed" containerName="ta-container" Oct 30 03:50:48.560722 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:48.560652 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd1487c-4b5e-4019-aa35-45e5e4dea2ed" containerName="ta-container" Oct 30 03:50:48.560808 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:48.560733 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd1487c-4b5e-4019-aa35-45e5e4dea2ed" containerName="ta-container" Oct 30 03:50:48.565251 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:48.565229 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" Oct 30 03:50:48.595040 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:48.595012 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb"] Oct 30 03:50:48.644848 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:48.644817 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm2cw\" (UniqueName: \"kubernetes.io/projected/37d9e564-dbad-442a-b702-50202d2a82cd-kube-api-access-fm2cw\") pod \"data-science-collector-targetallocator-b6f46c55d-2sxfb\" (UID: \"37d9e564-dbad-442a-b702-50202d2a82cd\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" Oct 30 03:50:48.644985 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:48.644935 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/37d9e564-dbad-442a-b702-50202d2a82cd-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-b6f46c55d-2sxfb\" (UID: \"37d9e564-dbad-442a-b702-50202d2a82cd\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" Oct 30 03:50:48.644985 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:48.644964 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/37d9e564-dbad-442a-b702-50202d2a82cd-ta-internal\") pod \"data-science-collector-targetallocator-b6f46c55d-2sxfb\" (UID: \"37d9e564-dbad-442a-b702-50202d2a82cd\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" Oct 30 03:50:48.746023 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:48.745990 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/37d9e564-dbad-442a-b702-50202d2a82cd-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-b6f46c55d-2sxfb\" (UID: \"37d9e564-dbad-442a-b702-50202d2a82cd\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" Oct 30 03:50:48.746178 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:48.746034 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/37d9e564-dbad-442a-b702-50202d2a82cd-ta-internal\") pod \"data-science-collector-targetallocator-b6f46c55d-2sxfb\" (UID: \"37d9e564-dbad-442a-b702-50202d2a82cd\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" Oct 30 03:50:48.746178 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:48.746079 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm2cw\" (UniqueName: \"kubernetes.io/projected/37d9e564-dbad-442a-b702-50202d2a82cd-kube-api-access-fm2cw\") pod \"data-science-collector-targetallocator-b6f46c55d-2sxfb\" (UID: \"37d9e564-dbad-442a-b702-50202d2a82cd\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" Oct 30 03:50:48.746817 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:48.746790 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/37d9e564-dbad-442a-b702-50202d2a82cd-ta-internal\") pod \"data-science-collector-targetallocator-b6f46c55d-2sxfb\" (UID: \"37d9e564-dbad-442a-b702-50202d2a82cd\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" Oct 30 03:50:48.748466 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:48.748447 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/37d9e564-dbad-442a-b702-50202d2a82cd-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-b6f46c55d-2sxfb\" (UID: \"37d9e564-dbad-442a-b702-50202d2a82cd\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" Oct 30 03:50:48.770103 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:48.770081 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm2cw\" (UniqueName: \"kubernetes.io/projected/37d9e564-dbad-442a-b702-50202d2a82cd-kube-api-access-fm2cw\") pod \"data-science-collector-targetallocator-b6f46c55d-2sxfb\" (UID: \"37d9e564-dbad-442a-b702-50202d2a82cd\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" Oct 30 03:50:48.876489 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:48.876409 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" Oct 30 03:50:49.190052 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:49.189976 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb"] Oct 30 03:50:49.194918 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:50:49.194887 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37d9e564_dbad_442a_b702_50202d2a82cd.slice/crio-6d1ebceed97cd0d26952abbb77e3e6a2bea84c93f5567efa8f0a4535303d8fb5 WatchSource:0}: Error finding container 6d1ebceed97cd0d26952abbb77e3e6a2bea84c93f5567efa8f0a4535303d8fb5: Status 404 returned error can't find the container with id 6d1ebceed97cd0d26952abbb77e3e6a2bea84c93f5567efa8f0a4535303d8fb5 Oct 30 03:50:49.196828 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:49.196794 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:50:49.196932 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:49.196851 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:50:49.372002 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:49.371969 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" event={"ID":"37d9e564-dbad-442a-b702-50202d2a82cd","Type":"ContainerStarted","Data":"f5b3b813af3fe1c468d027ae99122640fc3697934732d8a288e6a690129c3dea"} Oct 30 03:50:49.372162 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:49.372014 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" event={"ID":"37d9e564-dbad-442a-b702-50202d2a82cd","Type":"ContainerStarted","Data":"6d1ebceed97cd0d26952abbb77e3e6a2bea84c93f5567efa8f0a4535303d8fb5"} Oct 30 03:50:49.372246 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:49.372183 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" Oct 30 03:50:49.373189 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:49.373161 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" podUID="37d9e564-dbad-442a-b702-50202d2a82cd" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.67:8080/readyz\": dial tcp 10.134.0.67:8080: connect: connection refused" Oct 30 03:50:49.417822 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:49.417777 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" podStartSLOduration=1.4177638670000001 podStartE2EDuration="1.417763867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:50:49.416139929 +0000 UTC m=+4161.530454905" watchObservedRunningTime="2026-10-30 03:50:49.417763867 +0000 UTC m=+4161.532078905" Oct 30 03:50:50.377110 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:50.377079 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" Oct 30 03:50:50.447946 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:50.447914 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4"] Oct 30 03:50:50.448441 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:50.448122 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" podUID="02b1b79d-9199-4492-bd48-c15d39082277" containerName="ta-container" containerID="cri-o://b2e770a515279bdfe6b1a028a0d0da0f9d3056f76985df2f4b20324a128616d1" gracePeriod=30 Oct 30 03:50:50.688738 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:50.688712 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" Oct 30 03:50:50.764824 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:50.764790 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02b1b79d-9199-4492-bd48-c15d39082277-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/02b1b79d-9199-4492-bd48-c15d39082277-data-science-collector-ta-server-cert\") pod \"02b1b79d-9199-4492-bd48-c15d39082277\" (UID: \"02b1b79d-9199-4492-bd48-c15d39082277\") " Oct 30 03:50:50.765007 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:50.764884 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02b1b79d-9199-4492-bd48-c15d39082277-ta-internal\" (UniqueName: \"kubernetes.io/configmap/02b1b79d-9199-4492-bd48-c15d39082277-ta-internal\") pod \"02b1b79d-9199-4492-bd48-c15d39082277\" (UID: \"02b1b79d-9199-4492-bd48-c15d39082277\") " Oct 30 03:50:50.765007 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:50.764914 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02b1b79d-9199-4492-bd48-c15d39082277-kube-api-access-c44tt\" (UniqueName: \"kubernetes.io/projected/02b1b79d-9199-4492-bd48-c15d39082277-kube-api-access-c44tt\") pod \"02b1b79d-9199-4492-bd48-c15d39082277\" (UID: \"02b1b79d-9199-4492-bd48-c15d39082277\") " Oct 30 03:50:50.765237 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:50.765206 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02b1b79d-9199-4492-bd48-c15d39082277-ta-internal" pod "02b1b79d-9199-4492-bd48-c15d39082277" (UID: "02b1b79d-9199-4492-bd48-c15d39082277"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:50:50.766949 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:50.766924 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b1b79d-9199-4492-bd48-c15d39082277-data-science-collector-ta-server-cert" pod "02b1b79d-9199-4492-bd48-c15d39082277" (UID: "02b1b79d-9199-4492-bd48-c15d39082277"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:50:50.767045 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:50.766923 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b1b79d-9199-4492-bd48-c15d39082277-kube-api-access-c44tt" pod "02b1b79d-9199-4492-bd48-c15d39082277" (UID: "02b1b79d-9199-4492-bd48-c15d39082277"). InnerVolumeSpecName "kube-api-access-c44tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:50:50.866159 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:50.866124 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/02b1b79d-9199-4492-bd48-c15d39082277-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:50:50.866159 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:50.866151 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/02b1b79d-9199-4492-bd48-c15d39082277-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:50:50.866159 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:50.866163 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c44tt\" (UniqueName: \"kubernetes.io/projected/02b1b79d-9199-4492-bd48-c15d39082277-kube-api-access-c44tt\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:50:51.380938 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:51.380907 5882 generic.go:356] "Generic (PLEG): container finished" podID="02b1b79d-9199-4492-bd48-c15d39082277" containerID="b2e770a515279bdfe6b1a028a0d0da0f9d3056f76985df2f4b20324a128616d1" exitCode=0 Oct 30 03:50:51.381332 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:51.380952 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" Oct 30 03:50:51.381332 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:51.380984 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" event={"ID":"02b1b79d-9199-4492-bd48-c15d39082277","Type":"ContainerDied","Data":"b2e770a515279bdfe6b1a028a0d0da0f9d3056f76985df2f4b20324a128616d1"} Oct 30 03:50:51.381332 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:51.381022 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4" event={"ID":"02b1b79d-9199-4492-bd48-c15d39082277","Type":"ContainerDied","Data":"8a30e52c3b1004dafa5bd54f1bea7556c9364c18e7f63b86bfe9bdd1aaab3147"} Oct 30 03:50:51.381332 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:51.381038 5882 scope.go:122] "RemoveContainer" containerID="b2e770a515279bdfe6b1a028a0d0da0f9d3056f76985df2f4b20324a128616d1" Oct 30 03:50:51.390295 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:51.390279 5882 scope.go:122] "RemoveContainer" containerID="b2e770a515279bdfe6b1a028a0d0da0f9d3056f76985df2f4b20324a128616d1" Oct 30 03:50:51.390562 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:50:51.390541 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e770a515279bdfe6b1a028a0d0da0f9d3056f76985df2f4b20324a128616d1\": container with ID starting with b2e770a515279bdfe6b1a028a0d0da0f9d3056f76985df2f4b20324a128616d1 not found: ID does not exist" containerID="b2e770a515279bdfe6b1a028a0d0da0f9d3056f76985df2f4b20324a128616d1" Oct 30 03:50:51.390648 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:51.390568 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e770a515279bdfe6b1a028a0d0da0f9d3056f76985df2f4b20324a128616d1"} err="failed to get container status \"b2e770a515279bdfe6b1a028a0d0da0f9d3056f76985df2f4b20324a128616d1\": rpc error: code = NotFound desc = could not find container \"b2e770a515279bdfe6b1a028a0d0da0f9d3056f76985df2f4b20324a128616d1\": container with ID starting with b2e770a515279bdfe6b1a028a0d0da0f9d3056f76985df2f4b20324a128616d1 not found: ID does not exist" Oct 30 03:50:51.429035 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:51.429011 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4"] Oct 30 03:50:51.435475 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:51.435453 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-df9b59866-kbzp4"] Oct 30 03:50:52.009525 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:52.009491 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02b1b79d-9199-4492-bd48-c15d39082277" path="/var/lib/kubelet/pods/02b1b79d-9199-4492-bd48-c15d39082277/volumes" Oct 30 03:50:53.942442 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:50:53.942406 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:50:59.278855 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:59.278764 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm"] Oct 30 03:50:59.279371 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:59.279349 5882 cpu_manager.go:414] "containerMap: removing container" podUID="02b1b79d-9199-4492-bd48-c15d39082277" containerName="ta-container" Oct 30 03:50:59.279446 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:59.279373 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02b1b79d-9199-4492-bd48-c15d39082277" containerName="ta-container" Oct 30 03:50:59.279517 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:59.279505 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="02b1b79d-9199-4492-bd48-c15d39082277" containerName="ta-container" Oct 30 03:50:59.283987 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:59.283965 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" Oct 30 03:50:59.302673 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:59.302639 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm"] Oct 30 03:50:59.445854 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:59.445822 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lp6l\" (UniqueName: \"kubernetes.io/projected/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-kube-api-access-5lp6l\") pod \"data-science-collector-targetallocator-5888c49b67-mcqqm\" (UID: \"7e91285f-356d-4cc5-9e2c-0e9ca7973d40\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" Oct 30 03:50:59.446038 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:59.445954 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5888c49b67-mcqqm\" (UID: \"7e91285f-356d-4cc5-9e2c-0e9ca7973d40\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" Oct 30 03:50:59.446038 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:59.445978 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-ta-internal\") pod \"data-science-collector-targetallocator-5888c49b67-mcqqm\" (UID: \"7e91285f-356d-4cc5-9e2c-0e9ca7973d40\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" Oct 30 03:50:59.547248 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:59.547164 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5888c49b67-mcqqm\" (UID: \"7e91285f-356d-4cc5-9e2c-0e9ca7973d40\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" Oct 30 03:50:59.547248 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:59.547198 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-ta-internal\") pod \"data-science-collector-targetallocator-5888c49b67-mcqqm\" (UID: \"7e91285f-356d-4cc5-9e2c-0e9ca7973d40\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" Oct 30 03:50:59.547248 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:59.547225 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lp6l\" (UniqueName: \"kubernetes.io/projected/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-kube-api-access-5lp6l\") pod \"data-science-collector-targetallocator-5888c49b67-mcqqm\" (UID: \"7e91285f-356d-4cc5-9e2c-0e9ca7973d40\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" Oct 30 03:50:59.547877 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:59.547853 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-ta-internal\") pod \"data-science-collector-targetallocator-5888c49b67-mcqqm\" (UID: \"7e91285f-356d-4cc5-9e2c-0e9ca7973d40\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" Oct 30 03:50:59.549823 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:59.549787 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5888c49b67-mcqqm\" (UID: \"7e91285f-356d-4cc5-9e2c-0e9ca7973d40\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" Oct 30 03:50:59.570053 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:59.570026 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lp6l\" (UniqueName: \"kubernetes.io/projected/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-kube-api-access-5lp6l\") pod \"data-science-collector-targetallocator-5888c49b67-mcqqm\" (UID: \"7e91285f-356d-4cc5-9e2c-0e9ca7973d40\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" Oct 30 03:50:59.594731 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:59.594707 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" Oct 30 03:50:59.902266 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:59.902193 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm"] Oct 30 03:50:59.905823 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:50:59.905796 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e91285f_356d_4cc5_9e2c_0e9ca7973d40.slice/crio-c6a1cb9d77f8d4962177b6155d1cd3e62ea6366b1c1786e28c7c0fa202fae5b8 WatchSource:0}: Error finding container c6a1cb9d77f8d4962177b6155d1cd3e62ea6366b1c1786e28c7c0fa202fae5b8: Status 404 returned error can't find the container with id c6a1cb9d77f8d4962177b6155d1cd3e62ea6366b1c1786e28c7c0fa202fae5b8 Oct 30 03:50:59.908095 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:59.908066 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:50:59.908169 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:50:59.908121 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:51:00.422059 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:00.422025 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" event={"ID":"7e91285f-356d-4cc5-9e2c-0e9ca7973d40","Type":"ContainerStarted","Data":"e9245fb7bd1cc06ac24f3dc9c3b189ff10c2739345cda700eccc846d44e2c541"} Oct 30 03:51:00.422435 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:00.422076 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" event={"ID":"7e91285f-356d-4cc5-9e2c-0e9ca7973d40","Type":"ContainerStarted","Data":"c6a1cb9d77f8d4962177b6155d1cd3e62ea6366b1c1786e28c7c0fa202fae5b8"} Oct 30 03:51:00.422435 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:00.422098 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" Oct 30 03:51:00.423719 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:00.423700 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" Oct 30 03:51:00.462333 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:00.462290 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" podStartSLOduration=1.462276505 podStartE2EDuration="1.462276505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:00.461897273 +0000 UTC m=+4172.576212248" watchObservedRunningTime="2026-10-30 03:51:00.462276505 +0000 UTC m=+4172.576591479" Oct 30 03:51:00.531986 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:00.531957 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb"] Oct 30 03:51:00.532240 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:00.532216 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" podUID="37d9e564-dbad-442a-b702-50202d2a82cd" containerName="ta-container" containerID="cri-o://f5b3b813af3fe1c468d027ae99122640fc3697934732d8a288e6a690129c3dea" gracePeriod=30 Oct 30 03:51:00.782502 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:00.782477 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" Oct 30 03:51:00.960605 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:00.960516 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d9e564-dbad-442a-b702-50202d2a82cd-kube-api-access-fm2cw\" (UniqueName: \"kubernetes.io/projected/37d9e564-dbad-442a-b702-50202d2a82cd-kube-api-access-fm2cw\") pod \"37d9e564-dbad-442a-b702-50202d2a82cd\" (UID: \"37d9e564-dbad-442a-b702-50202d2a82cd\") " Oct 30 03:51:00.960605 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:00.960567 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37d9e564-dbad-442a-b702-50202d2a82cd-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/37d9e564-dbad-442a-b702-50202d2a82cd-data-science-collector-ta-server-cert\") pod \"37d9e564-dbad-442a-b702-50202d2a82cd\" (UID: \"37d9e564-dbad-442a-b702-50202d2a82cd\") " Oct 30 03:51:00.960864 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:00.960706 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d9e564-dbad-442a-b702-50202d2a82cd-ta-internal\" (UniqueName: \"kubernetes.io/configmap/37d9e564-dbad-442a-b702-50202d2a82cd-ta-internal\") pod \"37d9e564-dbad-442a-b702-50202d2a82cd\" (UID: \"37d9e564-dbad-442a-b702-50202d2a82cd\") " Oct 30 03:51:00.961035 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:00.961012 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d9e564-dbad-442a-b702-50202d2a82cd-ta-internal" pod "37d9e564-dbad-442a-b702-50202d2a82cd" (UID: "37d9e564-dbad-442a-b702-50202d2a82cd"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:51:00.962641 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:00.962604 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37d9e564-dbad-442a-b702-50202d2a82cd-data-science-collector-ta-server-cert" pod "37d9e564-dbad-442a-b702-50202d2a82cd" (UID: "37d9e564-dbad-442a-b702-50202d2a82cd"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:51:00.962726 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:00.962649 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d9e564-dbad-442a-b702-50202d2a82cd-kube-api-access-fm2cw" pod "37d9e564-dbad-442a-b702-50202d2a82cd" (UID: "37d9e564-dbad-442a-b702-50202d2a82cd"). InnerVolumeSpecName "kube-api-access-fm2cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:51:01.061954 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.061917 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm2cw\" (UniqueName: \"kubernetes.io/projected/37d9e564-dbad-442a-b702-50202d2a82cd-kube-api-access-fm2cw\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:51:01.061954 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.061949 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/37d9e564-dbad-442a-b702-50202d2a82cd-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:51:01.061954 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.061960 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/37d9e564-dbad-442a-b702-50202d2a82cd-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:51:01.331599 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.331561 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/operator-metrics-auth-probe"] Oct 30 03:51:01.331954 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.331939 5882 cpu_manager.go:414] "containerMap: removing container" podUID="37d9e564-dbad-442a-b702-50202d2a82cd" containerName="ta-container" Oct 30 03:51:01.331996 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.331957 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d9e564-dbad-442a-b702-50202d2a82cd" containerName="ta-container" Oct 30 03:51:01.332111 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.332092 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d9e564-dbad-442a-b702-50202d2a82cd" containerName="ta-container" Oct 30 03:51:01.335211 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.335192 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/operator-metrics-auth-probe" Oct 30 03:51:01.349381 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.349316 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"odh-observability-metrics-ca\"" Oct 30 03:51:01.350073 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.350048 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/operator-metrics-auth-probe"] Oct 30 03:51:01.427800 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.427767 5882 generic.go:356] "Generic (PLEG): container finished" podID="37d9e564-dbad-442a-b702-50202d2a82cd" containerID="f5b3b813af3fe1c468d027ae99122640fc3697934732d8a288e6a690129c3dea" exitCode=0 Oct 30 03:51:01.428207 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.427809 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" Oct 30 03:51:01.428207 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.427857 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" event={"ID":"37d9e564-dbad-442a-b702-50202d2a82cd","Type":"ContainerDied","Data":"f5b3b813af3fe1c468d027ae99122640fc3697934732d8a288e6a690129c3dea"} Oct 30 03:51:01.428207 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.427889 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb" event={"ID":"37d9e564-dbad-442a-b702-50202d2a82cd","Type":"ContainerDied","Data":"6d1ebceed97cd0d26952abbb77e3e6a2bea84c93f5567efa8f0a4535303d8fb5"} Oct 30 03:51:01.428207 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.427906 5882 scope.go:122] "RemoveContainer" containerID="f5b3b813af3fe1c468d027ae99122640fc3697934732d8a288e6a690129c3dea" Oct 30 03:51:01.438015 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.437998 5882 scope.go:122] "RemoveContainer" containerID="f5b3b813af3fe1c468d027ae99122640fc3697934732d8a288e6a690129c3dea" Oct 30 03:51:01.438306 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:51:01.438285 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b3b813af3fe1c468d027ae99122640fc3697934732d8a288e6a690129c3dea\": container with ID starting with f5b3b813af3fe1c468d027ae99122640fc3697934732d8a288e6a690129c3dea not found: ID does not exist" containerID="f5b3b813af3fe1c468d027ae99122640fc3697934732d8a288e6a690129c3dea" Oct 30 03:51:01.438399 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.438316 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b3b813af3fe1c468d027ae99122640fc3697934732d8a288e6a690129c3dea"} err="failed to get container status \"f5b3b813af3fe1c468d027ae99122640fc3697934732d8a288e6a690129c3dea\": rpc error: code = NotFound desc = could not find container \"f5b3b813af3fe1c468d027ae99122640fc3697934732d8a288e6a690129c3dea\": container with ID starting with f5b3b813af3fe1c468d027ae99122640fc3697934732d8a288e6a690129c3dea not found: ID does not exist" Oct 30 03:51:01.466108 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.466079 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-ca\" (UniqueName: \"kubernetes.io/configmap/bca46020-25d8-4db3-a374-dd1e5df58b1e-metrics-ca\") pod \"operator-metrics-auth-probe\" (UID: \"bca46020-25d8-4db3-a374-dd1e5df58b1e\") " pod="redhat-ods-monitoring/operator-metrics-auth-probe" Oct 30 03:51:01.466225 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.466208 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwj9b\" (UniqueName: \"kubernetes.io/projected/bca46020-25d8-4db3-a374-dd1e5df58b1e-kube-api-access-mwj9b\") pod \"operator-metrics-auth-probe\" (UID: \"bca46020-25d8-4db3-a374-dd1e5df58b1e\") " pod="redhat-ods-monitoring/operator-metrics-auth-probe" Oct 30 03:51:01.477706 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.477679 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb"] Oct 30 03:51:01.481784 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.481760 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b6f46c55d-2sxfb"] Oct 30 03:51:01.567266 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.567231 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwj9b\" (UniqueName: \"kubernetes.io/projected/bca46020-25d8-4db3-a374-dd1e5df58b1e-kube-api-access-mwj9b\") pod \"operator-metrics-auth-probe\" (UID: \"bca46020-25d8-4db3-a374-dd1e5df58b1e\") " pod="redhat-ods-monitoring/operator-metrics-auth-probe" Oct 30 03:51:01.567440 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.567314 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-ca\" (UniqueName: \"kubernetes.io/configmap/bca46020-25d8-4db3-a374-dd1e5df58b1e-metrics-ca\") pod \"operator-metrics-auth-probe\" (UID: \"bca46020-25d8-4db3-a374-dd1e5df58b1e\") " pod="redhat-ods-monitoring/operator-metrics-auth-probe" Oct 30 03:51:01.568022 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.567997 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-ca\" (UniqueName: \"kubernetes.io/configmap/bca46020-25d8-4db3-a374-dd1e5df58b1e-metrics-ca\") pod \"operator-metrics-auth-probe\" (UID: \"bca46020-25d8-4db3-a374-dd1e5df58b1e\") " pod="redhat-ods-monitoring/operator-metrics-auth-probe" Oct 30 03:51:01.597319 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.597246 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwj9b\" (UniqueName: \"kubernetes.io/projected/bca46020-25d8-4db3-a374-dd1e5df58b1e-kube-api-access-mwj9b\") pod \"operator-metrics-auth-probe\" (UID: \"bca46020-25d8-4db3-a374-dd1e5df58b1e\") " pod="redhat-ods-monitoring/operator-metrics-auth-probe" Oct 30 03:51:01.646354 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.646323 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/operator-metrics-auth-probe" Oct 30 03:51:01.936888 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:01.936695 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/operator-metrics-auth-probe"] Oct 30 03:51:01.939538 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:51:01.939500 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbca46020_25d8_4db3_a374_dd1e5df58b1e.slice/crio-7648dcc1f5bb7674b4547de1f0dd534d8477ef9d6fb459b2093d59af0208a73b WatchSource:0}: Error finding container 7648dcc1f5bb7674b4547de1f0dd534d8477ef9d6fb459b2093d59af0208a73b: Status 404 returned error can't find the container with id 7648dcc1f5bb7674b4547de1f0dd534d8477ef9d6fb459b2093d59af0208a73b Oct 30 03:51:02.009281 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:02.009256 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37d9e564-dbad-442a-b702-50202d2a82cd" path="/var/lib/kubelet/pods/37d9e564-dbad-442a-b702-50202d2a82cd/volumes" Oct 30 03:51:02.434386 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:02.434347 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/operator-metrics-auth-probe" event={"ID":"bca46020-25d8-4db3-a374-dd1e5df58b1e","Type":"ContainerStarted","Data":"296469d1f014701e75d8fffc273e88499e3887ced634ca6311c2cc1d42ecc06b"} Oct 30 03:51:02.434855 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:02.434400 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/operator-metrics-auth-probe" event={"ID":"bca46020-25d8-4db3-a374-dd1e5df58b1e","Type":"ContainerStarted","Data":"7648dcc1f5bb7674b4547de1f0dd534d8477ef9d6fb459b2093d59af0208a73b"} Oct 30 03:51:02.473279 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:02.473224 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/operator-metrics-auth-probe" podStartSLOduration=1.473209398 podStartE2EDuration="1.473209398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:02.472037156 +0000 UTC m=+4174.586352129" watchObservedRunningTime="2026-10-30 03:51:02.473209398 +0000 UTC m=+4174.587524373" Oct 30 03:51:07.295068 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:07.295033 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/operator-metrics-auth-probe"] Oct 30 03:51:07.295708 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:07.295215 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/operator-metrics-auth-probe" podUID="bca46020-25d8-4db3-a374-dd1e5df58b1e" containerName="curl" containerID="cri-o://296469d1f014701e75d8fffc273e88499e3887ced634ca6311c2cc1d42ecc06b" gracePeriod=30 Oct 30 03:51:15.910101 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:15.910066 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-prometheus-allowed-18e332341108ead6"] Oct 30 03:51:15.913476 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:15.913457 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-prometheus-allowed-18e332341108ead6" Oct 30 03:51:15.923905 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:15.923884 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-prometheus-allowed-18e332341108ead6" Oct 30 03:51:15.932585 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:15.932562 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-prometheus-allowed-18e332341108ead6"] Oct 30 03:51:16.220042 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:16.219957 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-prometheus-allowed-18e332341108ead6"] Oct 30 03:51:16.223186 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:51:16.223154 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2800745_e6b3_44b0_a20a_c11b893ebcfb.slice/crio-58043d3904a72ed44e9d4b2b75f8e489242ed3dd1fe4b8d399aff414abab3d3f WatchSource:0}: Error finding container 58043d3904a72ed44e9d4b2b75f8e489242ed3dd1fe4b8d399aff414abab3d3f: Status 404 returned error can't find the container with id 58043d3904a72ed44e9d4b2b75f8e489242ed3dd1fe4b8d399aff414abab3d3f Oct 30 03:51:16.225111 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:16.225093 5882 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:51:16.497054 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:16.497020 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-prometheus-allowed-18e332341108ead6" event={"ID":"f2800745-e6b3-44b0-a20a-c11b893ebcfb","Type":"ContainerStarted","Data":"58043d3904a72ed44e9d4b2b75f8e489242ed3dd1fe4b8d399aff414abab3d3f"} Oct 30 03:51:20.518829 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:20.518796 5882 generic.go:356] "Generic (PLEG): container finished" podID="f2800745-e6b3-44b0-a20a-c11b893ebcfb" containerID="ed5ba24a3848cf9cecf66d4ec9df7da747622e6f3cdb8ca41967dfe3d0db793d" exitCode=0 Oct 30 03:51:20.519233 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:20.518861 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-prometheus-allowed-18e332341108ead6" event={"ID":"f2800745-e6b3-44b0-a20a-c11b893ebcfb","Type":"ContainerDied","Data":"ed5ba24a3848cf9cecf66d4ec9df7da747622e6f3cdb8ca41967dfe3d0db793d"} Oct 30 03:51:21.656779 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:21.656752 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-prometheus-allowed-18e332341108ead6" Oct 30 03:51:22.528889 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:22.528858 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-prometheus-allowed-18e332341108ead6" event={"ID":"f2800745-e6b3-44b0-a20a-c11b893ebcfb","Type":"ContainerDied","Data":"58043d3904a72ed44e9d4b2b75f8e489242ed3dd1fe4b8d399aff414abab3d3f"} Oct 30 03:51:22.528889 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:22.528884 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-prometheus-allowed-18e332341108ead6" Oct 30 03:51:22.528889 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:22.528896 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58043d3904a72ed44e9d4b2b75f8e489242ed3dd1fe4b8d399aff414abab3d3f" Oct 30 03:51:23.942501 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:51:23.942459 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:51:28.405011 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:28.404884 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:51:28.421401 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:28.416522 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:51:36.644678 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:36.644640 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq"] Oct 30 03:51:36.645337 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:36.645305 5882 cpu_manager.go:414] "containerMap: removing container" podUID="f2800745-e6b3-44b0-a20a-c11b893ebcfb" containerName="curl" Oct 30 03:51:36.645337 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:36.645331 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2800745-e6b3-44b0-a20a-c11b893ebcfb" containerName="curl" Oct 30 03:51:36.645516 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:36.645467 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2800745-e6b3-44b0-a20a-c11b893ebcfb" containerName="curl" Oct 30 03:51:36.648770 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:36.648751 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" Oct 30 03:51:36.669116 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:36.669084 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq"] Oct 30 03:51:36.684080 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:36.684045 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2574d0b1-493c-4a95-b047-7d28c5495896-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5bddf8ff7c-s97xq\" (UID: \"2574d0b1-493c-4a95-b047-7d28c5495896\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" Oct 30 03:51:36.684250 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:36.684221 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2574d0b1-493c-4a95-b047-7d28c5495896-ta-internal\") pod \"data-science-collector-targetallocator-5bddf8ff7c-s97xq\" (UID: \"2574d0b1-493c-4a95-b047-7d28c5495896\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" Oct 30 03:51:36.684294 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:36.684267 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfd62\" (UniqueName: \"kubernetes.io/projected/2574d0b1-493c-4a95-b047-7d28c5495896-kube-api-access-mfd62\") pod \"data-science-collector-targetallocator-5bddf8ff7c-s97xq\" (UID: \"2574d0b1-493c-4a95-b047-7d28c5495896\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" Oct 30 03:51:36.785520 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:36.785485 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2574d0b1-493c-4a95-b047-7d28c5495896-ta-internal\") pod \"data-science-collector-targetallocator-5bddf8ff7c-s97xq\" (UID: \"2574d0b1-493c-4a95-b047-7d28c5495896\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" Oct 30 03:51:36.785723 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:36.785530 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfd62\" (UniqueName: \"kubernetes.io/projected/2574d0b1-493c-4a95-b047-7d28c5495896-kube-api-access-mfd62\") pod \"data-science-collector-targetallocator-5bddf8ff7c-s97xq\" (UID: \"2574d0b1-493c-4a95-b047-7d28c5495896\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" Oct 30 03:51:36.785723 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:36.785597 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2574d0b1-493c-4a95-b047-7d28c5495896-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5bddf8ff7c-s97xq\" (UID: \"2574d0b1-493c-4a95-b047-7d28c5495896\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" Oct 30 03:51:36.786316 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:36.786294 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2574d0b1-493c-4a95-b047-7d28c5495896-ta-internal\") pod \"data-science-collector-targetallocator-5bddf8ff7c-s97xq\" (UID: \"2574d0b1-493c-4a95-b047-7d28c5495896\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" Oct 30 03:51:36.788469 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:36.788434 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2574d0b1-493c-4a95-b047-7d28c5495896-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5bddf8ff7c-s97xq\" (UID: \"2574d0b1-493c-4a95-b047-7d28c5495896\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" Oct 30 03:51:36.808450 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:36.808427 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfd62\" (UniqueName: \"kubernetes.io/projected/2574d0b1-493c-4a95-b047-7d28c5495896-kube-api-access-mfd62\") pod \"data-science-collector-targetallocator-5bddf8ff7c-s97xq\" (UID: \"2574d0b1-493c-4a95-b047-7d28c5495896\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" Oct 30 03:51:36.964001 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:36.963917 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" Oct 30 03:51:37.252248 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.252216 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq"] Oct 30 03:51:37.257735 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.257700 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:51:37.257838 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.257759 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:51:37.328848 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:51:37.328811 5882 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbca46020_25d8_4db3_a374_dd1e5df58b1e.slice/crio-296469d1f014701e75d8fffc273e88499e3887ced634ca6311c2cc1d42ecc06b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbca46020_25d8_4db3_a374_dd1e5df58b1e.slice/crio-conmon-296469d1f014701e75d8fffc273e88499e3887ced634ca6311c2cc1d42ecc06b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 03:51:37.427710 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.427682 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/operator-metrics-auth-probe" Oct 30 03:51:37.492805 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.492777 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bca46020-25d8-4db3-a374-dd1e5df58b1e-metrics-ca\" (UniqueName: \"kubernetes.io/configmap/bca46020-25d8-4db3-a374-dd1e5df58b1e-metrics-ca\") pod \"bca46020-25d8-4db3-a374-dd1e5df58b1e\" (UID: \"bca46020-25d8-4db3-a374-dd1e5df58b1e\") " Oct 30 03:51:37.492987 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.492813 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bca46020-25d8-4db3-a374-dd1e5df58b1e-kube-api-access-mwj9b\" (UniqueName: \"kubernetes.io/projected/bca46020-25d8-4db3-a374-dd1e5df58b1e-kube-api-access-mwj9b\") pod \"bca46020-25d8-4db3-a374-dd1e5df58b1e\" (UID: \"bca46020-25d8-4db3-a374-dd1e5df58b1e\") " Oct 30 03:51:37.493183 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.493159 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca46020-25d8-4db3-a374-dd1e5df58b1e-metrics-ca" pod "bca46020-25d8-4db3-a374-dd1e5df58b1e" (UID: "bca46020-25d8-4db3-a374-dd1e5df58b1e"). InnerVolumeSpecName "metrics-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:51:37.494872 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.494830 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca46020-25d8-4db3-a374-dd1e5df58b1e-kube-api-access-mwj9b" pod "bca46020-25d8-4db3-a374-dd1e5df58b1e" (UID: "bca46020-25d8-4db3-a374-dd1e5df58b1e"). InnerVolumeSpecName "kube-api-access-mwj9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:51:37.593690 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.593597 5882 reconciler_common.go:299] "Volume detached for volume \"metrics-ca\" (UniqueName: \"kubernetes.io/configmap/bca46020-25d8-4db3-a374-dd1e5df58b1e-metrics-ca\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:51:37.593690 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.593646 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwj9b\" (UniqueName: \"kubernetes.io/projected/bca46020-25d8-4db3-a374-dd1e5df58b1e-kube-api-access-mwj9b\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:51:37.602322 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.602292 5882 generic.go:356] "Generic (PLEG): container finished" podID="bca46020-25d8-4db3-a374-dd1e5df58b1e" containerID="296469d1f014701e75d8fffc273e88499e3887ced634ca6311c2cc1d42ecc06b" exitCode=137 Oct 30 03:51:37.602468 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.602336 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/operator-metrics-auth-probe" Oct 30 03:51:37.602468 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.602346 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/operator-metrics-auth-probe" event={"ID":"bca46020-25d8-4db3-a374-dd1e5df58b1e","Type":"ContainerDied","Data":"296469d1f014701e75d8fffc273e88499e3887ced634ca6311c2cc1d42ecc06b"} Oct 30 03:51:37.602468 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.602393 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/operator-metrics-auth-probe" event={"ID":"bca46020-25d8-4db3-a374-dd1e5df58b1e","Type":"ContainerDied","Data":"7648dcc1f5bb7674b4547de1f0dd534d8477ef9d6fb459b2093d59af0208a73b"} Oct 30 03:51:37.602468 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.602415 5882 scope.go:122] "RemoveContainer" containerID="296469d1f014701e75d8fffc273e88499e3887ced634ca6311c2cc1d42ecc06b" Oct 30 03:51:37.604631 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.604587 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" event={"ID":"2574d0b1-493c-4a95-b047-7d28c5495896","Type":"ContainerStarted","Data":"266f98e92335261b581ed8ab36586cb081d5e8c2145113e89cf5babdab4b4fdf"} Oct 30 03:51:37.604742 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.604657 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" event={"ID":"2574d0b1-493c-4a95-b047-7d28c5495896","Type":"ContainerStarted","Data":"61ead3dfc0ccf1a67052b9a8b8ebe96ce50d0b56f07c69002b0ff1068943bd20"} Oct 30 03:51:37.604742 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.604682 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" Oct 30 03:51:37.606268 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.606242 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" podUID="2574d0b1-493c-4a95-b047-7d28c5495896" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.71:8080/readyz\": dial tcp 10.134.0.71:8080: connect: connection refused" Oct 30 03:51:37.614163 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.614145 5882 scope.go:122] "RemoveContainer" containerID="296469d1f014701e75d8fffc273e88499e3887ced634ca6311c2cc1d42ecc06b" Oct 30 03:51:37.614517 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:51:37.614494 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296469d1f014701e75d8fffc273e88499e3887ced634ca6311c2cc1d42ecc06b\": container with ID starting with 296469d1f014701e75d8fffc273e88499e3887ced634ca6311c2cc1d42ecc06b not found: ID does not exist" containerID="296469d1f014701e75d8fffc273e88499e3887ced634ca6311c2cc1d42ecc06b" Oct 30 03:51:37.614574 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.614526 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296469d1f014701e75d8fffc273e88499e3887ced634ca6311c2cc1d42ecc06b"} err="failed to get container status \"296469d1f014701e75d8fffc273e88499e3887ced634ca6311c2cc1d42ecc06b\": rpc error: code = NotFound desc = could not find container \"296469d1f014701e75d8fffc273e88499e3887ced634ca6311c2cc1d42ecc06b\": container with ID starting with 296469d1f014701e75d8fffc273e88499e3887ced634ca6311c2cc1d42ecc06b not found: ID does not exist" Oct 30 03:51:37.656485 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.656439 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" podStartSLOduration=1.656426351 podStartE2EDuration="1.656426351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:37.655730796 +0000 UTC m=+4209.770045772" watchObservedRunningTime="2026-10-30 03:51:37.656426351 +0000 UTC m=+4209.770741329" Oct 30 03:51:37.690649 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.690601 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/operator-metrics-auth-probe"] Oct 30 03:51:37.691250 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:37.691230 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/operator-metrics-auth-probe"] Oct 30 03:51:38.009609 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:38.009575 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bca46020-25d8-4db3-a374-dd1e5df58b1e" path="/var/lib/kubelet/pods/bca46020-25d8-4db3-a374-dd1e5df58b1e/volumes" Oct 30 03:51:38.611521 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:38.611487 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" Oct 30 03:51:38.681242 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:38.681211 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm"] Oct 30 03:51:38.681713 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:38.681409 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" podUID="7e91285f-356d-4cc5-9e2c-0e9ca7973d40" containerName="ta-container" containerID="cri-o://e9245fb7bd1cc06ac24f3dc9c3b189ff10c2739345cda700eccc846d44e2c541" gracePeriod=30 Oct 30 03:51:38.924550 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:38.924525 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" Oct 30 03:51:39.004939 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:39.004911 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-ta-internal\" (UniqueName: \"kubernetes.io/configmap/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-ta-internal\") pod \"7e91285f-356d-4cc5-9e2c-0e9ca7973d40\" (UID: \"7e91285f-356d-4cc5-9e2c-0e9ca7973d40\") " Oct 30 03:51:39.005133 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:39.004984 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-data-science-collector-ta-server-cert\") pod \"7e91285f-356d-4cc5-9e2c-0e9ca7973d40\" (UID: \"7e91285f-356d-4cc5-9e2c-0e9ca7973d40\") " Oct 30 03:51:39.005133 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:39.005018 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-kube-api-access-5lp6l\" (UniqueName: \"kubernetes.io/projected/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-kube-api-access-5lp6l\") pod \"7e91285f-356d-4cc5-9e2c-0e9ca7973d40\" (UID: \"7e91285f-356d-4cc5-9e2c-0e9ca7973d40\") " Oct 30 03:51:39.005290 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:39.005261 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-ta-internal" pod "7e91285f-356d-4cc5-9e2c-0e9ca7973d40" (UID: "7e91285f-356d-4cc5-9e2c-0e9ca7973d40"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:51:39.007191 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:39.007158 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-data-science-collector-ta-server-cert" pod "7e91285f-356d-4cc5-9e2c-0e9ca7973d40" (UID: "7e91285f-356d-4cc5-9e2c-0e9ca7973d40"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:51:39.007191 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:39.007174 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-kube-api-access-5lp6l" pod "7e91285f-356d-4cc5-9e2c-0e9ca7973d40" (UID: "7e91285f-356d-4cc5-9e2c-0e9ca7973d40"). InnerVolumeSpecName "kube-api-access-5lp6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:51:39.106102 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:39.106051 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:51:39.106102 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:39.106098 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:51:39.106102 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:39.106109 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lp6l\" (UniqueName: \"kubernetes.io/projected/7e91285f-356d-4cc5-9e2c-0e9ca7973d40-kube-api-access-5lp6l\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:51:39.616117 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:39.616080 5882 generic.go:356] "Generic (PLEG): container finished" podID="7e91285f-356d-4cc5-9e2c-0e9ca7973d40" containerID="e9245fb7bd1cc06ac24f3dc9c3b189ff10c2739345cda700eccc846d44e2c541" exitCode=0 Oct 30 03:51:39.616306 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:39.616130 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" Oct 30 03:51:39.616306 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:39.616166 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" event={"ID":"7e91285f-356d-4cc5-9e2c-0e9ca7973d40","Type":"ContainerDied","Data":"e9245fb7bd1cc06ac24f3dc9c3b189ff10c2739345cda700eccc846d44e2c541"} Oct 30 03:51:39.616306 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:39.616223 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm" event={"ID":"7e91285f-356d-4cc5-9e2c-0e9ca7973d40","Type":"ContainerDied","Data":"c6a1cb9d77f8d4962177b6155d1cd3e62ea6366b1c1786e28c7c0fa202fae5b8"} Oct 30 03:51:39.616306 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:39.616251 5882 scope.go:122] "RemoveContainer" containerID="e9245fb7bd1cc06ac24f3dc9c3b189ff10c2739345cda700eccc846d44e2c541" Oct 30 03:51:39.625040 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:39.625021 5882 scope.go:122] "RemoveContainer" containerID="e9245fb7bd1cc06ac24f3dc9c3b189ff10c2739345cda700eccc846d44e2c541" Oct 30 03:51:39.625311 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:51:39.625291 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9245fb7bd1cc06ac24f3dc9c3b189ff10c2739345cda700eccc846d44e2c541\": container with ID starting with e9245fb7bd1cc06ac24f3dc9c3b189ff10c2739345cda700eccc846d44e2c541 not found: ID does not exist" containerID="e9245fb7bd1cc06ac24f3dc9c3b189ff10c2739345cda700eccc846d44e2c541" Oct 30 03:51:39.625359 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:39.625318 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9245fb7bd1cc06ac24f3dc9c3b189ff10c2739345cda700eccc846d44e2c541"} err="failed to get container status \"e9245fb7bd1cc06ac24f3dc9c3b189ff10c2739345cda700eccc846d44e2c541\": rpc error: code = NotFound desc = could not find container \"e9245fb7bd1cc06ac24f3dc9c3b189ff10c2739345cda700eccc846d44e2c541\": container with ID starting with e9245fb7bd1cc06ac24f3dc9c3b189ff10c2739345cda700eccc846d44e2c541 not found: ID does not exist" Oct 30 03:51:39.664984 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:39.664959 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm"] Oct 30 03:51:39.676441 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:39.676413 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-mcqqm"] Oct 30 03:51:40.009073 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:40.009042 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e91285f-356d-4cc5-9e2c-0e9ca7973d40" path="/var/lib/kubelet/pods/7e91285f-356d-4cc5-9e2c-0e9ca7973d40/volumes" Oct 30 03:51:48.616508 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:48.616471 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-prefill-allowed-18e3323bad47956e"] Oct 30 03:51:48.616915 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:48.616856 5882 cpu_manager.go:414] "containerMap: removing container" podUID="bca46020-25d8-4db3-a374-dd1e5df58b1e" containerName="curl" Oct 30 03:51:48.616915 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:48.616868 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bca46020-25d8-4db3-a374-dd1e5df58b1e" containerName="curl" Oct 30 03:51:48.616915 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:48.616882 5882 cpu_manager.go:414] "containerMap: removing container" podUID="7e91285f-356d-4cc5-9e2c-0e9ca7973d40" containerName="ta-container" Oct 30 03:51:48.616915 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:48.616886 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e91285f-356d-4cc5-9e2c-0e9ca7973d40" containerName="ta-container" Oct 30 03:51:48.617044 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:48.616945 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="bca46020-25d8-4db3-a374-dd1e5df58b1e" containerName="curl" Oct 30 03:51:48.617044 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:48.616955 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e91285f-356d-4cc5-9e2c-0e9ca7973d40" containerName="ta-container" Oct 30 03:51:48.622566 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:48.622545 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-prefill-allowed-18e3323bad47956e" Oct 30 03:51:48.633818 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:48.633798 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-prefill-allowed-18e3323bad47956e" Oct 30 03:51:48.638299 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:48.638277 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-prefill-allowed-18e3323bad47956e"] Oct 30 03:51:48.928321 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:48.928227 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-prefill-allowed-18e3323bad47956e"] Oct 30 03:51:48.931106 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:51:48.931068 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec856d2e_d84d_4b96_aebd_4c69e2662633.slice/crio-09cc59f7e3889676cc2f8cce8a679c69c845ecf178da2ea259c86b84dc20196a WatchSource:0}: Error finding container 09cc59f7e3889676cc2f8cce8a679c69c845ecf178da2ea259c86b84dc20196a: Status 404 returned error can't find the container with id 09cc59f7e3889676cc2f8cce8a679c69c845ecf178da2ea259c86b84dc20196a Oct 30 03:51:49.666443 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:49.666403 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-prefill-allowed-18e3323bad47956e" event={"ID":"ec856d2e-d84d-4b96-aebd-4c69e2662633","Type":"ContainerStarted","Data":"498842a5e30a1c5ad9dac1b8e9a38088c6b5c726170cba53ea506429afe187c2"} Oct 30 03:51:49.666924 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:49.666452 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-prefill-allowed-18e3323bad47956e" event={"ID":"ec856d2e-d84d-4b96-aebd-4c69e2662633","Type":"ContainerStarted","Data":"09cc59f7e3889676cc2f8cce8a679c69c845ecf178da2ea259c86b84dc20196a"} Oct 30 03:51:49.703306 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:49.703253 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-prefill-allowed-18e3323bad47956e" podStartSLOduration=1.703239993 podStartE2EDuration="1.703239993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:49.702912491 +0000 UTC m=+4221.817227466" watchObservedRunningTime="2026-10-30 03:51:49.703239993 +0000 UTC m=+4221.817554968" Oct 30 03:51:53.942385 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:51:53.942353 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:51:59.711669 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:59.711634 5882 generic.go:356] "Generic (PLEG): container finished" podID="ec856d2e-d84d-4b96-aebd-4c69e2662633" containerID="498842a5e30a1c5ad9dac1b8e9a38088c6b5c726170cba53ea506429afe187c2" exitCode=0 Oct 30 03:51:59.712057 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:51:59.711690 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-prefill-allowed-18e3323bad47956e" event={"ID":"ec856d2e-d84d-4b96-aebd-4c69e2662633","Type":"ContainerDied","Data":"498842a5e30a1c5ad9dac1b8e9a38088c6b5c726170cba53ea506429afe187c2"} Oct 30 03:52:00.847953 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:00.847928 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-prefill-allowed-18e3323bad47956e" Oct 30 03:52:01.727290 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:01.727258 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-prefill-allowed-18e3323bad47956e" Oct 30 03:52:01.727474 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:01.727296 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-prefill-allowed-18e3323bad47956e" event={"ID":"ec856d2e-d84d-4b96-aebd-4c69e2662633","Type":"ContainerDied","Data":"09cc59f7e3889676cc2f8cce8a679c69c845ecf178da2ea259c86b84dc20196a"} Oct 30 03:52:01.727474 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:01.727330 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09cc59f7e3889676cc2f8cce8a679c69c845ecf178da2ea259c86b84dc20196a" Oct 30 03:52:02.847434 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:02.847400 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-worker-allowed-18e3323efdca6819"] Oct 30 03:52:02.847862 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:02.847788 5882 cpu_manager.go:414] "containerMap: removing container" podUID="ec856d2e-d84d-4b96-aebd-4c69e2662633" containerName="curl" Oct 30 03:52:02.847862 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:02.847800 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec856d2e-d84d-4b96-aebd-4c69e2662633" containerName="curl" Oct 30 03:52:02.847947 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:02.847877 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec856d2e-d84d-4b96-aebd-4c69e2662633" containerName="curl" Oct 30 03:52:02.851571 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:02.851550 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-worker-allowed-18e3323efdca6819" Oct 30 03:52:02.862568 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:02.862547 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-worker-allowed-18e3323efdca6819" Oct 30 03:52:02.866048 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:02.865779 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-worker-allowed-18e3323efdca6819"] Oct 30 03:52:03.152494 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:03.152423 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-worker-allowed-18e3323efdca6819"] Oct 30 03:52:03.156093 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:52:03.156067 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4140e1d6_b13d_4307_b299_6a099f1f499a.slice/crio-83a5e2ee64fcb5804f4c2b6ecf30ddb66689f4a7ce4e49da91745c566f122bf8 WatchSource:0}: Error finding container 83a5e2ee64fcb5804f4c2b6ecf30ddb66689f4a7ce4e49da91745c566f122bf8: Status 404 returned error can't find the container with id 83a5e2ee64fcb5804f4c2b6ecf30ddb66689f4a7ce4e49da91745c566f122bf8 Oct 30 03:52:03.738673 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:03.738632 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-worker-allowed-18e3323efdca6819" event={"ID":"4140e1d6-b13d-4307-b299-6a099f1f499a","Type":"ContainerStarted","Data":"fc198c0f2c8336bfd96d2fec338675aa7a5684fd021a164006ff44f169f51b52"} Oct 30 03:52:03.738673 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:03.738680 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-worker-allowed-18e3323efdca6819" event={"ID":"4140e1d6-b13d-4307-b299-6a099f1f499a","Type":"ContainerStarted","Data":"83a5e2ee64fcb5804f4c2b6ecf30ddb66689f4a7ce4e49da91745c566f122bf8"} Oct 30 03:52:13.786095 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:13.786063 5882 generic.go:356] "Generic (PLEG): container finished" podID="4140e1d6-b13d-4307-b299-6a099f1f499a" containerID="fc198c0f2c8336bfd96d2fec338675aa7a5684fd021a164006ff44f169f51b52" exitCode=0 Oct 30 03:52:13.786514 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:13.786122 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-worker-allowed-18e3323efdca6819" event={"ID":"4140e1d6-b13d-4307-b299-6a099f1f499a","Type":"ContainerDied","Data":"fc198c0f2c8336bfd96d2fec338675aa7a5684fd021a164006ff44f169f51b52"} Oct 30 03:52:14.915173 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:14.915147 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-worker-allowed-18e3323efdca6819" Oct 30 03:52:15.124044 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:15.123964 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-leader-allowed-18e33241d5b86fde"] Oct 30 03:52:15.124353 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:15.124340 5882 cpu_manager.go:414] "containerMap: removing container" podUID="4140e1d6-b13d-4307-b299-6a099f1f499a" containerName="curl" Oct 30 03:52:15.124450 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:15.124355 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4140e1d6-b13d-4307-b299-6a099f1f499a" containerName="curl" Oct 30 03:52:15.124450 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:15.124417 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4140e1d6-b13d-4307-b299-6a099f1f499a" containerName="curl" Oct 30 03:52:15.127469 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:15.127453 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-leader-allowed-18e33241d5b86fde" Oct 30 03:52:15.138094 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:15.138075 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-leader-allowed-18e33241d5b86fde" Oct 30 03:52:15.151056 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:15.150998 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-leader-allowed-18e33241d5b86fde"] Oct 30 03:52:15.430384 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:15.430310 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-leader-allowed-18e33241d5b86fde"] Oct 30 03:52:15.433694 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:52:15.433668 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2aacfaed_75d5_4217_9f88_caa00e952522.slice/crio-02d850840d9e4f0c9646bb73ff377fe722de58e36a5e8e5d504b19cd87a7965a WatchSource:0}: Error finding container 02d850840d9e4f0c9646bb73ff377fe722de58e36a5e8e5d504b19cd87a7965a: Status 404 returned error can't find the container with id 02d850840d9e4f0c9646bb73ff377fe722de58e36a5e8e5d504b19cd87a7965a Oct 30 03:52:15.796830 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:15.796791 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-leader-allowed-18e33241d5b86fde" event={"ID":"2aacfaed-75d5-4217-9f88-caa00e952522","Type":"ContainerStarted","Data":"290a9ad22f964bfcdb29b58308f5ad18d2b766ecd22b3b27120f18a8c2f2b34d"} Oct 30 03:52:15.796830 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:15.796835 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-leader-allowed-18e33241d5b86fde" event={"ID":"2aacfaed-75d5-4217-9f88-caa00e952522","Type":"ContainerStarted","Data":"02d850840d9e4f0c9646bb73ff377fe722de58e36a5e8e5d504b19cd87a7965a"} Oct 30 03:52:15.798023 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:15.798004 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-worker-allowed-18e3323efdca6819" Oct 30 03:52:15.798132 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:15.798021 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-worker-allowed-18e3323efdca6819" event={"ID":"4140e1d6-b13d-4307-b299-6a099f1f499a","Type":"ContainerDied","Data":"83a5e2ee64fcb5804f4c2b6ecf30ddb66689f4a7ce4e49da91745c566f122bf8"} Oct 30 03:52:15.798132 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:15.798046 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83a5e2ee64fcb5804f4c2b6ecf30ddb66689f4a7ce4e49da91745c566f122bf8" Oct 30 03:52:23.943103 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:52:23.943066 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:52:25.849042 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:25.848958 5882 generic.go:356] "Generic (PLEG): container finished" podID="2aacfaed-75d5-4217-9f88-caa00e952522" containerID="290a9ad22f964bfcdb29b58308f5ad18d2b766ecd22b3b27120f18a8c2f2b34d" exitCode=0 Oct 30 03:52:25.849401 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:25.849032 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-leader-allowed-18e33241d5b86fde" event={"ID":"2aacfaed-75d5-4217-9f88-caa00e952522","Type":"ContainerDied","Data":"290a9ad22f964bfcdb29b58308f5ad18d2b766ecd22b3b27120f18a8c2f2b34d"} Oct 30 03:52:26.985132 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:26.985103 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-leader-allowed-18e33241d5b86fde" Oct 30 03:52:27.321479 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:27.321439 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e33244b15ff220"] Oct 30 03:52:27.321975 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:27.321952 5882 cpu_manager.go:414] "containerMap: removing container" podUID="2aacfaed-75d5-4217-9f88-caa00e952522" containerName="curl" Oct 30 03:52:27.321975 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:27.321974 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aacfaed-75d5-4217-9f88-caa00e952522" containerName="curl" Oct 30 03:52:27.322157 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:27.322092 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aacfaed-75d5-4217-9f88-caa00e952522" containerName="curl" Oct 30 03:52:27.325200 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:27.325179 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e33244b15ff220" Oct 30 03:52:27.335772 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:27.335756 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e33244b15ff220" Oct 30 03:52:27.341071 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:27.341050 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e33244b15ff220"] Oct 30 03:52:27.627455 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:27.627381 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e33244b15ff220"] Oct 30 03:52:27.630369 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:52:27.630340 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf30bda8_aefa_4b4b_89dc_edad5acb7850.slice/crio-1608257d3cbd6978b44fa800cf09b5bc58ce183e5cfbf4cfc6888d3d2ddcba42 WatchSource:0}: Error finding container 1608257d3cbd6978b44fa800cf09b5bc58ce183e5cfbf4cfc6888d3d2ddcba42: Status 404 returned error can't find the container with id 1608257d3cbd6978b44fa800cf09b5bc58ce183e5cfbf4cfc6888d3d2ddcba42 Oct 30 03:52:27.859072 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:27.859042 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-leader-allowed-18e33241d5b86fde" Oct 30 03:52:27.859262 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:27.859038 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-leader-allowed-18e33241d5b86fde" event={"ID":"2aacfaed-75d5-4217-9f88-caa00e952522","Type":"ContainerDied","Data":"02d850840d9e4f0c9646bb73ff377fe722de58e36a5e8e5d504b19cd87a7965a"} Oct 30 03:52:27.859262 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:27.859158 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02d850840d9e4f0c9646bb73ff377fe722de58e36a5e8e5d504b19cd87a7965a" Oct 30 03:52:27.860672 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:27.860643 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e33244b15ff220" event={"ID":"bf30bda8-aefa-4b4b-89dc-edad5acb7850","Type":"ContainerStarted","Data":"58ace56a3734018779814aec900038b296e31bd506ff400bff57248fa8114565"} Oct 30 03:52:27.860785 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:27.860682 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e33244b15ff220" event={"ID":"bf30bda8-aefa-4b4b-89dc-edad5acb7850","Type":"ContainerStarted","Data":"1608257d3cbd6978b44fa800cf09b5bc58ce183e5cfbf4cfc6888d3d2ddcba42"} Oct 30 03:52:27.897440 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:27.897350 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e33244b15ff220" podStartSLOduration=0.897336663 podStartE2EDuration="897.336663ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:52:27.896857274 +0000 UTC m=+4260.011172243" watchObservedRunningTime="2026-10-30 03:52:27.897336663 +0000 UTC m=+4260.011651637" Oct 30 03:52:37.906903 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:37.906860 5882 generic.go:356] "Generic (PLEG): container finished" podID="bf30bda8-aefa-4b4b-89dc-edad5acb7850" containerID="58ace56a3734018779814aec900038b296e31bd506ff400bff57248fa8114565" exitCode=0 Oct 30 03:52:37.907272 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:37.906932 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e33244b15ff220" event={"ID":"bf30bda8-aefa-4b4b-89dc-edad5acb7850","Type":"ContainerDied","Data":"58ace56a3734018779814aec900038b296e31bd506ff400bff57248fa8114565"} Oct 30 03:52:39.035930 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:39.035907 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e33244b15ff220" Oct 30 03:52:39.536003 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:39.535969 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e332478850b604"] Oct 30 03:52:39.536320 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:39.536309 5882 cpu_manager.go:414] "containerMap: removing container" podUID="bf30bda8-aefa-4b4b-89dc-edad5acb7850" containerName="curl" Oct 30 03:52:39.536381 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:39.536322 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf30bda8-aefa-4b4b-89dc-edad5acb7850" containerName="curl" Oct 30 03:52:39.536416 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:39.536404 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf30bda8-aefa-4b4b-89dc-edad5acb7850" containerName="curl" Oct 30 03:52:39.539336 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:39.539317 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e332478850b604" Oct 30 03:52:39.549797 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:39.549778 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e332478850b604" Oct 30 03:52:39.556095 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:39.556070 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e332478850b604"] Oct 30 03:52:39.840238 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:39.840168 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e332478850b604"] Oct 30 03:52:39.843564 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:52:39.843493 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29e7a712_9985_4dd1_a66a_4ec81beae6b1.slice/crio-9e989c11587ad5d0f56e6f5cbf3dcccdc3bd30587be2ee7206ed8ab1e9db45d8 WatchSource:0}: Error finding container 9e989c11587ad5d0f56e6f5cbf3dcccdc3bd30587be2ee7206ed8ab1e9db45d8: Status 404 returned error can't find the container with id 9e989c11587ad5d0f56e6f5cbf3dcccdc3bd30587be2ee7206ed8ab1e9db45d8 Oct 30 03:52:39.917431 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:39.917402 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e332478850b604" event={"ID":"29e7a712-9985-4dd1-a66a-4ec81beae6b1","Type":"ContainerStarted","Data":"3ed7ab8720efc6aced60992b54743ea63d9c790fcff839b4d2c2fb9d8b38c022"} Oct 30 03:52:39.917594 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:39.917447 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e332478850b604" event={"ID":"29e7a712-9985-4dd1-a66a-4ec81beae6b1","Type":"ContainerStarted","Data":"9e989c11587ad5d0f56e6f5cbf3dcccdc3bd30587be2ee7206ed8ab1e9db45d8"} Oct 30 03:52:39.918594 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:39.918576 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e33244b15ff220" Oct 30 03:52:39.918700 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:39.918600 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e33244b15ff220" event={"ID":"bf30bda8-aefa-4b4b-89dc-edad5acb7850","Type":"ContainerDied","Data":"1608257d3cbd6978b44fa800cf09b5bc58ce183e5cfbf4cfc6888d3d2ddcba42"} Oct 30 03:52:39.918700 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:39.918649 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1608257d3cbd6978b44fa800cf09b5bc58ce183e5cfbf4cfc6888d3d2ddcba42" Oct 30 03:52:39.955820 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:39.955769 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e332478850b604" podStartSLOduration=0.955757733 podStartE2EDuration="955.757733ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:52:39.954635038 +0000 UTC m=+4272.068950004" watchObservedRunningTime="2026-10-30 03:52:39.955757733 +0000 UTC m=+4272.070072753" Oct 30 03:52:49.965929 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:49.965897 5882 generic.go:356] "Generic (PLEG): container finished" podID="29e7a712-9985-4dd1-a66a-4ec81beae6b1" containerID="3ed7ab8720efc6aced60992b54743ea63d9c790fcff839b4d2c2fb9d8b38c022" exitCode=0 Oct 30 03:52:49.966328 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:49.965976 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e332478850b604" event={"ID":"29e7a712-9985-4dd1-a66a-4ec81beae6b1","Type":"ContainerDied","Data":"3ed7ab8720efc6aced60992b54743ea63d9c790fcff839b4d2c2fb9d8b38c022"} Oct 30 03:52:51.096610 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:51.096583 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e332478850b604" Oct 30 03:52:51.745502 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:51.745466 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e3324a6073205f"] Oct 30 03:52:51.745871 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:51.745858 5882 cpu_manager.go:414] "containerMap: removing container" podUID="29e7a712-9985-4dd1-a66a-4ec81beae6b1" containerName="curl" Oct 30 03:52:51.745934 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:51.745871 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e7a712-9985-4dd1-a66a-4ec81beae6b1" containerName="curl" Oct 30 03:52:51.745993 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:51.745935 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e7a712-9985-4dd1-a66a-4ec81beae6b1" containerName="curl" Oct 30 03:52:51.749037 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:51.749019 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e3324a6073205f" Oct 30 03:52:51.759517 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:51.759498 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e3324a6073205f" Oct 30 03:52:51.763045 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:51.763019 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e3324a6073205f"] Oct 30 03:52:51.976921 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:51.976892 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e332478850b604" Oct 30 03:52:51.976921 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:51.976904 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e332478850b604" event={"ID":"29e7a712-9985-4dd1-a66a-4ec81beae6b1","Type":"ContainerDied","Data":"9e989c11587ad5d0f56e6f5cbf3dcccdc3bd30587be2ee7206ed8ab1e9db45d8"} Oct 30 03:52:51.977124 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:51.976941 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e989c11587ad5d0f56e6f5cbf3dcccdc3bd30587be2ee7206ed8ab1e9db45d8" Oct 30 03:52:52.048665 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:52.048582 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e3324a6073205f"] Oct 30 03:52:52.052571 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:52:52.052542 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60629d32_8098_4865_b472_c2ebb38effb9.slice/crio-fee913ff130878b87e3cf2e6f34831bcadc5227834916ac66a710863d68cf714 WatchSource:0}: Error finding container fee913ff130878b87e3cf2e6f34831bcadc5227834916ac66a710863d68cf714: Status 404 returned error can't find the container with id fee913ff130878b87e3cf2e6f34831bcadc5227834916ac66a710863d68cf714 Oct 30 03:52:52.983033 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:52.983002 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e3324a6073205f" event={"ID":"60629d32-8098-4865-b472-c2ebb38effb9","Type":"ContainerStarted","Data":"d7b67aea388f0a1676849178e92461484ce6a443e7f4dc2f445bc18874f75d62"} Oct 30 03:52:52.983442 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:52.983042 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e3324a6073205f" event={"ID":"60629d32-8098-4865-b472-c2ebb38effb9","Type":"ContainerStarted","Data":"fee913ff130878b87e3cf2e6f34831bcadc5227834916ac66a710863d68cf714"} Oct 30 03:52:53.020075 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:52:53.020027 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e3324a6073205f" podStartSLOduration=2.020014704 podStartE2EDuration="2.020014704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:52:53.01964887 +0000 UTC m=+4285.133963845" watchObservedRunningTime="2026-10-30 03:52:53.020014704 +0000 UTC m=+4285.134329678" Oct 30 03:52:53.943154 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:52:53.943116 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:53:03.033406 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:03.033374 5882 generic.go:356] "Generic (PLEG): container finished" podID="60629d32-8098-4865-b472-c2ebb38effb9" containerID="d7b67aea388f0a1676849178e92461484ce6a443e7f4dc2f445bc18874f75d62" exitCode=0 Oct 30 03:53:03.033798 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:03.033443 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e3324a6073205f" event={"ID":"60629d32-8098-4865-b472-c2ebb38effb9","Type":"ContainerDied","Data":"d7b67aea388f0a1676849178e92461484ce6a443e7f4dc2f445bc18874f75d62"} Oct 30 03:53:04.167369 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:04.167345 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e3324a6073205f" Oct 30 03:53:05.043174 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:05.043140 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e3324a6073205f" event={"ID":"60629d32-8098-4865-b472-c2ebb38effb9","Type":"ContainerDied","Data":"fee913ff130878b87e3cf2e6f34831bcadc5227834916ac66a710863d68cf714"} Oct 30 03:53:05.043174 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:05.043176 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fee913ff130878b87e3cf2e6f34831bcadc5227834916ac66a710863d68cf714" Oct 30 03:53:05.043382 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:05.043149 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e3324a6073205f" Oct 30 03:53:05.963182 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:05.963148 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3324db09a84c9"] Oct 30 03:53:05.963772 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:05.963722 5882 cpu_manager.go:414] "containerMap: removing container" podUID="60629d32-8098-4865-b472-c2ebb38effb9" containerName="curl" Oct 30 03:53:05.963772 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:05.963744 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60629d32-8098-4865-b472-c2ebb38effb9" containerName="curl" Oct 30 03:53:05.963920 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:05.963886 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="60629d32-8098-4865-b472-c2ebb38effb9" containerName="curl" Oct 30 03:53:05.966817 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:05.966797 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3324db09a84c9" Oct 30 03:53:05.977366 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:05.977348 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3324db09a84c9" Oct 30 03:53:05.980728 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:05.980706 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3324db09a84c9"] Oct 30 03:53:06.269521 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:06.269490 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3324db09a84c9"] Oct 30 03:53:06.272640 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:53:06.272596 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd38e1a10_1192_418b_bfbb_65fa776781fe.slice/crio-4a023c2b8da6b02f1d36b091c7a7d250b52be90a0f3aa194bc6b477de959b349 WatchSource:0}: Error finding container 4a023c2b8da6b02f1d36b091c7a7d250b52be90a0f3aa194bc6b477de959b349: Status 404 returned error can't find the container with id 4a023c2b8da6b02f1d36b091c7a7d250b52be90a0f3aa194bc6b477de959b349 Oct 30 03:53:07.057455 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:07.057413 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3324db09a84c9" event={"ID":"d38e1a10-1192-418b-bfbb-65fa776781fe","Type":"ContainerStarted","Data":"73bd312e8f7c08cddcdd4c17f3edaa50430d9323558b709cc821db7d4853fd7b"} Oct 30 03:53:07.057955 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:07.057462 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3324db09a84c9" event={"ID":"d38e1a10-1192-418b-bfbb-65fa776781fe","Type":"ContainerStarted","Data":"4a023c2b8da6b02f1d36b091c7a7d250b52be90a0f3aa194bc6b477de959b349"} Oct 30 03:53:07.095031 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:07.094973 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3324db09a84c9" podStartSLOduration=2.094957276 podStartE2EDuration="2.094957276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:53:07.094263671 +0000 UTC m=+4299.208578648" watchObservedRunningTime="2026-10-30 03:53:07.094957276 +0000 UTC m=+4299.209272252" Oct 30 03:53:17.103585 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:17.103548 5882 generic.go:356] "Generic (PLEG): container finished" podID="d38e1a10-1192-418b-bfbb-65fa776781fe" containerID="73bd312e8f7c08cddcdd4c17f3edaa50430d9323558b709cc821db7d4853fd7b" exitCode=0 Oct 30 03:53:17.104085 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:17.103645 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3324db09a84c9" event={"ID":"d38e1a10-1192-418b-bfbb-65fa776781fe","Type":"ContainerDied","Data":"73bd312e8f7c08cddcdd4c17f3edaa50430d9323558b709cc821db7d4853fd7b"} Oct 30 03:53:18.246012 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:18.245987 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3324db09a84c9" Oct 30 03:53:19.113951 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:19.113913 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3324db09a84c9" event={"ID":"d38e1a10-1192-418b-bfbb-65fa776781fe","Type":"ContainerDied","Data":"4a023c2b8da6b02f1d36b091c7a7d250b52be90a0f3aa194bc6b477de959b349"} Oct 30 03:53:19.113951 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:19.113943 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3324db09a84c9" Oct 30 03:53:19.114157 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:19.113950 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a023c2b8da6b02f1d36b091c7a7d250b52be90a0f3aa194bc6b477de959b349" Oct 30 03:53:20.176877 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:20.176845 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-part-only-denied-18e33250ffd97e46"] Oct 30 03:53:20.177319 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:20.177247 5882 cpu_manager.go:414] "containerMap: removing container" podUID="d38e1a10-1192-418b-bfbb-65fa776781fe" containerName="curl" Oct 30 03:53:20.177319 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:20.177263 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d38e1a10-1192-418b-bfbb-65fa776781fe" containerName="curl" Oct 30 03:53:20.177442 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:20.177356 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d38e1a10-1192-418b-bfbb-65fa776781fe" containerName="curl" Oct 30 03:53:20.180450 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:20.180434 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-part-only-denied-18e33250ffd97e46" Oct 30 03:53:20.191966 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:20.191938 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-part-only-denied-18e33250ffd97e46" Oct 30 03:53:20.197961 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:20.197933 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-part-only-denied-18e33250ffd97e46"] Oct 30 03:53:20.485719 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:20.485640 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-part-only-denied-18e33250ffd97e46"] Oct 30 03:53:20.488997 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:53:20.488967 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9d0cec5_4013_41f1_9fbb_d1abd1295803.slice/crio-e6770afa62e0fc2d1f8781fd4dd07f95de662fb615df6c7c8f4b5c87b1708099 WatchSource:0}: Error finding container e6770afa62e0fc2d1f8781fd4dd07f95de662fb615df6c7c8f4b5c87b1708099: Status 404 returned error can't find the container with id e6770afa62e0fc2d1f8781fd4dd07f95de662fb615df6c7c8f4b5c87b1708099 Oct 30 03:53:21.124345 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:21.124308 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-part-only-denied-18e33250ffd97e46" event={"ID":"f9d0cec5-4013-41f1-9fbb-d1abd1295803","Type":"ContainerStarted","Data":"b77466ac0b5da2f9af6a99f1a5167f445310b206aae4c3cad531e351d0567cd8"} Oct 30 03:53:21.124345 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:21.124349 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-part-only-denied-18e33250ffd97e46" event={"ID":"f9d0cec5-4013-41f1-9fbb-d1abd1295803","Type":"ContainerStarted","Data":"e6770afa62e0fc2d1f8781fd4dd07f95de662fb615df6c7c8f4b5c87b1708099"} Oct 30 03:53:21.161676 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:21.161602 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-part-only-denied-18e33250ffd97e46" podStartSLOduration=1.161588424 podStartE2EDuration="1.161588424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:53:21.161150686 +0000 UTC m=+4313.275465662" watchObservedRunningTime="2026-10-30 03:53:21.161588424 +0000 UTC m=+4313.275903395" Oct 30 03:53:23.942935 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:53:23.942897 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:53:31.169453 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:31.169419 5882 generic.go:356] "Generic (PLEG): container finished" podID="f9d0cec5-4013-41f1-9fbb-d1abd1295803" containerID="b77466ac0b5da2f9af6a99f1a5167f445310b206aae4c3cad531e351d0567cd8" exitCode=0 Oct 30 03:53:31.169861 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:31.169497 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-part-only-denied-18e33250ffd97e46" event={"ID":"f9d0cec5-4013-41f1-9fbb-d1abd1295803","Type":"ContainerDied","Data":"b77466ac0b5da2f9af6a99f1a5167f445310b206aae4c3cad531e351d0567cd8"} Oct 30 03:53:32.306460 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:32.306438 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-part-only-denied-18e33250ffd97e46" Oct 30 03:53:33.179126 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:33.179090 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-part-only-denied-18e33250ffd97e46" event={"ID":"f9d0cec5-4013-41f1-9fbb-d1abd1295803","Type":"ContainerDied","Data":"e6770afa62e0fc2d1f8781fd4dd07f95de662fb615df6c7c8f4b5c87b1708099"} Oct 30 03:53:33.179126 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:33.179126 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-part-only-denied-18e33250ffd97e46" Oct 30 03:53:33.179126 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:33.179133 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6770afa62e0fc2d1f8781fd4dd07f95de662fb615df6c7c8f4b5c87b1708099" Oct 30 03:53:34.404249 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:34.404212 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-role-only-denied-18e332544f0e5c1d"] Oct 30 03:53:34.404648 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:34.404595 5882 cpu_manager.go:414] "containerMap: removing container" podUID="f9d0cec5-4013-41f1-9fbb-d1abd1295803" containerName="curl" Oct 30 03:53:34.404648 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:34.404606 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d0cec5-4013-41f1-9fbb-d1abd1295803" containerName="curl" Oct 30 03:53:34.404732 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:34.404673 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d0cec5-4013-41f1-9fbb-d1abd1295803" containerName="curl" Oct 30 03:53:34.410741 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:34.410724 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-role-only-denied-18e332544f0e5c1d" Oct 30 03:53:34.421370 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:34.421346 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-role-only-denied-18e332544f0e5c1d" Oct 30 03:53:34.424196 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:34.423842 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-role-only-denied-18e332544f0e5c1d"] Oct 30 03:53:34.711432 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:34.711400 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-role-only-denied-18e332544f0e5c1d"] Oct 30 03:53:34.712637 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:53:34.712597 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode07f19bb_04b5_41fb_8645_6e7ee8c4bd8d.slice/crio-36f9916d26b4c577ab5356d9cf7fa6138939136c3e0475a1466076d606ffe673 WatchSource:0}: Error finding container 36f9916d26b4c577ab5356d9cf7fa6138939136c3e0475a1466076d606ffe673: Status 404 returned error can't find the container with id 36f9916d26b4c577ab5356d9cf7fa6138939136c3e0475a1466076d606ffe673 Oct 30 03:53:35.189815 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:35.189780 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-role-only-denied-18e332544f0e5c1d" event={"ID":"e07f19bb-04b5-41fb-8645-6e7ee8c4bd8d","Type":"ContainerStarted","Data":"ff055f6eb939e236d5015579139343b2c443c7b5af897f09c06c34b45c7a60e5"} Oct 30 03:53:35.189815 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:35.189821 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-role-only-denied-18e332544f0e5c1d" event={"ID":"e07f19bb-04b5-41fb-8645-6e7ee8c4bd8d","Type":"ContainerStarted","Data":"36f9916d26b4c577ab5356d9cf7fa6138939136c3e0475a1466076d606ffe673"} Oct 30 03:53:35.230532 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:35.230479 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-role-only-denied-18e332544f0e5c1d" podStartSLOduration=1.230463599 podStartE2EDuration="1.230463599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:53:35.23007543 +0000 UTC m=+4327.344390404" watchObservedRunningTime="2026-10-30 03:53:35.230463599 +0000 UTC m=+4327.344778594" Oct 30 03:53:45.238308 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:45.238217 5882 generic.go:356] "Generic (PLEG): container finished" podID="e07f19bb-04b5-41fb-8645-6e7ee8c4bd8d" containerID="ff055f6eb939e236d5015579139343b2c443c7b5af897f09c06c34b45c7a60e5" exitCode=0 Oct 30 03:53:45.238308 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:45.238286 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-role-only-denied-18e332544f0e5c1d" event={"ID":"e07f19bb-04b5-41fb-8645-6e7ee8c4bd8d","Type":"ContainerDied","Data":"ff055f6eb939e236d5015579139343b2c443c7b5af897f09c06c34b45c7a60e5"} Oct 30 03:53:46.373767 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:46.373745 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-role-only-denied-18e332544f0e5c1d" Oct 30 03:53:46.609393 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:46.609356 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3325726f52a13"] Oct 30 03:53:46.609750 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:46.609737 5882 cpu_manager.go:414] "containerMap: removing container" podUID="e07f19bb-04b5-41fb-8645-6e7ee8c4bd8d" containerName="curl" Oct 30 03:53:46.609799 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:46.609751 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e07f19bb-04b5-41fb-8645-6e7ee8c4bd8d" containerName="curl" Oct 30 03:53:46.609834 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:46.609820 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e07f19bb-04b5-41fb-8645-6e7ee8c4bd8d" containerName="curl" Oct 30 03:53:46.612905 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:46.612889 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3325726f52a13" Oct 30 03:53:46.623815 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:46.623794 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3325726f52a13" Oct 30 03:53:46.629890 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:46.629864 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3325726f52a13"] Oct 30 03:53:46.913917 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:46.913890 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3325726f52a13"] Oct 30 03:53:46.914758 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:53:46.914734 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9a42ed6_aaf4_43df_ae69_7400c7fe25c7.slice/crio-2a69fb91da015e1a28bfdcd21e7abefe717d529e1a7c7e4f90307a3378968a27 WatchSource:0}: Error finding container 2a69fb91da015e1a28bfdcd21e7abefe717d529e1a7c7e4f90307a3378968a27: Status 404 returned error can't find the container with id 2a69fb91da015e1a28bfdcd21e7abefe717d529e1a7c7e4f90307a3378968a27 Oct 30 03:53:47.249031 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:47.248999 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-role-only-denied-18e332544f0e5c1d" Oct 30 03:53:47.249031 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:47.249018 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-role-only-denied-18e332544f0e5c1d" event={"ID":"e07f19bb-04b5-41fb-8645-6e7ee8c4bd8d","Type":"ContainerDied","Data":"36f9916d26b4c577ab5356d9cf7fa6138939136c3e0475a1466076d606ffe673"} Oct 30 03:53:47.249286 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:47.249052 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36f9916d26b4c577ab5356d9cf7fa6138939136c3e0475a1466076d606ffe673" Oct 30 03:53:47.250501 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:47.250476 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3325726f52a13" event={"ID":"e9a42ed6-aaf4-43df-ae69-7400c7fe25c7","Type":"ContainerStarted","Data":"5a1b93df1e6c403659cf1b7b7ec5825c47237a7bb9c7abc3d73b26bba4ea6910"} Oct 30 03:53:47.250606 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:47.250511 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3325726f52a13" event={"ID":"e9a42ed6-aaf4-43df-ae69-7400c7fe25c7","Type":"ContainerStarted","Data":"2a69fb91da015e1a28bfdcd21e7abefe717d529e1a7c7e4f90307a3378968a27"} Oct 30 03:53:47.292143 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:47.292096 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3325726f52a13" podStartSLOduration=1.292083371 podStartE2EDuration="1.292083371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:53:47.291086767 +0000 UTC m=+4339.405401746" watchObservedRunningTime="2026-10-30 03:53:47.292083371 +0000 UTC m=+4339.406398412" Oct 30 03:53:53.943284 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:53:53.943244 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:53:57.297674 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:57.297637 5882 generic.go:356] "Generic (PLEG): container finished" podID="e9a42ed6-aaf4-43df-ae69-7400c7fe25c7" containerID="5a1b93df1e6c403659cf1b7b7ec5825c47237a7bb9c7abc3d73b26bba4ea6910" exitCode=0 Oct 30 03:53:57.298077 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:57.297685 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3325726f52a13" event={"ID":"e9a42ed6-aaf4-43df-ae69-7400c7fe25c7","Type":"ContainerDied","Data":"5a1b93df1e6c403659cf1b7b7ec5825c47237a7bb9c7abc3d73b26bba4ea6910"} Oct 30 03:53:58.432812 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:58.432783 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3325726f52a13" Oct 30 03:53:58.836698 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:58.836666 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-workload-allowed-18e33259ff43b8a5"] Oct 30 03:53:58.837082 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:58.837070 5882 cpu_manager.go:414] "containerMap: removing container" podUID="e9a42ed6-aaf4-43df-ae69-7400c7fe25c7" containerName="curl" Oct 30 03:53:58.837128 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:58.837084 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9a42ed6-aaf4-43df-ae69-7400c7fe25c7" containerName="curl" Oct 30 03:53:58.837163 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:58.837141 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9a42ed6-aaf4-43df-ae69-7400c7fe25c7" containerName="curl" Oct 30 03:53:58.840144 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:58.840126 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-workload-allowed-18e33259ff43b8a5" Oct 30 03:53:58.856785 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:58.856758 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-workload-allowed-18e33259ff43b8a5"] Oct 30 03:53:58.868792 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:58.868755 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"collector-network-policy-18e332385848ec88\"/\"default-dockercfg-87bsr\"" Oct 30 03:53:58.871876 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:58.871853 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-workload-allowed-18e33259ff43b8a5" Oct 30 03:53:59.165442 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:59.165412 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-workload-allowed-18e33259ff43b8a5"] Oct 30 03:53:59.166647 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:53:59.166588 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72875129_c483_4607_a4ab_60b0ee81e55f.slice/crio-bc435d066b1dbe5808b02352d22f14d54f3137dc11ba53c9447822b809fd75e7 WatchSource:0}: Error finding container bc435d066b1dbe5808b02352d22f14d54f3137dc11ba53c9447822b809fd75e7: Status 404 returned error can't find the container with id bc435d066b1dbe5808b02352d22f14d54f3137dc11ba53c9447822b809fd75e7 Oct 30 03:53:59.309331 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:59.309305 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3325726f52a13" Oct 30 03:53:59.309510 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:59.309347 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3325726f52a13" event={"ID":"e9a42ed6-aaf4-43df-ae69-7400c7fe25c7","Type":"ContainerDied","Data":"2a69fb91da015e1a28bfdcd21e7abefe717d529e1a7c7e4f90307a3378968a27"} Oct 30 03:53:59.309510 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:59.309377 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a69fb91da015e1a28bfdcd21e7abefe717d529e1a7c7e4f90307a3378968a27" Oct 30 03:53:59.311155 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:59.311123 5882 generic.go:356] "Generic (PLEG): container finished" podID="72875129-c483-4607-a4ab-60b0ee81e55f" containerID="424dae49fb36c9381094d7be178ebe534542cd9f398353847813e8338a38fc6f" exitCode=0 Oct 30 03:53:59.311253 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:59.311180 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-workload-allowed-18e33259ff43b8a5" event={"ID":"72875129-c483-4607-a4ab-60b0ee81e55f","Type":"ContainerDied","Data":"424dae49fb36c9381094d7be178ebe534542cd9f398353847813e8338a38fc6f"} Oct 30 03:53:59.311253 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:53:59.311212 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-workload-allowed-18e33259ff43b8a5" event={"ID":"72875129-c483-4607-a4ab-60b0ee81e55f","Type":"ContainerStarted","Data":"bc435d066b1dbe5808b02352d22f14d54f3137dc11ba53c9447822b809fd75e7"} Oct 30 03:54:00.437736 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:00.437710 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-workload-allowed-18e33259ff43b8a5" Oct 30 03:54:00.962337 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:00.962301 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-prefill-allowed-18e3325a7dd8aa7c"] Oct 30 03:54:00.962761 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:00.962744 5882 cpu_manager.go:414] "containerMap: removing container" podUID="72875129-c483-4607-a4ab-60b0ee81e55f" containerName="curl" Oct 30 03:54:00.962877 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:00.962762 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72875129-c483-4607-a4ab-60b0ee81e55f" containerName="curl" Oct 30 03:54:00.962935 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:00.962908 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="72875129-c483-4607-a4ab-60b0ee81e55f" containerName="curl" Oct 30 03:54:00.965910 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:00.965891 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-prefill-allowed-18e3325a7dd8aa7c" Oct 30 03:54:00.976637 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:00.976605 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-prefill-allowed-18e3325a7dd8aa7c" Oct 30 03:54:00.982445 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:00.982421 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-prefill-allowed-18e3325a7dd8aa7c"] Oct 30 03:54:01.265403 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:01.265377 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-prefill-allowed-18e3325a7dd8aa7c"] Oct 30 03:54:01.266912 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:54:01.266879 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod732febb0_93cb_4b4a_9d93_35fc2ed63a6b.slice/crio-44018d18b6fecb823fbe4931ff698903e18bf654ed5db0480e4110483b2ffa1b WatchSource:0}: Error finding container 44018d18b6fecb823fbe4931ff698903e18bf654ed5db0480e4110483b2ffa1b: Status 404 returned error can't find the container with id 44018d18b6fecb823fbe4931ff698903e18bf654ed5db0480e4110483b2ffa1b Oct 30 03:54:01.321494 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:01.321468 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-prefill-allowed-18e3325a7dd8aa7c" event={"ID":"732febb0-93cb-4b4a-9d93-35fc2ed63a6b","Type":"ContainerStarted","Data":"44018d18b6fecb823fbe4931ff698903e18bf654ed5db0480e4110483b2ffa1b"} Oct 30 03:54:01.322670 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:01.322646 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-workload-allowed-18e33259ff43b8a5" event={"ID":"72875129-c483-4607-a4ab-60b0ee81e55f","Type":"ContainerDied","Data":"bc435d066b1dbe5808b02352d22f14d54f3137dc11ba53c9447822b809fd75e7"} Oct 30 03:54:01.322773 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:01.322685 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc435d066b1dbe5808b02352d22f14d54f3137dc11ba53c9447822b809fd75e7" Oct 30 03:54:01.322773 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:01.322654 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-workload-allowed-18e33259ff43b8a5" Oct 30 03:54:02.328730 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:02.328691 5882 generic.go:356] "Generic (PLEG): container finished" podID="732febb0-93cb-4b4a-9d93-35fc2ed63a6b" containerID="23abf57f5bfc681e0e2dfbc606e7e41d914f13b28579a81f985d14fcbf762a58" exitCode=0 Oct 30 03:54:02.329196 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:02.328779 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-prefill-allowed-18e3325a7dd8aa7c" event={"ID":"732febb0-93cb-4b4a-9d93-35fc2ed63a6b","Type":"ContainerDied","Data":"23abf57f5bfc681e0e2dfbc606e7e41d914f13b28579a81f985d14fcbf762a58"} Oct 30 03:54:03.464301 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:03.464275 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-prefill-allowed-18e3325a7dd8aa7c" Oct 30 03:54:04.339394 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:04.339311 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-prefill-allowed-18e3325a7dd8aa7c" Oct 30 03:54:04.339394 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:04.339340 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-prefill-allowed-18e3325a7dd8aa7c" event={"ID":"732febb0-93cb-4b4a-9d93-35fc2ed63a6b","Type":"ContainerDied","Data":"44018d18b6fecb823fbe4931ff698903e18bf654ed5db0480e4110483b2ffa1b"} Oct 30 03:54:04.339394 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:04.339376 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44018d18b6fecb823fbe4931ff698903e18bf654ed5db0480e4110483b2ffa1b" Oct 30 03:54:05.102412 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:05.102379 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-worker-allowed-18e3325b74c17aa9"] Oct 30 03:54:05.102816 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:05.102759 5882 cpu_manager.go:414] "containerMap: removing container" podUID="732febb0-93cb-4b4a-9d93-35fc2ed63a6b" containerName="curl" Oct 30 03:54:05.102816 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:05.102769 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732febb0-93cb-4b4a-9d93-35fc2ed63a6b" containerName="curl" Oct 30 03:54:05.102899 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:05.102844 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="732febb0-93cb-4b4a-9d93-35fc2ed63a6b" containerName="curl" Oct 30 03:54:05.105960 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:05.105942 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-worker-allowed-18e3325b74c17aa9" Oct 30 03:54:05.121627 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:05.121590 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-worker-allowed-18e3325b74c17aa9"] Oct 30 03:54:05.133462 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:05.133440 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"collector-network-policy-18e332385848ec88\"/\"default-dockercfg-87bsr\"" Oct 30 03:54:05.137519 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:05.137500 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-worker-allowed-18e3325b74c17aa9" Oct 30 03:54:05.428189 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:05.428159 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-worker-allowed-18e3325b74c17aa9"] Oct 30 03:54:05.429302 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:54:05.429276 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod624d48a6_a547_422b_8c32_83097a4f05cc.slice/crio-c3da8f001b993b938b1a4e072a582ed5de745f38d2317c638ffd53706b57c491 WatchSource:0}: Error finding container c3da8f001b993b938b1a4e072a582ed5de745f38d2317c638ffd53706b57c491: Status 404 returned error can't find the container with id c3da8f001b993b938b1a4e072a582ed5de745f38d2317c638ffd53706b57c491 Oct 30 03:54:06.350787 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:06.350750 5882 generic.go:356] "Generic (PLEG): container finished" podID="624d48a6-a547-422b-8c32-83097a4f05cc" containerID="de6bedfa62efb6fe45ab1a3b8f4d7f8d763b8654eca31a677ab09f149e8a1533" exitCode=0 Oct 30 03:54:06.351188 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:06.350837 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-worker-allowed-18e3325b74c17aa9" event={"ID":"624d48a6-a547-422b-8c32-83097a4f05cc","Type":"ContainerDied","Data":"de6bedfa62efb6fe45ab1a3b8f4d7f8d763b8654eca31a677ab09f149e8a1533"} Oct 30 03:54:06.351188 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:06.350886 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-worker-allowed-18e3325b74c17aa9" event={"ID":"624d48a6-a547-422b-8c32-83097a4f05cc","Type":"ContainerStarted","Data":"c3da8f001b993b938b1a4e072a582ed5de745f38d2317c638ffd53706b57c491"} Oct 30 03:54:07.494332 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:07.494309 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-worker-allowed-18e3325b74c17aa9" Oct 30 03:54:08.361086 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:08.360998 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-worker-allowed-18e3325b74c17aa9" event={"ID":"624d48a6-a547-422b-8c32-83097a4f05cc","Type":"ContainerDied","Data":"c3da8f001b993b938b1a4e072a582ed5de745f38d2317c638ffd53706b57c491"} Oct 30 03:54:08.361086 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:08.361026 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-worker-allowed-18e3325b74c17aa9" Oct 30 03:54:08.361086 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:08.361045 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3da8f001b993b938b1a4e072a582ed5de745f38d2317c638ffd53706b57c491" Oct 30 03:54:09.250101 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:09.250068 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-leader-allowed-18e3325c6b9fda3c"] Oct 30 03:54:09.250581 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:09.250453 5882 cpu_manager.go:414] "containerMap: removing container" podUID="624d48a6-a547-422b-8c32-83097a4f05cc" containerName="curl" Oct 30 03:54:09.250581 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:09.250465 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="624d48a6-a547-422b-8c32-83097a4f05cc" containerName="curl" Oct 30 03:54:09.250581 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:09.250552 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="624d48a6-a547-422b-8c32-83097a4f05cc" containerName="curl" Oct 30 03:54:09.253690 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:09.253671 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-leader-allowed-18e3325c6b9fda3c" Oct 30 03:54:09.270034 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:09.270010 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-leader-allowed-18e3325c6b9fda3c"] Oct 30 03:54:09.281810 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:09.281792 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"collector-network-policy-18e332385848ec88\"/\"default-dockercfg-87bsr\"" Oct 30 03:54:09.284908 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:09.284883 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-leader-allowed-18e3325c6b9fda3c" Oct 30 03:54:09.576996 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:09.576967 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-leader-allowed-18e3325c6b9fda3c"] Oct 30 03:54:09.578384 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:54:09.578355 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda187a012_2907_4052_bd2d_33241877ac55.slice/crio-a4316e675c9485d16a4051ca3ab52220b7aca74dcddd0b4b2e9cf73e4b5892be WatchSource:0}: Error finding container a4316e675c9485d16a4051ca3ab52220b7aca74dcddd0b4b2e9cf73e4b5892be: Status 404 returned error can't find the container with id a4316e675c9485d16a4051ca3ab52220b7aca74dcddd0b4b2e9cf73e4b5892be Oct 30 03:54:10.371789 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:10.371752 5882 generic.go:356] "Generic (PLEG): container finished" podID="a187a012-2907-4052-bd2d-33241877ac55" containerID="788dbed1297aca3a684a05e46215a32f0c006d063e7bc9f61a201ba4aff5a4d7" exitCode=0 Oct 30 03:54:10.371789 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:10.371792 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-leader-allowed-18e3325c6b9fda3c" event={"ID":"a187a012-2907-4052-bd2d-33241877ac55","Type":"ContainerDied","Data":"788dbed1297aca3a684a05e46215a32f0c006d063e7bc9f61a201ba4aff5a4d7"} Oct 30 03:54:10.372276 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:10.371819 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-leader-allowed-18e3325c6b9fda3c" event={"ID":"a187a012-2907-4052-bd2d-33241877ac55","Type":"ContainerStarted","Data":"a4316e675c9485d16a4051ca3ab52220b7aca74dcddd0b4b2e9cf73e4b5892be"} Oct 30 03:54:11.499237 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:11.499210 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-leader-allowed-18e3325c6b9fda3c" Oct 30 03:54:12.382286 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:12.382203 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-leader-allowed-18e3325c6b9fda3c" event={"ID":"a187a012-2907-4052-bd2d-33241877ac55","Type":"ContainerDied","Data":"a4316e675c9485d16a4051ca3ab52220b7aca74dcddd0b4b2e9cf73e4b5892be"} Oct 30 03:54:12.382286 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:12.382236 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-leader-allowed-18e3325c6b9fda3c" Oct 30 03:54:12.382286 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:12.382248 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4316e675c9485d16a4051ca3ab52220b7aca74dcddd0b4b2e9cf73e4b5892be" Oct 30 03:54:13.387953 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:13.387913 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-worker-prefill-allowed-18e3325d6279ed00"] Oct 30 03:54:13.388472 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:13.388451 5882 cpu_manager.go:414] "containerMap: removing container" podUID="a187a012-2907-4052-bd2d-33241877ac55" containerName="curl" Oct 30 03:54:13.388576 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:13.388475 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a187a012-2907-4052-bd2d-33241877ac55" containerName="curl" Oct 30 03:54:13.388576 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:13.388562 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a187a012-2907-4052-bd2d-33241877ac55" containerName="curl" Oct 30 03:54:13.391311 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:13.391285 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-worker-prefill-allowed-18e3325d6279ed00" Oct 30 03:54:13.407570 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:13.407543 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-worker-prefill-allowed-18e3325d6279ed00"] Oct 30 03:54:13.419507 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:13.419482 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"collector-network-policy-18e332385848ec88\"/\"default-dockercfg-87bsr\"" Oct 30 03:54:13.422316 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:13.422297 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-worker-prefill-allowed-18e3325d6279ed00" Oct 30 03:54:13.711866 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:13.711833 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-worker-prefill-allowed-18e3325d6279ed00"] Oct 30 03:54:13.713078 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:54:13.713046 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c0541c9_85f7_42ac_9e86_83f4e8290bd6.slice/crio-c17b29f2f97f26990307b9709003f0898ebeaa2330d89e7c4db0685d60e450f0 WatchSource:0}: Error finding container c17b29f2f97f26990307b9709003f0898ebeaa2330d89e7c4db0685d60e450f0: Status 404 returned error can't find the container with id c17b29f2f97f26990307b9709003f0898ebeaa2330d89e7c4db0685d60e450f0 Oct 30 03:54:14.393570 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:14.393537 5882 generic.go:356] "Generic (PLEG): container finished" podID="8c0541c9-85f7-42ac-9e86-83f4e8290bd6" containerID="d1a72a082f0b7c2e4472cf7db7be5d46dcbc551f34dec414191637768af44e3f" exitCode=0 Oct 30 03:54:14.394046 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:14.393645 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-worker-prefill-allowed-18e3325d6279ed00" event={"ID":"8c0541c9-85f7-42ac-9e86-83f4e8290bd6","Type":"ContainerDied","Data":"d1a72a082f0b7c2e4472cf7db7be5d46dcbc551f34dec414191637768af44e3f"} Oct 30 03:54:14.394046 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:14.393685 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-worker-prefill-allowed-18e3325d6279ed00" event={"ID":"8c0541c9-85f7-42ac-9e86-83f4e8290bd6","Type":"ContainerStarted","Data":"c17b29f2f97f26990307b9709003f0898ebeaa2330d89e7c4db0685d60e450f0"} Oct 30 03:54:15.531756 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:15.531733 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-worker-prefill-allowed-18e3325d6279ed00" Oct 30 03:54:16.404553 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:16.404470 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-worker-prefill-allowed-18e3325d6279ed00" Oct 30 03:54:16.404553 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:16.404478 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-worker-prefill-allowed-18e3325d6279ed00" event={"ID":"8c0541c9-85f7-42ac-9e86-83f4e8290bd6","Type":"ContainerDied","Data":"c17b29f2f97f26990307b9709003f0898ebeaa2330d89e7c4db0685d60e450f0"} Oct 30 03:54:16.404757 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:16.404576 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c17b29f2f97f26990307b9709003f0898ebeaa2330d89e7c4db0685d60e450f0" Oct 30 03:54:17.530459 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:17.530424 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-leader-prefill-allowed-18e3325e596a7ee3"] Oct 30 03:54:17.530877 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:17.530817 5882 cpu_manager.go:414] "containerMap: removing container" podUID="8c0541c9-85f7-42ac-9e86-83f4e8290bd6" containerName="curl" Oct 30 03:54:17.530877 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:17.530831 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c0541c9-85f7-42ac-9e86-83f4e8290bd6" containerName="curl" Oct 30 03:54:17.530949 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:17.530913 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c0541c9-85f7-42ac-9e86-83f4e8290bd6" containerName="curl" Oct 30 03:54:17.533925 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:17.533899 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-leader-prefill-allowed-18e3325e596a7ee3" Oct 30 03:54:17.550857 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:17.550834 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-leader-prefill-allowed-18e3325e596a7ee3"] Oct 30 03:54:17.562749 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:17.562724 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"collector-network-policy-18e332385848ec88\"/\"default-dockercfg-87bsr\"" Oct 30 03:54:17.564767 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:17.564745 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-leader-prefill-allowed-18e3325e596a7ee3" Oct 30 03:54:17.858747 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:17.858717 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-leader-prefill-allowed-18e3325e596a7ee3"] Oct 30 03:54:17.859710 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:54:17.859679 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod924992da_80de_435b_ad4c_bbfd8dc26af2.slice/crio-16c63c32604a120751a2d2ead36f54efd8dc03f30ef8600185acbfd2bcdaaded WatchSource:0}: Error finding container 16c63c32604a120751a2d2ead36f54efd8dc03f30ef8600185acbfd2bcdaaded: Status 404 returned error can't find the container with id 16c63c32604a120751a2d2ead36f54efd8dc03f30ef8600185acbfd2bcdaaded Oct 30 03:54:18.417019 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:18.416977 5882 generic.go:356] "Generic (PLEG): container finished" podID="924992da-80de-435b-ad4c-bbfd8dc26af2" containerID="fe27740702cb261933154d1439bd7875f3749991b20479938592da1dc2c16fb0" exitCode=0 Oct 30 03:54:18.417199 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:18.417054 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-leader-prefill-allowed-18e3325e596a7ee3" event={"ID":"924992da-80de-435b-ad4c-bbfd8dc26af2","Type":"ContainerDied","Data":"fe27740702cb261933154d1439bd7875f3749991b20479938592da1dc2c16fb0"} Oct 30 03:54:18.417199 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:18.417091 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-leader-prefill-allowed-18e3325e596a7ee3" event={"ID":"924992da-80de-435b-ad4c-bbfd8dc26af2","Type":"ContainerStarted","Data":"16c63c32604a120751a2d2ead36f54efd8dc03f30ef8600185acbfd2bcdaaded"} Oct 30 03:54:19.555624 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:19.555585 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-leader-prefill-allowed-18e3325e596a7ee3" Oct 30 03:54:20.430229 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:20.430142 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-leader-prefill-allowed-18e3325e596a7ee3" Oct 30 03:54:20.430391 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:20.430133 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-leader-prefill-allowed-18e3325e596a7ee3" event={"ID":"924992da-80de-435b-ad4c-bbfd8dc26af2","Type":"ContainerDied","Data":"16c63c32604a120751a2d2ead36f54efd8dc03f30ef8600185acbfd2bcdaaded"} Oct 30 03:54:20.430391 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:20.430263 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16c63c32604a120751a2d2ead36f54efd8dc03f30ef8600185acbfd2bcdaaded" Oct 30 03:54:21.675252 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:21.675218 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-scheduler-allowed-18e3325f5047df47"] Oct 30 03:54:21.675639 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:21.675590 5882 cpu_manager.go:414] "containerMap: removing container" podUID="924992da-80de-435b-ad4c-bbfd8dc26af2" containerName="curl" Oct 30 03:54:21.675639 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:21.675602 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924992da-80de-435b-ad4c-bbfd8dc26af2" containerName="curl" Oct 30 03:54:21.675713 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:21.675685 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="924992da-80de-435b-ad4c-bbfd8dc26af2" containerName="curl" Oct 30 03:54:21.678771 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:21.678754 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-scheduler-allowed-18e3325f5047df47" Oct 30 03:54:21.695534 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:21.695509 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-scheduler-allowed-18e3325f5047df47"] Oct 30 03:54:21.707355 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:21.707335 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"collector-network-policy-18e332385848ec88\"/\"default-dockercfg-87bsr\"" Oct 30 03:54:21.709547 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:21.709532 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-scheduler-allowed-18e3325f5047df47" Oct 30 03:54:22.002201 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:22.002172 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-scheduler-allowed-18e3325f5047df47"] Oct 30 03:54:22.003894 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:54:22.003870 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod485f30c1_fe8a_4369_b9b5_168c5b8ae02d.slice/crio-fbdfea57ab529389f36c0c09bba80569b806880e7595d76d6d3e9fe9b1e8cb85 WatchSource:0}: Error finding container fbdfea57ab529389f36c0c09bba80569b806880e7595d76d6d3e9fe9b1e8cb85: Status 404 returned error can't find the container with id fbdfea57ab529389f36c0c09bba80569b806880e7595d76d6d3e9fe9b1e8cb85 Oct 30 03:54:22.440767 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:22.440676 5882 generic.go:356] "Generic (PLEG): container finished" podID="485f30c1-fe8a-4369-b9b5-168c5b8ae02d" containerID="873c2761acbf47c40ad8e9b7f4747e50d7c834e0fa6cbc0b7d9740366359b52c" exitCode=0 Oct 30 03:54:22.440767 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:22.440744 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-scheduler-allowed-18e3325f5047df47" event={"ID":"485f30c1-fe8a-4369-b9b5-168c5b8ae02d","Type":"ContainerDied","Data":"873c2761acbf47c40ad8e9b7f4747e50d7c834e0fa6cbc0b7d9740366359b52c"} Oct 30 03:54:22.440952 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:22.440774 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-scheduler-allowed-18e3325f5047df47" event={"ID":"485f30c1-fe8a-4369-b9b5-168c5b8ae02d","Type":"ContainerStarted","Data":"fbdfea57ab529389f36c0c09bba80569b806880e7595d76d6d3e9fe9b1e8cb85"} Oct 30 03:54:23.570054 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:23.570028 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-scheduler-allowed-18e3325f5047df47" Oct 30 03:54:23.788903 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:23.788868 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-unlabeled-denied-18e3325fcef86f7a"] Oct 30 03:54:23.789325 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:23.789306 5882 cpu_manager.go:414] "containerMap: removing container" podUID="485f30c1-fe8a-4369-b9b5-168c5b8ae02d" containerName="curl" Oct 30 03:54:23.789410 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:23.789328 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="485f30c1-fe8a-4369-b9b5-168c5b8ae02d" containerName="curl" Oct 30 03:54:23.789463 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:23.789446 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="485f30c1-fe8a-4369-b9b5-168c5b8ae02d" containerName="curl" Oct 30 03:54:23.792264 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:23.792241 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-unlabeled-denied-18e3325fcef86f7a" Oct 30 03:54:23.802762 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:23.802743 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-unlabeled-denied-18e3325fcef86f7a" Oct 30 03:54:23.809524 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:23.809501 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-unlabeled-denied-18e3325fcef86f7a"] Oct 30 03:54:23.942725 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:54:23.942691 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:54:24.096510 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:24.096483 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-unlabeled-denied-18e3325fcef86f7a"] Oct 30 03:54:24.097583 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:54:24.097556 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7038f143_bce1_418d_ab3e_ce833c2c1410.slice/crio-f17f6f505d280eb449eff5a86513d19421c4e7fe1751ef967a3f27b06f770343 WatchSource:0}: Error finding container f17f6f505d280eb449eff5a86513d19421c4e7fe1751ef967a3f27b06f770343: Status 404 returned error can't find the container with id f17f6f505d280eb449eff5a86513d19421c4e7fe1751ef967a3f27b06f770343 Oct 30 03:54:24.454665 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:24.454560 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-scheduler-allowed-18e3325f5047df47" Oct 30 03:54:24.454665 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:24.454590 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-scheduler-allowed-18e3325f5047df47" event={"ID":"485f30c1-fe8a-4369-b9b5-168c5b8ae02d","Type":"ContainerDied","Data":"fbdfea57ab529389f36c0c09bba80569b806880e7595d76d6d3e9fe9b1e8cb85"} Oct 30 03:54:24.454665 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:24.454644 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbdfea57ab529389f36c0c09bba80569b806880e7595d76d6d3e9fe9b1e8cb85" Oct 30 03:54:24.456070 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:24.456044 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-unlabeled-denied-18e3325fcef86f7a" event={"ID":"7038f143-bce1-418d-ab3e-ce833c2c1410","Type":"ContainerStarted","Data":"c59cd5bcfe4ca6c10564c1d309a483c451c7b42da05c04d90c3c0bebdf1f27a5"} Oct 30 03:54:24.456208 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:24.456080 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-unlabeled-denied-18e3325fcef86f7a" event={"ID":"7038f143-bce1-418d-ab3e-ce833c2c1410","Type":"ContainerStarted","Data":"f17f6f505d280eb449eff5a86513d19421c4e7fe1751ef967a3f27b06f770343"} Oct 30 03:54:24.498936 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:24.498871 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="collector-network-policy-18e332385848ec88/collector-otlp-unlabeled-denied-18e3325fcef86f7a" podStartSLOduration=1.498854944 podStartE2EDuration="1.498854944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:54:24.496938627 +0000 UTC m=+4376.611253603" watchObservedRunningTime="2026-10-30 03:54:24.498854944 +0000 UTC m=+4376.613169920" Oct 30 03:54:34.506367 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:34.506337 5882 generic.go:356] "Generic (PLEG): container finished" podID="7038f143-bce1-418d-ab3e-ce833c2c1410" containerID="c59cd5bcfe4ca6c10564c1d309a483c451c7b42da05c04d90c3c0bebdf1f27a5" exitCode=0 Oct 30 03:54:34.506820 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:34.506413 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-unlabeled-denied-18e3325fcef86f7a" event={"ID":"7038f143-bce1-418d-ab3e-ce833c2c1410","Type":"ContainerDied","Data":"c59cd5bcfe4ca6c10564c1d309a483c451c7b42da05c04d90c3c0bebdf1f27a5"} Oct 30 03:54:35.647157 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:35.647134 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-unlabeled-denied-18e3325fcef86f7a" Oct 30 03:54:35.986952 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:35.986872 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-part-only-denied-18e33262a6541f30"] Oct 30 03:54:35.987288 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:35.987276 5882 cpu_manager.go:414] "containerMap: removing container" podUID="7038f143-bce1-418d-ab3e-ce833c2c1410" containerName="curl" Oct 30 03:54:35.987331 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:35.987290 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7038f143-bce1-418d-ab3e-ce833c2c1410" containerName="curl" Oct 30 03:54:35.987365 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:35.987355 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7038f143-bce1-418d-ab3e-ce833c2c1410" containerName="curl" Oct 30 03:54:35.990401 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:35.990384 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-part-only-denied-18e33262a6541f30" Oct 30 03:54:36.000762 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:36.000742 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-part-only-denied-18e33262a6541f30" Oct 30 03:54:36.009399 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:36.009376 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-part-only-denied-18e33262a6541f30"] Oct 30 03:54:36.294384 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:36.294355 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-part-only-denied-18e33262a6541f30"] Oct 30 03:54:36.295118 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:54:36.295094 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f6782e1_f253_4835_8d35_3e635f99f081.slice/crio-48ffefb2b76f98c8e6f5a927a039e5a2894c702c445492d9bb2c9d20d2f3d218 WatchSource:0}: Error finding container 48ffefb2b76f98c8e6f5a927a039e5a2894c702c445492d9bb2c9d20d2f3d218: Status 404 returned error can't find the container with id 48ffefb2b76f98c8e6f5a927a039e5a2894c702c445492d9bb2c9d20d2f3d218 Oct 30 03:54:36.516948 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:36.516919 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-unlabeled-denied-18e3325fcef86f7a" Oct 30 03:54:36.516948 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:36.516927 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-unlabeled-denied-18e3325fcef86f7a" event={"ID":"7038f143-bce1-418d-ab3e-ce833c2c1410","Type":"ContainerDied","Data":"f17f6f505d280eb449eff5a86513d19421c4e7fe1751ef967a3f27b06f770343"} Oct 30 03:54:36.517196 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:36.516961 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f17f6f505d280eb449eff5a86513d19421c4e7fe1751ef967a3f27b06f770343" Oct 30 03:54:36.518687 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:36.518624 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-part-only-denied-18e33262a6541f30" event={"ID":"1f6782e1-f253-4835-8d35-3e635f99f081","Type":"ContainerStarted","Data":"3a67655c583a8baae360410848bb14196879bc8c5f2b3f55c1fc6fc4b37ac170"} Oct 30 03:54:36.518687 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:36.518661 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-part-only-denied-18e33262a6541f30" event={"ID":"1f6782e1-f253-4835-8d35-3e635f99f081","Type":"ContainerStarted","Data":"48ffefb2b76f98c8e6f5a927a039e5a2894c702c445492d9bb2c9d20d2f3d218"} Oct 30 03:54:36.561046 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:36.560962 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="collector-network-policy-18e332385848ec88/collector-otlp-part-only-denied-18e33262a6541f30" podStartSLOduration=1.560950813 podStartE2EDuration="1.560950813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:54:36.560019644 +0000 UTC m=+4388.674334622" watchObservedRunningTime="2026-10-30 03:54:36.560950813 +0000 UTC m=+4388.675265788" Oct 30 03:54:46.568488 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:46.568456 5882 generic.go:356] "Generic (PLEG): container finished" podID="1f6782e1-f253-4835-8d35-3e635f99f081" containerID="3a67655c583a8baae360410848bb14196879bc8c5f2b3f55c1fc6fc4b37ac170" exitCode=0 Oct 30 03:54:46.568931 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:46.568526 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-part-only-denied-18e33262a6541f30" event={"ID":"1f6782e1-f253-4835-8d35-3e635f99f081","Type":"ContainerDied","Data":"3a67655c583a8baae360410848bb14196879bc8c5f2b3f55c1fc6fc4b37ac170"} Oct 30 03:54:47.712538 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:47.712516 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-part-only-denied-18e33262a6541f30" Oct 30 03:54:48.203821 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:48.203784 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-role-only-denied-18e332657d59fd8c"] Oct 30 03:54:48.204156 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:48.204144 5882 cpu_manager.go:414] "containerMap: removing container" podUID="1f6782e1-f253-4835-8d35-3e635f99f081" containerName="curl" Oct 30 03:54:48.204195 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:48.204157 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f6782e1-f253-4835-8d35-3e635f99f081" containerName="curl" Oct 30 03:54:48.204241 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:48.204232 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f6782e1-f253-4835-8d35-3e635f99f081" containerName="curl" Oct 30 03:54:48.207394 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:48.207377 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-role-only-denied-18e332657d59fd8c" Oct 30 03:54:48.217839 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:48.217817 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-role-only-denied-18e332657d59fd8c" Oct 30 03:54:48.223851 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:48.223823 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-role-only-denied-18e332657d59fd8c"] Oct 30 03:54:48.509774 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:48.509746 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-role-only-denied-18e332657d59fd8c"] Oct 30 03:54:48.510490 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:54:48.510464 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a444187_bfbc_46ba_b811_bf65124f873c.slice/crio-f91ba5273e3f02e53e517c6081b526186139c9cff049aeb00871330e4e7c83de WatchSource:0}: Error finding container f91ba5273e3f02e53e517c6081b526186139c9cff049aeb00871330e4e7c83de: Status 404 returned error can't find the container with id f91ba5273e3f02e53e517c6081b526186139c9cff049aeb00871330e4e7c83de Oct 30 03:54:48.580907 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:48.580886 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-part-only-denied-18e33262a6541f30" Oct 30 03:54:48.581026 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:48.580901 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-part-only-denied-18e33262a6541f30" event={"ID":"1f6782e1-f253-4835-8d35-3e635f99f081","Type":"ContainerDied","Data":"48ffefb2b76f98c8e6f5a927a039e5a2894c702c445492d9bb2c9d20d2f3d218"} Oct 30 03:54:48.581026 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:48.580938 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48ffefb2b76f98c8e6f5a927a039e5a2894c702c445492d9bb2c9d20d2f3d218" Oct 30 03:54:48.582415 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:48.582385 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-role-only-denied-18e332657d59fd8c" event={"ID":"2a444187-bfbc-46ba-b811-bf65124f873c","Type":"ContainerStarted","Data":"717d99a833370d31790639cb181e82d5b48c92d5562331b738fee5b56763b4ce"} Oct 30 03:54:48.582540 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:48.582424 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-role-only-denied-18e332657d59fd8c" event={"ID":"2a444187-bfbc-46ba-b811-bf65124f873c","Type":"ContainerStarted","Data":"f91ba5273e3f02e53e517c6081b526186139c9cff049aeb00871330e4e7c83de"} Oct 30 03:54:48.624523 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:48.624479 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="collector-network-policy-18e332385848ec88/collector-otlp-role-only-denied-18e332657d59fd8c" podStartSLOduration=0.624467512 podStartE2EDuration="624.467512ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:54:48.623769909 +0000 UTC m=+4400.738084884" watchObservedRunningTime="2026-10-30 03:54:48.624467512 +0000 UTC m=+4400.738782495" Oct 30 03:54:53.943059 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:54:53.943024 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:54:58.634414 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:58.634383 5882 generic.go:356] "Generic (PLEG): container finished" podID="2a444187-bfbc-46ba-b811-bf65124f873c" containerID="717d99a833370d31790639cb181e82d5b48c92d5562331b738fee5b56763b4ce" exitCode=0 Oct 30 03:54:58.634743 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:58.634454 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-role-only-denied-18e332657d59fd8c" event={"ID":"2a444187-bfbc-46ba-b811-bf65124f873c","Type":"ContainerDied","Data":"717d99a833370d31790639cb181e82d5b48c92d5562331b738fee5b56763b4ce"} Oct 30 03:54:59.770764 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:54:59.770740 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-role-only-denied-18e332657d59fd8c" Oct 30 03:55:00.398463 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:00.398386 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-wrong-role-denied-18e33268556b0970"] Oct 30 03:55:00.398779 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:00.398766 5882 cpu_manager.go:414] "containerMap: removing container" podUID="2a444187-bfbc-46ba-b811-bf65124f873c" containerName="curl" Oct 30 03:55:00.398831 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:00.398781 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a444187-bfbc-46ba-b811-bf65124f873c" containerName="curl" Oct 30 03:55:00.398882 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:00.398854 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a444187-bfbc-46ba-b811-bf65124f873c" containerName="curl" Oct 30 03:55:00.401760 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:00.401744 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-wrong-role-denied-18e33268556b0970" Oct 30 03:55:00.412554 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:00.412535 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-wrong-role-denied-18e33268556b0970" Oct 30 03:55:00.419373 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:00.419351 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-wrong-role-denied-18e33268556b0970"] Oct 30 03:55:00.646004 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:00.645976 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-role-only-denied-18e332657d59fd8c" Oct 30 03:55:00.646173 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:00.645975 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-role-only-denied-18e332657d59fd8c" event={"ID":"2a444187-bfbc-46ba-b811-bf65124f873c","Type":"ContainerDied","Data":"f91ba5273e3f02e53e517c6081b526186139c9cff049aeb00871330e4e7c83de"} Oct 30 03:55:00.646173 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:00.646081 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f91ba5273e3f02e53e517c6081b526186139c9cff049aeb00871330e4e7c83de" Oct 30 03:55:00.703466 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:00.703435 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-wrong-role-denied-18e33268556b0970"] Oct 30 03:55:00.704666 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:55:00.704635 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc32fb9ec_12f9_41d7_b2c9_ebaed84cc0d0.slice/crio-ed282acc69968121b399879746f433eb4605bc02948f5f067f228f1faa48cca1 WatchSource:0}: Error finding container ed282acc69968121b399879746f433eb4605bc02948f5f067f228f1faa48cca1: Status 404 returned error can't find the container with id ed282acc69968121b399879746f433eb4605bc02948f5f067f228f1faa48cca1 Oct 30 03:55:01.653121 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:01.653071 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-wrong-role-denied-18e33268556b0970" event={"ID":"c32fb9ec-12f9-41d7-b2c9-ebaed84cc0d0","Type":"ContainerStarted","Data":"5afcf24e9aae5ce2262bff3a5e89f1fe044621310fe58b32f7dc49fafb6dd0d2"} Oct 30 03:55:01.653121 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:01.653129 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-wrong-role-denied-18e33268556b0970" event={"ID":"c32fb9ec-12f9-41d7-b2c9-ebaed84cc0d0","Type":"ContainerStarted","Data":"ed282acc69968121b399879746f433eb4605bc02948f5f067f228f1faa48cca1"} Oct 30 03:55:01.696277 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:01.696205 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="collector-network-policy-18e332385848ec88/collector-otlp-wrong-role-denied-18e33268556b0970" podStartSLOduration=1.696192007 podStartE2EDuration="1.696192007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:55:01.694952389 +0000 UTC m=+4413.809267363" watchObservedRunningTime="2026-10-30 03:55:01.696192007 +0000 UTC m=+4413.810507002" Oct 30 03:55:11.703285 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:11.703252 5882 generic.go:356] "Generic (PLEG): container finished" podID="c32fb9ec-12f9-41d7-b2c9-ebaed84cc0d0" containerID="5afcf24e9aae5ce2262bff3a5e89f1fe044621310fe58b32f7dc49fafb6dd0d2" exitCode=0 Oct 30 03:55:11.703698 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:11.703325 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-wrong-role-denied-18e33268556b0970" event={"ID":"c32fb9ec-12f9-41d7-b2c9-ebaed84cc0d0","Type":"ContainerDied","Data":"5afcf24e9aae5ce2262bff3a5e89f1fe044621310fe58b32f7dc49fafb6dd0d2"} Oct 30 03:55:12.839598 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:12.839573 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-wrong-role-denied-18e33268556b0970" Oct 30 03:55:13.714162 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:13.714135 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e332385848ec88/collector-otlp-wrong-role-denied-18e33268556b0970" Oct 30 03:55:13.714162 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:13.714156 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e332385848ec88/collector-otlp-wrong-role-denied-18e33268556b0970" event={"ID":"c32fb9ec-12f9-41d7-b2c9-ebaed84cc0d0","Type":"ContainerDied","Data":"ed282acc69968121b399879746f433eb4605bc02948f5f067f228f1faa48cca1"} Oct 30 03:55:13.714452 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:13.714186 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed282acc69968121b399879746f433eb4605bc02948f5f067f228f1faa48cca1" Oct 30 03:55:14.631002 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:14.630963 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/targetallocator-collector-allowed-18e3326ba4bd5170"] Oct 30 03:55:14.631575 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:14.631553 5882 cpu_manager.go:414] "containerMap: removing container" podUID="c32fb9ec-12f9-41d7-b2c9-ebaed84cc0d0" containerName="curl" Oct 30 03:55:14.631679 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:14.631577 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c32fb9ec-12f9-41d7-b2c9-ebaed84cc0d0" containerName="curl" Oct 30 03:55:14.631744 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:14.631686 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c32fb9ec-12f9-41d7-b2c9-ebaed84cc0d0" containerName="curl" Oct 30 03:55:14.634760 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:14.634741 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-collector-allowed-18e3326ba4bd5170" Oct 30 03:55:14.645433 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:14.645415 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-collector-allowed-18e3326ba4bd5170" Oct 30 03:55:14.651043 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:14.651022 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/targetallocator-collector-allowed-18e3326ba4bd5170"] Oct 30 03:55:14.940031 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:14.939988 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/targetallocator-collector-allowed-18e3326ba4bd5170"] Oct 30 03:55:15.726241 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:15.726205 5882 generic.go:356] "Generic (PLEG): container finished" podID="62daefb2-b4f8-4280-a22d-adb7a42d8c16" containerID="52efdbfa9decfe856bdbce472d292bd63135d01beaac4da2373a555804481bf9" exitCode=0 Oct 30 03:55:15.726655 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:15.726288 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-collector-allowed-18e3326ba4bd5170" event={"ID":"62daefb2-b4f8-4280-a22d-adb7a42d8c16","Type":"ContainerDied","Data":"52efdbfa9decfe856bdbce472d292bd63135d01beaac4da2373a555804481bf9"} Oct 30 03:55:15.726655 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:15.726327 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-collector-allowed-18e3326ba4bd5170" event={"ID":"62daefb2-b4f8-4280-a22d-adb7a42d8c16","Type":"ContainerStarted","Data":"a8dd7df146ad2e3252e4f455241a49d6897decdf21f061868d70d51cf2631367"} Oct 30 03:55:16.856432 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:16.856407 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-collector-allowed-18e3326ba4bd5170" Oct 30 03:55:17.736402 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:17.736370 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-collector-allowed-18e3326ba4bd5170" event={"ID":"62daefb2-b4f8-4280-a22d-adb7a42d8c16","Type":"ContainerDied","Data":"a8dd7df146ad2e3252e4f455241a49d6897decdf21f061868d70d51cf2631367"} Oct 30 03:55:17.736585 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:17.736410 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-collector-allowed-18e3326ba4bd5170" Oct 30 03:55:17.736585 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:17.736412 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8dd7df146ad2e3252e4f455241a49d6897decdf21f061868d70d51cf2631367" Oct 30 03:55:18.770879 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:18.770841 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/targetallocator-untrusted-denied-18e3326c9ba74439"] Oct 30 03:55:18.771258 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:18.771225 5882 cpu_manager.go:414] "containerMap: removing container" podUID="62daefb2-b4f8-4280-a22d-adb7a42d8c16" containerName="curl" Oct 30 03:55:18.771258 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:18.771237 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62daefb2-b4f8-4280-a22d-adb7a42d8c16" containerName="curl" Oct 30 03:55:18.771333 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:18.771325 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="62daefb2-b4f8-4280-a22d-adb7a42d8c16" containerName="curl" Oct 30 03:55:18.774342 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:18.774327 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-untrusted-denied-18e3326c9ba74439" Oct 30 03:55:18.784812 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:18.784790 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-untrusted-denied-18e3326c9ba74439" Oct 30 03:55:18.788955 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:18.788934 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/targetallocator-untrusted-denied-18e3326c9ba74439"] Oct 30 03:55:19.078272 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:19.078239 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/targetallocator-untrusted-denied-18e3326c9ba74439"] Oct 30 03:55:19.078591 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:55:19.078564 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51ac2f3c_8143_43d6_83bc_05b566b62540.slice/crio-f85fab82a1ca3de4c250d8545a38fc0f0efe8a12be91b15ef34177b4f417a368 WatchSource:0}: Error finding container f85fab82a1ca3de4c250d8545a38fc0f0efe8a12be91b15ef34177b4f417a368: Status 404 returned error can't find the container with id f85fab82a1ca3de4c250d8545a38fc0f0efe8a12be91b15ef34177b4f417a368 Oct 30 03:55:19.748240 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:19.748204 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-untrusted-denied-18e3326c9ba74439" event={"ID":"51ac2f3c-8143-43d6-83bc-05b566b62540","Type":"ContainerStarted","Data":"c44bc1ade1fb3b5535544e845aaca393d7a80d869872fc1eded73a65d1da35d2"} Oct 30 03:55:19.748581 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:19.748248 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-untrusted-denied-18e3326c9ba74439" event={"ID":"51ac2f3c-8143-43d6-83bc-05b566b62540","Type":"ContainerStarted","Data":"f85fab82a1ca3de4c250d8545a38fc0f0efe8a12be91b15ef34177b4f417a368"} Oct 30 03:55:19.796357 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:19.790778 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/targetallocator-untrusted-denied-18e3326c9ba74439" podStartSLOduration=1.790760211 podStartE2EDuration="1.790760211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:55:19.789047278 +0000 UTC m=+4431.903362256" watchObservedRunningTime="2026-10-30 03:55:19.790760211 +0000 UTC m=+4431.905075187" Oct 30 03:55:23.941490 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:55:23.941453 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:55:24.774924 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:24.774893 5882 generic.go:356] "Generic (PLEG): container finished" podID="51ac2f3c-8143-43d6-83bc-05b566b62540" containerID="c44bc1ade1fb3b5535544e845aaca393d7a80d869872fc1eded73a65d1da35d2" exitCode=0 Oct 30 03:55:24.775139 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:24.774970 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-untrusted-denied-18e3326c9ba74439" event={"ID":"51ac2f3c-8143-43d6-83bc-05b566b62540","Type":"ContainerDied","Data":"c44bc1ade1fb3b5535544e845aaca393d7a80d869872fc1eded73a65d1da35d2"} Oct 30 03:55:25.913800 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:25.913774 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-untrusted-denied-18e3326c9ba74439" Oct 30 03:55:26.787182 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:26.787150 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-untrusted-denied-18e3326c9ba74439" Oct 30 03:55:26.787182 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:26.787162 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-untrusted-denied-18e3326c9ba74439" event={"ID":"51ac2f3c-8143-43d6-83bc-05b566b62540","Type":"ContainerDied","Data":"f85fab82a1ca3de4c250d8545a38fc0f0efe8a12be91b15ef34177b4f417a368"} Oct 30 03:55:26.787390 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:26.787202 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f85fab82a1ca3de4c250d8545a38fc0f0efe8a12be91b15ef34177b4f417a368" Oct 30 03:55:26.958922 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:26.958891 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/targetallocator-mtls-allowed-18e3326e843548ad"] Oct 30 03:55:26.959370 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:26.959314 5882 cpu_manager.go:414] "containerMap: removing container" podUID="51ac2f3c-8143-43d6-83bc-05b566b62540" containerName="curl" Oct 30 03:55:26.959370 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:26.959330 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ac2f3c-8143-43d6-83bc-05b566b62540" containerName="curl" Oct 30 03:55:26.959489 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:26.959413 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ac2f3c-8143-43d6-83bc-05b566b62540" containerName="curl" Oct 30 03:55:26.962777 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:26.962757 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-mtls-allowed-18e3326e843548ad" Oct 30 03:55:26.979776 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:26.979750 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/targetallocator-mtls-allowed-18e3326e843548ad"] Oct 30 03:55:26.979938 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:26.979925 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 03:55:27.019485 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:27.019451 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-certificate\" (UniqueName: \"kubernetes.io/secret/df9e7f18-babc-4572-b409-7bc8845d02b0-client-certificate\") pod \"targetallocator-mtls-allowed-18e3326e843548ad\" (UID: \"df9e7f18-babc-4572-b409-7bc8845d02b0\") " pod="redhat-ods-monitoring/targetallocator-mtls-allowed-18e3326e843548ad" Oct 30 03:55:27.120137 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:27.120061 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-certificate\" (UniqueName: \"kubernetes.io/secret/df9e7f18-babc-4572-b409-7bc8845d02b0-client-certificate\") pod \"targetallocator-mtls-allowed-18e3326e843548ad\" (UID: \"df9e7f18-babc-4572-b409-7bc8845d02b0\") " pod="redhat-ods-monitoring/targetallocator-mtls-allowed-18e3326e843548ad" Oct 30 03:55:27.122515 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:27.122488 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-certificate\" (UniqueName: \"kubernetes.io/secret/df9e7f18-babc-4572-b409-7bc8845d02b0-client-certificate\") pod \"targetallocator-mtls-allowed-18e3326e843548ad\" (UID: \"df9e7f18-babc-4572-b409-7bc8845d02b0\") " pod="redhat-ods-monitoring/targetallocator-mtls-allowed-18e3326e843548ad" Oct 30 03:55:27.274935 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:27.274879 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-mtls-allowed-18e3326e843548ad" Oct 30 03:55:27.566819 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:27.566790 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/targetallocator-mtls-allowed-18e3326e843548ad"] Oct 30 03:55:27.567330 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:55:27.567302 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf9e7f18_babc_4572_b409_7bc8845d02b0.slice/crio-fcdc8b11eb7c4904bd2859aa206f2b5e6275a890152609564527d5236bedb205 WatchSource:0}: Error finding container fcdc8b11eb7c4904bd2859aa206f2b5e6275a890152609564527d5236bedb205: Status 404 returned error can't find the container with id fcdc8b11eb7c4904bd2859aa206f2b5e6275a890152609564527d5236bedb205 Oct 30 03:55:27.793770 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:27.793729 5882 generic.go:356] "Generic (PLEG): container finished" podID="df9e7f18-babc-4572-b409-7bc8845d02b0" containerID="e2dbdc1bac6993055089bf572a3d07fec69bb8af6d944663059386db9caa643f" exitCode=0 Oct 30 03:55:27.793945 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:27.793778 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-mtls-allowed-18e3326e843548ad" event={"ID":"df9e7f18-babc-4572-b409-7bc8845d02b0","Type":"ContainerDied","Data":"e2dbdc1bac6993055089bf572a3d07fec69bb8af6d944663059386db9caa643f"} Oct 30 03:55:27.793945 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:27.793820 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-mtls-allowed-18e3326e843548ad" event={"ID":"df9e7f18-babc-4572-b409-7bc8845d02b0","Type":"ContainerStarted","Data":"fcdc8b11eb7c4904bd2859aa206f2b5e6275a890152609564527d5236bedb205"} Oct 30 03:55:28.927723 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:28.927699 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-mtls-allowed-18e3326e843548ad" Oct 30 03:55:28.934807 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:28.934763 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df9e7f18-babc-4572-b409-7bc8845d02b0-client-certificate\" (UniqueName: \"kubernetes.io/secret/df9e7f18-babc-4572-b409-7bc8845d02b0-client-certificate\") pod \"df9e7f18-babc-4572-b409-7bc8845d02b0\" (UID: \"df9e7f18-babc-4572-b409-7bc8845d02b0\") " Oct 30 03:55:28.936639 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:28.936601 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df9e7f18-babc-4572-b409-7bc8845d02b0-client-certificate" pod "df9e7f18-babc-4572-b409-7bc8845d02b0" (UID: "df9e7f18-babc-4572-b409-7bc8845d02b0"). InnerVolumeSpecName "client-certificate". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:55:29.035355 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:29.035325 5882 reconciler_common.go:299] "Volume detached for volume \"client-certificate\" (UniqueName: \"kubernetes.io/secret/df9e7f18-babc-4572-b409-7bc8845d02b0-client-certificate\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:55:29.065449 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:29.065372 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e3326f0214aebe"] Oct 30 03:55:29.065767 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:29.065754 5882 cpu_manager.go:414] "containerMap: removing container" podUID="df9e7f18-babc-4572-b409-7bc8845d02b0" containerName="curl" Oct 30 03:55:29.065829 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:29.065769 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df9e7f18-babc-4572-b409-7bc8845d02b0" containerName="curl" Oct 30 03:55:29.065869 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:29.065845 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="df9e7f18-babc-4572-b409-7bc8845d02b0" containerName="curl" Oct 30 03:55:29.068814 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:29.068797 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e3326f0214aebe" Oct 30 03:55:29.079447 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:29.079425 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e3326f0214aebe" Oct 30 03:55:29.085434 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:29.085410 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e3326f0214aebe"] Oct 30 03:55:29.378291 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:29.378260 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e3326f0214aebe"] Oct 30 03:55:29.378807 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:55:29.378785 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b4d68d2_2079_4b28_8dfa_0972de4a724c.slice/crio-e2b83d7436fa572599999585727908bfaf25f5d6d6db5548c3b092fc4b0a513e WatchSource:0}: Error finding container e2b83d7436fa572599999585727908bfaf25f5d6d6db5548c3b092fc4b0a513e: Status 404 returned error can't find the container with id e2b83d7436fa572599999585727908bfaf25f5d6d6db5548c3b092fc4b0a513e Oct 30 03:55:29.805732 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:29.805704 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-mtls-allowed-18e3326e843548ad" Oct 30 03:55:29.805732 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:29.805719 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-mtls-allowed-18e3326e843548ad" event={"ID":"df9e7f18-babc-4572-b409-7bc8845d02b0","Type":"ContainerDied","Data":"fcdc8b11eb7c4904bd2859aa206f2b5e6275a890152609564527d5236bedb205"} Oct 30 03:55:29.805947 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:29.805751 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcdc8b11eb7c4904bd2859aa206f2b5e6275a890152609564527d5236bedb205" Oct 30 03:55:29.807420 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:29.807390 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e3326f0214aebe" event={"ID":"7b4d68d2-2079-4b28-8dfa-0972de4a724c","Type":"ContainerStarted","Data":"60b42b91f08d52fd5986bf71f3578614a0160faa38acab6956157d1879306686"} Oct 30 03:55:29.807545 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:29.807430 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e3326f0214aebe" event={"ID":"7b4d68d2-2079-4b28-8dfa-0972de4a724c","Type":"ContainerStarted","Data":"e2b83d7436fa572599999585727908bfaf25f5d6d6db5548c3b092fc4b0a513e"} Oct 30 03:55:29.850090 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:29.850045 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e3326f0214aebe" podStartSLOduration=0.850032833 podStartE2EDuration="850.032833ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:55:29.848951898 +0000 UTC m=+4441.963266873" watchObservedRunningTime="2026-10-30 03:55:29.850032833 +0000 UTC m=+4441.964347808" Oct 30 03:55:34.835207 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:34.835170 5882 generic.go:356] "Generic (PLEG): container finished" podID="7b4d68d2-2079-4b28-8dfa-0972de4a724c" containerID="60b42b91f08d52fd5986bf71f3578614a0160faa38acab6956157d1879306686" exitCode=0 Oct 30 03:55:34.835597 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:34.835246 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e3326f0214aebe" event={"ID":"7b4d68d2-2079-4b28-8dfa-0972de4a724c","Type":"ContainerDied","Data":"60b42b91f08d52fd5986bf71f3578614a0160faa38acab6956157d1879306686"} Oct 30 03:55:35.973263 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:35.973239 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e3326f0214aebe" Oct 30 03:55:36.847494 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:36.847466 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e3326f0214aebe" Oct 30 03:55:36.847694 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:36.847517 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e3326f0214aebe" event={"ID":"7b4d68d2-2079-4b28-8dfa-0972de4a724c","Type":"ContainerDied","Data":"e2b83d7436fa572599999585727908bfaf25f5d6d6db5548c3b092fc4b0a513e"} Oct 30 03:55:36.847694 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:36.847557 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2b83d7436fa572599999585727908bfaf25f5d6d6db5548c3b092fc4b0a513e" Oct 30 03:55:37.165172 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.165093 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e3326f0214aebe"] Oct 30 03:55:37.170953 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.170929 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e3326f0214aebe"] Oct 30 03:55:37.191823 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.191796 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/targetallocator-mtls-allowed-18e3326e843548ad"] Oct 30 03:55:37.194114 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.194091 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/targetallocator-mtls-allowed-18e3326e843548ad"] Oct 30 03:55:37.217125 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.217095 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/targetallocator-untrusted-denied-18e3326c9ba74439"] Oct 30 03:55:37.221496 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.221472 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/targetallocator-untrusted-denied-18e3326c9ba74439"] Oct 30 03:55:37.242738 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.242713 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/targetallocator-collector-allowed-18e3326ba4bd5170"] Oct 30 03:55:37.246270 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.246251 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/targetallocator-collector-allowed-18e3326ba4bd5170"] Oct 30 03:55:37.267020 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.266998 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-wrong-role-denied-18e33268556b0970"] Oct 30 03:55:37.270735 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.270717 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-wrong-role-denied-18e33268556b0970"] Oct 30 03:55:37.291154 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.291131 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-role-only-denied-18e332657d59fd8c"] Oct 30 03:55:37.295825 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.295807 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-role-only-denied-18e332657d59fd8c"] Oct 30 03:55:37.316209 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.316181 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-part-only-denied-18e33262a6541f30"] Oct 30 03:55:37.321784 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.321763 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-part-only-denied-18e33262a6541f30"] Oct 30 03:55:37.342518 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.342495 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-unlabeled-denied-18e3325fcef86f7a"] Oct 30 03:55:37.346526 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.346507 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-unlabeled-denied-18e3325fcef86f7a"] Oct 30 03:55:37.368644 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.368609 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-scheduler-allowed-18e3325f5047df47"] Oct 30 03:55:37.374914 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.374895 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-scheduler-allowed-18e3325f5047df47"] Oct 30 03:55:37.395171 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.395146 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-leader-prefill-allowed-18e3325e596a7ee3"] Oct 30 03:55:37.398832 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.398814 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-leader-prefill-allowed-18e3325e596a7ee3"] Oct 30 03:55:37.419583 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.419527 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-worker-prefill-allowed-18e3325d6279ed00"] Oct 30 03:55:37.423784 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.423766 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-worker-prefill-allowed-18e3325d6279ed00"] Oct 30 03:55:37.444896 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.444873 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-leader-allowed-18e3325c6b9fda3c"] Oct 30 03:55:37.448607 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.448589 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-leader-allowed-18e3325c6b9fda3c"] Oct 30 03:55:37.471603 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.471582 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-worker-allowed-18e3325b74c17aa9"] Oct 30 03:55:37.476318 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.476299 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-worker-allowed-18e3325b74c17aa9"] Oct 30 03:55:37.496818 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.496798 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-prefill-allowed-18e3325a7dd8aa7c"] Oct 30 03:55:37.500324 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.500305 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-prefill-allowed-18e3325a7dd8aa7c"] Oct 30 03:55:37.520943 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.520921 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-workload-allowed-18e33259ff43b8a5"] Oct 30 03:55:37.525223 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.525205 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e332385848ec88/collector-otlp-workload-allowed-18e33259ff43b8a5"] Oct 30 03:55:37.545992 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.545964 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3325726f52a13"] Oct 30 03:55:37.551797 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.551778 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3325726f52a13"] Oct 30 03:55:37.573982 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.573961 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-role-only-denied-18e332544f0e5c1d"] Oct 30 03:55:37.578176 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.578158 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-role-only-denied-18e332544f0e5c1d"] Oct 30 03:55:37.598547 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.598523 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-part-only-denied-18e33250ffd97e46"] Oct 30 03:55:37.604779 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.604758 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-part-only-denied-18e33250ffd97e46"] Oct 30 03:55:37.625911 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.625889 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3324db09a84c9"] Oct 30 03:55:37.630066 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.630047 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3324db09a84c9"] Oct 30 03:55:37.650926 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.650908 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e3324a6073205f"] Oct 30 03:55:37.655054 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.655037 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e3324a6073205f"] Oct 30 03:55:37.676933 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.676878 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e332478850b604"] Oct 30 03:55:37.681273 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.681255 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e332478850b604"] Oct 30 03:55:37.703867 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.703843 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e33244b15ff220"] Oct 30 03:55:37.708491 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.708472 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e33244b15ff220"] Oct 30 03:55:37.729556 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.729537 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-leader-allowed-18e33241d5b86fde"] Oct 30 03:55:37.734088 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.734072 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-leader-allowed-18e33241d5b86fde"] Oct 30 03:55:37.755030 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.755008 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-worker-allowed-18e3323efdca6819"] Oct 30 03:55:37.759482 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.759464 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-worker-allowed-18e3323efdca6819"] Oct 30 03:55:37.780154 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.780135 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-prefill-allowed-18e3323bad47956e"] Oct 30 03:55:37.784405 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.784381 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-prefill-allowed-18e3323bad47956e"] Oct 30 03:55:37.874395 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.874353 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-prometheus-allowed-18e332341108ead6"] Oct 30 03:55:37.884749 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:37.884719 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-prometheus-allowed-18e332341108ead6"] Oct 30 03:55:38.009978 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.009943 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f6782e1-f253-4835-8d35-3e635f99f081" path="/var/lib/kubelet/pods/1f6782e1-f253-4835-8d35-3e635f99f081/volumes" Oct 30 03:55:38.010192 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.010179 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29e7a712-9985-4dd1-a66a-4ec81beae6b1" path="/var/lib/kubelet/pods/29e7a712-9985-4dd1-a66a-4ec81beae6b1/volumes" Oct 30 03:55:38.010383 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.010371 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a444187-bfbc-46ba-b811-bf65124f873c" path="/var/lib/kubelet/pods/2a444187-bfbc-46ba-b811-bf65124f873c/volumes" Oct 30 03:55:38.010570 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.010559 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aacfaed-75d5-4217-9f88-caa00e952522" path="/var/lib/kubelet/pods/2aacfaed-75d5-4217-9f88-caa00e952522/volumes" Oct 30 03:55:38.010779 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.010768 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4140e1d6-b13d-4307-b299-6a099f1f499a" path="/var/lib/kubelet/pods/4140e1d6-b13d-4307-b299-6a099f1f499a/volumes" Oct 30 03:55:38.011294 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.011279 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="485f30c1-fe8a-4369-b9b5-168c5b8ae02d" path="/var/lib/kubelet/pods/485f30c1-fe8a-4369-b9b5-168c5b8ae02d/volumes" Oct 30 03:55:38.011478 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.011467 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51ac2f3c-8143-43d6-83bc-05b566b62540" path="/var/lib/kubelet/pods/51ac2f3c-8143-43d6-83bc-05b566b62540/volumes" Oct 30 03:55:38.011691 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.011680 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60629d32-8098-4865-b472-c2ebb38effb9" path="/var/lib/kubelet/pods/60629d32-8098-4865-b472-c2ebb38effb9/volumes" Oct 30 03:55:38.011879 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.011869 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="624d48a6-a547-422b-8c32-83097a4f05cc" path="/var/lib/kubelet/pods/624d48a6-a547-422b-8c32-83097a4f05cc/volumes" Oct 30 03:55:38.012061 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.012051 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62daefb2-b4f8-4280-a22d-adb7a42d8c16" path="/var/lib/kubelet/pods/62daefb2-b4f8-4280-a22d-adb7a42d8c16/volumes" Oct 30 03:55:38.012520 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.012508 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7038f143-bce1-418d-ab3e-ce833c2c1410" path="/var/lib/kubelet/pods/7038f143-bce1-418d-ab3e-ce833c2c1410/volumes" Oct 30 03:55:38.012725 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.012714 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72875129-c483-4607-a4ab-60b0ee81e55f" path="/var/lib/kubelet/pods/72875129-c483-4607-a4ab-60b0ee81e55f/volumes" Oct 30 03:55:38.012913 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.012903 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="732febb0-93cb-4b4a-9d93-35fc2ed63a6b" path="/var/lib/kubelet/pods/732febb0-93cb-4b4a-9d93-35fc2ed63a6b/volumes" Oct 30 03:55:38.013094 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.013084 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b4d68d2-2079-4b28-8dfa-0972de4a724c" path="/var/lib/kubelet/pods/7b4d68d2-2079-4b28-8dfa-0972de4a724c/volumes" Oct 30 03:55:38.013275 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.013265 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c0541c9-85f7-42ac-9e86-83f4e8290bd6" path="/var/lib/kubelet/pods/8c0541c9-85f7-42ac-9e86-83f4e8290bd6/volumes" Oct 30 03:55:38.013739 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.013726 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="924992da-80de-435b-ad4c-bbfd8dc26af2" path="/var/lib/kubelet/pods/924992da-80de-435b-ad4c-bbfd8dc26af2/volumes" Oct 30 03:55:38.013918 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.013908 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a187a012-2907-4052-bd2d-33241877ac55" path="/var/lib/kubelet/pods/a187a012-2907-4052-bd2d-33241877ac55/volumes" Oct 30 03:55:38.014100 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.014089 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf30bda8-aefa-4b4b-89dc-edad5acb7850" path="/var/lib/kubelet/pods/bf30bda8-aefa-4b4b-89dc-edad5acb7850/volumes" Oct 30 03:55:38.014318 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.014305 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c32fb9ec-12f9-41d7-b2c9-ebaed84cc0d0" path="/var/lib/kubelet/pods/c32fb9ec-12f9-41d7-b2c9-ebaed84cc0d0/volumes" Oct 30 03:55:38.014509 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.014497 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d38e1a10-1192-418b-bfbb-65fa776781fe" path="/var/lib/kubelet/pods/d38e1a10-1192-418b-bfbb-65fa776781fe/volumes" Oct 30 03:55:38.015081 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.015066 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df9e7f18-babc-4572-b409-7bc8845d02b0" path="/var/lib/kubelet/pods/df9e7f18-babc-4572-b409-7bc8845d02b0/volumes" Oct 30 03:55:38.015302 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.015291 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e07f19bb-04b5-41fb-8645-6e7ee8c4bd8d" path="/var/lib/kubelet/pods/e07f19bb-04b5-41fb-8645-6e7ee8c4bd8d/volumes" Oct 30 03:55:38.015488 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.015478 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9a42ed6-aaf4-43df-ae69-7400c7fe25c7" path="/var/lib/kubelet/pods/e9a42ed6-aaf4-43df-ae69-7400c7fe25c7/volumes" Oct 30 03:55:38.015689 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.015678 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec856d2e-d84d-4b96-aebd-4c69e2662633" path="/var/lib/kubelet/pods/ec856d2e-d84d-4b96-aebd-4c69e2662633/volumes" Oct 30 03:55:38.016149 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.016125 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2800745-e6b3-44b0-a20a-c11b893ebcfb" path="/var/lib/kubelet/pods/f2800745-e6b3-44b0-a20a-c11b893ebcfb/volumes" Oct 30 03:55:38.016379 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:38.016363 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d0cec5-4013-41f1-9fbb-d1abd1295803" path="/var/lib/kubelet/pods/f9d0cec5-4013-41f1-9fbb-d1abd1295803/volumes" Oct 30 03:55:39.018686 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:39.018655 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-namespace-auth-18e33271148ad497/unenrolled-spoof-denied-18e332714f8c65e1"] Oct 30 03:55:39.019059 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:39.019049 5882 cpu_manager.go:414] "containerMap: removing container" podUID="7b4d68d2-2079-4b28-8dfa-0972de4a724c" containerName="curl" Oct 30 03:55:39.019114 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:39.019062 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4d68d2-2079-4b28-8dfa-0972de4a724c" containerName="curl" Oct 30 03:55:39.019145 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:39.019116 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4d68d2-2079-4b28-8dfa-0972de4a724c" containerName="curl" Oct 30 03:55:39.023983 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:39.023966 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e33271148ad497/unenrolled-spoof-denied-18e332714f8c65e1" Oct 30 03:55:39.038172 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:39.038149 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-namespace-auth-18e33271148ad497/unenrolled-spoof-denied-18e332714f8c65e1"] Oct 30 03:55:39.051589 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:39.051560 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"collector-namespace-auth-18e33271148ad497\"/\"default-dockercfg-nsd8r\"" Oct 30 03:55:39.055658 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:39.055636 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e33271148ad497/unenrolled-spoof-denied-18e332714f8c65e1" Oct 30 03:55:39.346900 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:55:39.346861 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae1f3b5c_20d3_46b1_8564_6587872cd2b7.slice/crio-9242ab5d63b4fd21828a0038a750c27beb2f4d2e04dde063c507312e2f9459bf WatchSource:0}: Error finding container 9242ab5d63b4fd21828a0038a750c27beb2f4d2e04dde063c507312e2f9459bf: Status 404 returned error can't find the container with id 9242ab5d63b4fd21828a0038a750c27beb2f4d2e04dde063c507312e2f9459bf Oct 30 03:55:39.346900 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:39.346865 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-namespace-auth-18e33271148ad497/unenrolled-spoof-denied-18e332714f8c65e1"] Oct 30 03:55:39.867856 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:39.867817 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e33271148ad497/unenrolled-spoof-denied-18e332714f8c65e1" event={"ID":"ae1f3b5c-20d3-46b1-8564-6587872cd2b7","Type":"ContainerStarted","Data":"e2eeca322707c3df3ce1e609bf00b170425816c143404bd107c8097590f6dc9d"} Oct 30 03:55:39.868041 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:39.867865 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e33271148ad497/unenrolled-spoof-denied-18e332714f8c65e1" event={"ID":"ae1f3b5c-20d3-46b1-8564-6587872cd2b7","Type":"ContainerStarted","Data":"9242ab5d63b4fd21828a0038a750c27beb2f4d2e04dde063c507312e2f9459bf"} Oct 30 03:55:39.911283 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:39.911237 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="collector-namespace-auth-18e33271148ad497/unenrolled-spoof-denied-18e332714f8c65e1" podStartSLOduration=1.9112261 podStartE2EDuration="1.9112261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:55:39.909808479 +0000 UTC m=+4452.024123454" watchObservedRunningTime="2026-10-30 03:55:39.9112261 +0000 UTC m=+4452.025541072" Oct 30 03:55:40.440046 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:40.440008 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd"] Oct 30 03:55:40.443332 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:40.443313 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" Oct 30 03:55:40.477180 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:40.477155 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd"] Oct 30 03:55:40.534426 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:40.534399 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/5d11b295-8a7a-423c-812f-c27f91049256-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-657cd485df-7lgtd\" (UID: \"5d11b295-8a7a-423c-812f-c27f91049256\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" Oct 30 03:55:40.534546 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:40.534431 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp8sx\" (UniqueName: \"kubernetes.io/projected/5d11b295-8a7a-423c-812f-c27f91049256-kube-api-access-hp8sx\") pod \"data-science-collector-targetallocator-657cd485df-7lgtd\" (UID: \"5d11b295-8a7a-423c-812f-c27f91049256\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" Oct 30 03:55:40.534546 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:40.534527 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/5d11b295-8a7a-423c-812f-c27f91049256-ta-internal\") pod \"data-science-collector-targetallocator-657cd485df-7lgtd\" (UID: \"5d11b295-8a7a-423c-812f-c27f91049256\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" Oct 30 03:55:40.635200 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:40.635169 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/5d11b295-8a7a-423c-812f-c27f91049256-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-657cd485df-7lgtd\" (UID: \"5d11b295-8a7a-423c-812f-c27f91049256\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" Oct 30 03:55:40.635331 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:40.635208 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp8sx\" (UniqueName: \"kubernetes.io/projected/5d11b295-8a7a-423c-812f-c27f91049256-kube-api-access-hp8sx\") pod \"data-science-collector-targetallocator-657cd485df-7lgtd\" (UID: \"5d11b295-8a7a-423c-812f-c27f91049256\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" Oct 30 03:55:40.635331 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:40.635277 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/5d11b295-8a7a-423c-812f-c27f91049256-ta-internal\") pod \"data-science-collector-targetallocator-657cd485df-7lgtd\" (UID: \"5d11b295-8a7a-423c-812f-c27f91049256\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" Oct 30 03:55:40.636005 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:40.635965 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/5d11b295-8a7a-423c-812f-c27f91049256-ta-internal\") pod \"data-science-collector-targetallocator-657cd485df-7lgtd\" (UID: \"5d11b295-8a7a-423c-812f-c27f91049256\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" Oct 30 03:55:40.637595 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:40.637578 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/5d11b295-8a7a-423c-812f-c27f91049256-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-657cd485df-7lgtd\" (UID: \"5d11b295-8a7a-423c-812f-c27f91049256\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" Oct 30 03:55:40.660324 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:40.660302 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp8sx\" (UniqueName: \"kubernetes.io/projected/5d11b295-8a7a-423c-812f-c27f91049256-kube-api-access-hp8sx\") pod \"data-science-collector-targetallocator-657cd485df-7lgtd\" (UID: \"5d11b295-8a7a-423c-812f-c27f91049256\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" Oct 30 03:55:40.754593 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:40.754565 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" Oct 30 03:55:41.047138 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:41.047098 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd"] Oct 30 03:55:41.047542 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:55:41.047515 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d11b295_8a7a_423c_812f_c27f91049256.slice/crio-48bfd1ab7cc11049002d27a23b9770b79ef942e5801b4ab2a97afbc4fc6e3a83 WatchSource:0}: Error finding container 48bfd1ab7cc11049002d27a23b9770b79ef942e5801b4ab2a97afbc4fc6e3a83: Status 404 returned error can't find the container with id 48bfd1ab7cc11049002d27a23b9770b79ef942e5801b4ab2a97afbc4fc6e3a83 Oct 30 03:55:41.049741 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:41.049711 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:55:41.049829 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:41.049765 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:55:41.880293 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:41.880257 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" event={"ID":"5d11b295-8a7a-423c-812f-c27f91049256","Type":"ContainerStarted","Data":"1ecd7b076c2d022ff7ec77315114c81e71916dff67933c1ded218a3cc0aacb5e"} Oct 30 03:55:41.880293 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:41.880298 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" event={"ID":"5d11b295-8a7a-423c-812f-c27f91049256","Type":"ContainerStarted","Data":"48bfd1ab7cc11049002d27a23b9770b79ef942e5801b4ab2a97afbc4fc6e3a83"} Oct 30 03:55:41.880789 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:41.880386 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" Oct 30 03:55:41.882081 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:41.882058 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" Oct 30 03:55:41.925427 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:41.925379 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" podStartSLOduration=1.92536717 podStartE2EDuration="1.92536717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:55:41.924788898 +0000 UTC m=+4454.039103872" watchObservedRunningTime="2026-10-30 03:55:41.92536717 +0000 UTC m=+4454.039682139" Oct 30 03:55:41.998050 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:41.998017 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq"] Oct 30 03:55:41.998232 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:41.998207 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" podUID="2574d0b1-493c-4a95-b047-7d28c5495896" containerName="ta-container" containerID="cri-o://266f98e92335261b581ed8ab36586cb081d5e8c2145113e89cf5babdab4b4fdf" gracePeriod=30 Oct 30 03:55:42.236559 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:42.236536 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" Oct 30 03:55:42.351885 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:42.351798 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2574d0b1-493c-4a95-b047-7d28c5495896-ta-internal\" (UniqueName: \"kubernetes.io/configmap/2574d0b1-493c-4a95-b047-7d28c5495896-ta-internal\") pod \"2574d0b1-493c-4a95-b047-7d28c5495896\" (UID: \"2574d0b1-493c-4a95-b047-7d28c5495896\") " Oct 30 03:55:42.351885 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:42.351878 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2574d0b1-493c-4a95-b047-7d28c5495896-kube-api-access-mfd62\" (UniqueName: \"kubernetes.io/projected/2574d0b1-493c-4a95-b047-7d28c5495896-kube-api-access-mfd62\") pod \"2574d0b1-493c-4a95-b047-7d28c5495896\" (UID: \"2574d0b1-493c-4a95-b047-7d28c5495896\") " Oct 30 03:55:42.352122 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:42.351929 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2574d0b1-493c-4a95-b047-7d28c5495896-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2574d0b1-493c-4a95-b047-7d28c5495896-data-science-collector-ta-server-cert\") pod \"2574d0b1-493c-4a95-b047-7d28c5495896\" (UID: \"2574d0b1-493c-4a95-b047-7d28c5495896\") " Oct 30 03:55:42.352179 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:42.352136 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2574d0b1-493c-4a95-b047-7d28c5495896-ta-internal" pod "2574d0b1-493c-4a95-b047-7d28c5495896" (UID: "2574d0b1-493c-4a95-b047-7d28c5495896"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:55:42.354018 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:42.353997 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2574d0b1-493c-4a95-b047-7d28c5495896-kube-api-access-mfd62" pod "2574d0b1-493c-4a95-b047-7d28c5495896" (UID: "2574d0b1-493c-4a95-b047-7d28c5495896"). InnerVolumeSpecName "kube-api-access-mfd62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:55:42.354103 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:42.354058 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2574d0b1-493c-4a95-b047-7d28c5495896-data-science-collector-ta-server-cert" pod "2574d0b1-493c-4a95-b047-7d28c5495896" (UID: "2574d0b1-493c-4a95-b047-7d28c5495896"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:55:42.452968 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:42.452925 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2574d0b1-493c-4a95-b047-7d28c5495896-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:55:42.452968 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:42.452965 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfd62\" (UniqueName: \"kubernetes.io/projected/2574d0b1-493c-4a95-b047-7d28c5495896-kube-api-access-mfd62\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:55:42.452968 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:42.452978 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2574d0b1-493c-4a95-b047-7d28c5495896-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:55:42.886280 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:42.886247 5882 generic.go:356] "Generic (PLEG): container finished" podID="2574d0b1-493c-4a95-b047-7d28c5495896" containerID="266f98e92335261b581ed8ab36586cb081d5e8c2145113e89cf5babdab4b4fdf" exitCode=0 Oct 30 03:55:42.886792 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:42.886294 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" Oct 30 03:55:42.886792 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:42.886338 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" event={"ID":"2574d0b1-493c-4a95-b047-7d28c5495896","Type":"ContainerDied","Data":"266f98e92335261b581ed8ab36586cb081d5e8c2145113e89cf5babdab4b4fdf"} Oct 30 03:55:42.886792 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:42.886384 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq" event={"ID":"2574d0b1-493c-4a95-b047-7d28c5495896","Type":"ContainerDied","Data":"61ead3dfc0ccf1a67052b9a8b8ebe96ce50d0b56f07c69002b0ff1068943bd20"} Oct 30 03:55:42.886792 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:42.886407 5882 scope.go:122] "RemoveContainer" containerID="266f98e92335261b581ed8ab36586cb081d5e8c2145113e89cf5babdab4b4fdf" Oct 30 03:55:42.895831 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:42.895813 5882 scope.go:122] "RemoveContainer" containerID="266f98e92335261b581ed8ab36586cb081d5e8c2145113e89cf5babdab4b4fdf" Oct 30 03:55:42.896086 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:55:42.896067 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"266f98e92335261b581ed8ab36586cb081d5e8c2145113e89cf5babdab4b4fdf\": container with ID starting with 266f98e92335261b581ed8ab36586cb081d5e8c2145113e89cf5babdab4b4fdf not found: ID does not exist" containerID="266f98e92335261b581ed8ab36586cb081d5e8c2145113e89cf5babdab4b4fdf" Oct 30 03:55:42.896142 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:42.896094 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"266f98e92335261b581ed8ab36586cb081d5e8c2145113e89cf5babdab4b4fdf"} err="failed to get container status \"266f98e92335261b581ed8ab36586cb081d5e8c2145113e89cf5babdab4b4fdf\": rpc error: code = NotFound desc = could not find container \"266f98e92335261b581ed8ab36586cb081d5e8c2145113e89cf5babdab4b4fdf\": container with ID starting with 266f98e92335261b581ed8ab36586cb081d5e8c2145113e89cf5babdab4b4fdf not found: ID does not exist" Oct 30 03:55:42.941212 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:42.941182 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq"] Oct 30 03:55:42.955303 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:42.955275 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5bddf8ff7c-s97xq"] Oct 30 03:55:44.009895 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:44.009860 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2574d0b1-493c-4a95-b047-7d28c5495896" path="/var/lib/kubelet/pods/2574d0b1-493c-4a95-b047-7d28c5495896/volumes" Oct 30 03:55:49.924140 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:49.924107 5882 generic.go:356] "Generic (PLEG): container finished" podID="ae1f3b5c-20d3-46b1-8564-6587872cd2b7" containerID="e2eeca322707c3df3ce1e609bf00b170425816c143404bd107c8097590f6dc9d" exitCode=0 Oct 30 03:55:49.924534 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:49.924184 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e33271148ad497/unenrolled-spoof-denied-18e332714f8c65e1" event={"ID":"ae1f3b5c-20d3-46b1-8564-6587872cd2b7","Type":"ContainerDied","Data":"e2eeca322707c3df3ce1e609bf00b170425816c143404bd107c8097590f6dc9d"} Oct 30 03:55:51.055704 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:51.055680 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e33271148ad497/unenrolled-spoof-denied-18e332714f8c65e1" Oct 30 03:55:51.412476 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:51.412439 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-namespace-auth-18e33271148ad497/enrolled-inference-allowed-18e3327435fc530a"] Oct 30 03:55:51.412830 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:51.412815 5882 cpu_manager.go:414] "containerMap: removing container" podUID="2574d0b1-493c-4a95-b047-7d28c5495896" containerName="ta-container" Oct 30 03:55:51.412891 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:51.412832 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2574d0b1-493c-4a95-b047-7d28c5495896" containerName="ta-container" Oct 30 03:55:51.412891 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:51.412870 5882 cpu_manager.go:414] "containerMap: removing container" podUID="ae1f3b5c-20d3-46b1-8564-6587872cd2b7" containerName="curl" Oct 30 03:55:51.412891 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:51.412875 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1f3b5c-20d3-46b1-8564-6587872cd2b7" containerName="curl" Oct 30 03:55:51.412987 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:51.412938 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1f3b5c-20d3-46b1-8564-6587872cd2b7" containerName="curl" Oct 30 03:55:51.412987 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:51.412947 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2574d0b1-493c-4a95-b047-7d28c5495896" containerName="ta-container" Oct 30 03:55:51.415820 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:51.415804 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e33271148ad497/enrolled-inference-allowed-18e3327435fc530a" Oct 30 03:55:51.426375 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:51.426357 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e33271148ad497/enrolled-inference-allowed-18e3327435fc530a" Oct 30 03:55:51.432284 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:51.432234 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-namespace-auth-18e33271148ad497/enrolled-inference-allowed-18e3327435fc530a"] Oct 30 03:55:51.719206 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:51.719179 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-namespace-auth-18e33271148ad497/enrolled-inference-allowed-18e3327435fc530a"] Oct 30 03:55:51.720247 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:55:51.720223 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0861670_b1c5_4043_b926_66570537e342.slice/crio-d698025239e3532ca760b99e3dd7cbe255ba335f0702bcc67c4708793163c8ce WatchSource:0}: Error finding container d698025239e3532ca760b99e3dd7cbe255ba335f0702bcc67c4708793163c8ce: Status 404 returned error can't find the container with id d698025239e3532ca760b99e3dd7cbe255ba335f0702bcc67c4708793163c8ce Oct 30 03:55:51.935909 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:51.935875 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e33271148ad497/unenrolled-spoof-denied-18e332714f8c65e1" Oct 30 03:55:51.935909 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:51.935883 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e33271148ad497/unenrolled-spoof-denied-18e332714f8c65e1" event={"ID":"ae1f3b5c-20d3-46b1-8564-6587872cd2b7","Type":"ContainerDied","Data":"9242ab5d63b4fd21828a0038a750c27beb2f4d2e04dde063c507312e2f9459bf"} Oct 30 03:55:51.935909 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:51.935924 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9242ab5d63b4fd21828a0038a750c27beb2f4d2e04dde063c507312e2f9459bf" Oct 30 03:55:51.937607 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:51.937583 5882 generic.go:356] "Generic (PLEG): container finished" podID="d0861670-b1c5-4043-b926-66570537e342" containerID="4b1130e15ccc7f12e75147dc49d2c2fb79f2c9e33d30bfe202de9962a44201ab" exitCode=0 Oct 30 03:55:51.937703 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:51.937647 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e33271148ad497/enrolled-inference-allowed-18e3327435fc530a" event={"ID":"d0861670-b1c5-4043-b926-66570537e342","Type":"ContainerDied","Data":"4b1130e15ccc7f12e75147dc49d2c2fb79f2c9e33d30bfe202de9962a44201ab"} Oct 30 03:55:51.937703 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:51.937677 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e33271148ad497/enrolled-inference-allowed-18e3327435fc530a" event={"ID":"d0861670-b1c5-4043-b926-66570537e342","Type":"ContainerStarted","Data":"d698025239e3532ca760b99e3dd7cbe255ba335f0702bcc67c4708793163c8ce"} Oct 30 03:55:53.070312 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:53.070287 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e33271148ad497/enrolled-inference-allowed-18e3327435fc530a" Oct 30 03:55:53.700354 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:53.700316 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-namespace-auth-18e33271148ad497/revoked-inference-denied-18e33274be97b80e"] Oct 30 03:55:53.700747 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:53.700731 5882 cpu_manager.go:414] "containerMap: removing container" podUID="d0861670-b1c5-4043-b926-66570537e342" containerName="curl" Oct 30 03:55:53.700812 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:53.700748 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0861670-b1c5-4043-b926-66570537e342" containerName="curl" Oct 30 03:55:53.700852 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:53.700838 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0861670-b1c5-4043-b926-66570537e342" containerName="curl" Oct 30 03:55:53.703665 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:53.703649 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e33271148ad497/revoked-inference-denied-18e33274be97b80e" Oct 30 03:55:53.714247 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:53.714226 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e33271148ad497/revoked-inference-denied-18e33274be97b80e" Oct 30 03:55:53.721346 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:53.721322 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-namespace-auth-18e33271148ad497/revoked-inference-denied-18e33274be97b80e"] Oct 30 03:55:53.943410 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:55:53.943376 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:55:53.956882 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:53.956824 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e33271148ad497/enrolled-inference-allowed-18e3327435fc530a" Oct 30 03:55:53.956984 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:53.956826 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e33271148ad497/enrolled-inference-allowed-18e3327435fc530a" event={"ID":"d0861670-b1c5-4043-b926-66570537e342","Type":"ContainerDied","Data":"d698025239e3532ca760b99e3dd7cbe255ba335f0702bcc67c4708793163c8ce"} Oct 30 03:55:53.956984 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:53.956926 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d698025239e3532ca760b99e3dd7cbe255ba335f0702bcc67c4708793163c8ce" Oct 30 03:55:54.010596 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:55:54.010570 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d83388c_348d_4079_892a_014aba7ad1bc.slice/crio-87c318514f24f0faf0adc945d41b6a05fb9514719105b57e4af73ab2d115597a WatchSource:0}: Error finding container 87c318514f24f0faf0adc945d41b6a05fb9514719105b57e4af73ab2d115597a: Status 404 returned error can't find the container with id 87c318514f24f0faf0adc945d41b6a05fb9514719105b57e4af73ab2d115597a Oct 30 03:55:54.011548 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:54.011524 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-namespace-auth-18e33271148ad497/revoked-inference-denied-18e33274be97b80e"] Oct 30 03:55:54.963157 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:54.963121 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e33271148ad497/revoked-inference-denied-18e33274be97b80e" event={"ID":"7d83388c-348d-4079-892a-014aba7ad1bc","Type":"ContainerStarted","Data":"80ba224fb05e81ad14dab9704df102634d253b29f9b6acbc0a79a296c8687aeb"} Oct 30 03:55:54.963157 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:54.963163 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e33271148ad497/revoked-inference-denied-18e33274be97b80e" event={"ID":"7d83388c-348d-4079-892a-014aba7ad1bc","Type":"ContainerStarted","Data":"87c318514f24f0faf0adc945d41b6a05fb9514719105b57e4af73ab2d115597a"} Oct 30 03:55:55.005766 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:55:55.005716 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="collector-namespace-auth-18e33271148ad497/revoked-inference-denied-18e33274be97b80e" podStartSLOduration=2.005700879 podStartE2EDuration="2.005700879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:55:55.004964518 +0000 UTC m=+4467.119279492" watchObservedRunningTime="2026-10-30 03:55:55.005700879 +0000 UTC m=+4467.120015845" Oct 30 03:56:05.018563 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:05.018529 5882 generic.go:356] "Generic (PLEG): container finished" podID="7d83388c-348d-4079-892a-014aba7ad1bc" containerID="80ba224fb05e81ad14dab9704df102634d253b29f9b6acbc0a79a296c8687aeb" exitCode=0 Oct 30 03:56:05.019003 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:05.018602 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e33271148ad497/revoked-inference-denied-18e33274be97b80e" event={"ID":"7d83388c-348d-4079-892a-014aba7ad1bc","Type":"ContainerDied","Data":"80ba224fb05e81ad14dab9704df102634d253b29f9b6acbc0a79a296c8687aeb"} Oct 30 03:56:06.157126 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:06.157106 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e33271148ad497/revoked-inference-denied-18e33274be97b80e" Oct 30 03:56:07.029980 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:07.029950 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e33271148ad497/revoked-inference-denied-18e33274be97b80e" event={"ID":"7d83388c-348d-4079-892a-014aba7ad1bc","Type":"ContainerDied","Data":"87c318514f24f0faf0adc945d41b6a05fb9514719105b57e4af73ab2d115597a"} Oct 30 03:56:07.029980 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:07.029983 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87c318514f24f0faf0adc945d41b6a05fb9514719105b57e4af73ab2d115597a" Oct 30 03:56:07.030200 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:07.029988 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e33271148ad497/revoked-inference-denied-18e33274be97b80e" Oct 30 03:56:07.850946 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:07.850908 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-namespace-auth-18e33271148ad497/revoked-inference-denied-18e33274be97b80e"] Oct 30 03:56:07.854569 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:07.854545 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-namespace-auth-18e33271148ad497/revoked-inference-denied-18e33274be97b80e"] Oct 30 03:56:07.875160 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:07.875132 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-namespace-auth-18e33271148ad497/enrolled-inference-allowed-18e3327435fc530a"] Oct 30 03:56:07.879232 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:07.879205 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-namespace-auth-18e33271148ad497/enrolled-inference-allowed-18e3327435fc530a"] Oct 30 03:56:07.899634 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:07.899592 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-namespace-auth-18e33271148ad497/unenrolled-spoof-denied-18e332714f8c65e1"] Oct 30 03:56:07.903422 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:07.903402 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-namespace-auth-18e33271148ad497/unenrolled-spoof-denied-18e332714f8c65e1"] Oct 30 03:56:08.009133 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:08.009100 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d83388c-348d-4079-892a-014aba7ad1bc" path="/var/lib/kubelet/pods/7d83388c-348d-4079-892a-014aba7ad1bc/volumes" Oct 30 03:56:08.009364 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:08.009351 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae1f3b5c-20d3-46b1-8564-6587872cd2b7" path="/var/lib/kubelet/pods/ae1f3b5c-20d3-46b1-8564-6587872cd2b7/volumes" Oct 30 03:56:08.009557 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:08.009545 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0861670-b1c5-4043-b926-66570537e342" path="/var/lib/kubelet/pods/d0861670-b1c5-4043-b926-66570537e342/volumes" Oct 30 03:56:10.308801 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:10.308768 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6"] Oct 30 03:56:10.309316 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:10.309128 5882 cpu_manager.go:414] "containerMap: removing container" podUID="7d83388c-348d-4079-892a-014aba7ad1bc" containerName="curl" Oct 30 03:56:10.309316 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:10.309138 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d83388c-348d-4079-892a-014aba7ad1bc" containerName="curl" Oct 30 03:56:10.309316 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:10.309203 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d83388c-348d-4079-892a-014aba7ad1bc" containerName="curl" Oct 30 03:56:10.312092 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:10.312072 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" Oct 30 03:56:10.333290 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:10.333264 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6"] Oct 30 03:56:10.401689 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:10.401637 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jrdd\" (UniqueName: \"kubernetes.io/projected/7dd0218b-734c-48b5-9126-c13bc93a9a64-kube-api-access-2jrdd\") pod \"data-science-collector-targetallocator-5888c49b67-lrbc6\" (UID: \"7dd0218b-734c-48b5-9126-c13bc93a9a64\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" Oct 30 03:56:10.401853 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:10.401790 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7dd0218b-734c-48b5-9126-c13bc93a9a64-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5888c49b67-lrbc6\" (UID: \"7dd0218b-734c-48b5-9126-c13bc93a9a64\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" Oct 30 03:56:10.402030 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:10.402001 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7dd0218b-734c-48b5-9126-c13bc93a9a64-ta-internal\") pod \"data-science-collector-targetallocator-5888c49b67-lrbc6\" (UID: \"7dd0218b-734c-48b5-9126-c13bc93a9a64\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" Oct 30 03:56:10.502785 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:10.502748 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7dd0218b-734c-48b5-9126-c13bc93a9a64-ta-internal\") pod \"data-science-collector-targetallocator-5888c49b67-lrbc6\" (UID: \"7dd0218b-734c-48b5-9126-c13bc93a9a64\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" Oct 30 03:56:10.502947 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:10.502798 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jrdd\" (UniqueName: \"kubernetes.io/projected/7dd0218b-734c-48b5-9126-c13bc93a9a64-kube-api-access-2jrdd\") pod \"data-science-collector-targetallocator-5888c49b67-lrbc6\" (UID: \"7dd0218b-734c-48b5-9126-c13bc93a9a64\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" Oct 30 03:56:10.502947 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:10.502825 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7dd0218b-734c-48b5-9126-c13bc93a9a64-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5888c49b67-lrbc6\" (UID: \"7dd0218b-734c-48b5-9126-c13bc93a9a64\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" Oct 30 03:56:10.503382 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:10.503361 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7dd0218b-734c-48b5-9126-c13bc93a9a64-ta-internal\") pod \"data-science-collector-targetallocator-5888c49b67-lrbc6\" (UID: \"7dd0218b-734c-48b5-9126-c13bc93a9a64\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" Oct 30 03:56:10.505199 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:10.505180 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7dd0218b-734c-48b5-9126-c13bc93a9a64-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5888c49b67-lrbc6\" (UID: \"7dd0218b-734c-48b5-9126-c13bc93a9a64\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" Oct 30 03:56:10.530068 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:10.530043 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jrdd\" (UniqueName: \"kubernetes.io/projected/7dd0218b-734c-48b5-9126-c13bc93a9a64-kube-api-access-2jrdd\") pod \"data-science-collector-targetallocator-5888c49b67-lrbc6\" (UID: \"7dd0218b-734c-48b5-9126-c13bc93a9a64\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" Oct 30 03:56:10.623267 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:10.623189 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" Oct 30 03:56:10.915028 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:10.914999 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6"] Oct 30 03:56:10.915928 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:56:10.915903 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7dd0218b_734c_48b5_9126_c13bc93a9a64.slice/crio-2b516b4134088d8d315a6f3d845cc2cd769110eb2d3989bd9bfa1972bfe43321 WatchSource:0}: Error finding container 2b516b4134088d8d315a6f3d845cc2cd769110eb2d3989bd9bfa1972bfe43321: Status 404 returned error can't find the container with id 2b516b4134088d8d315a6f3d845cc2cd769110eb2d3989bd9bfa1972bfe43321 Oct 30 03:56:10.918525 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:10.918496 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:56:10.918639 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:10.918547 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:56:11.052506 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:11.052470 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" event={"ID":"7dd0218b-734c-48b5-9126-c13bc93a9a64","Type":"ContainerStarted","Data":"8be4c1d12fc5823c201ce9d5c9dcb9b18f6c77a4a4ec00eaee5aa4a4558a8927"} Oct 30 03:56:11.052506 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:11.052514 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" event={"ID":"7dd0218b-734c-48b5-9126-c13bc93a9a64","Type":"ContainerStarted","Data":"2b516b4134088d8d315a6f3d845cc2cd769110eb2d3989bd9bfa1972bfe43321"} Oct 30 03:56:11.052776 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:11.052637 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" Oct 30 03:56:11.053657 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:11.053633 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" podUID="7dd0218b-734c-48b5-9126-c13bc93a9a64" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.101:8080/readyz\": dial tcp 10.134.0.101:8080: connect: connection refused" Oct 30 03:56:11.111640 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:11.111580 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" podStartSLOduration=1.111568077 podStartE2EDuration="1.111568077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:11.109369792 +0000 UTC m=+4483.223684768" watchObservedRunningTime="2026-10-30 03:56:11.111568077 +0000 UTC m=+4483.225883122" Oct 30 03:56:12.028109 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.028076 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp"] Oct 30 03:56:12.028506 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.028263 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp" podUID="b49d0601-a592-4f95-af65-26fbda077c59" containerName="thanos-querier" containerID="cri-o://1207645ce3ce1aeceb76a12b9cee9ab0296676525e4bb7109aaa63c8699295f7" gracePeriod=30 Oct 30 03:56:12.059428 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.059398 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" Oct 30 03:56:12.188526 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.188454 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd"] Oct 30 03:56:12.189190 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.189131 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" podUID="5d11b295-8a7a-423c-812f-c27f91049256" containerName="ta-container" containerID="cri-o://1ecd7b076c2d022ff7ec77315114c81e71916dff67933c1ded218a3cc0aacb5e" gracePeriod=30 Oct 30 03:56:12.228685 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.228655 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 03:56:12.228890 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.228870 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" containerName="alertmanager" containerID="cri-o://f9898b1c1ab8fa1f0702685e306c91131cd81332df2dfbbb8a0b0daec7f3a6fb" gracePeriod=120 Oct 30 03:56:12.228970 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.228946 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" containerName="config-reloader" containerID="cri-o://59489c0d247cd6c31a81ba7c637048a8553ff57cf1595fce97cec2d168ab6992" gracePeriod=120 Oct 30 03:56:12.309392 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.309368 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp" Oct 30 03:56:12.419466 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.419429 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49d0601-a592-4f95-af65-26fbda077c59-kube-api-access-lrs9p\" (UniqueName: \"kubernetes.io/projected/b49d0601-a592-4f95-af65-26fbda077c59-kube-api-access-lrs9p\") pod \"b49d0601-a592-4f95-af65-26fbda077c59\" (UID: \"b49d0601-a592-4f95-af65-26fbda077c59\") " Oct 30 03:56:12.421508 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.421483 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49d0601-a592-4f95-af65-26fbda077c59-kube-api-access-lrs9p" pod "b49d0601-a592-4f95-af65-26fbda077c59" (UID: "b49d0601-a592-4f95-af65-26fbda077c59"). InnerVolumeSpecName "kube-api-access-lrs9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:12.442585 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.442565 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" Oct 30 03:56:12.520138 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.520111 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrs9p\" (UniqueName: \"kubernetes.io/projected/b49d0601-a592-4f95-af65-26fbda077c59-kube-api-access-lrs9p\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.621227 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.621156 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d11b295-8a7a-423c-812f-c27f91049256-ta-internal\" (UniqueName: \"kubernetes.io/configmap/5d11b295-8a7a-423c-812f-c27f91049256-ta-internal\") pod \"5d11b295-8a7a-423c-812f-c27f91049256\" (UID: \"5d11b295-8a7a-423c-812f-c27f91049256\") " Oct 30 03:56:12.621227 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.621198 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d11b295-8a7a-423c-812f-c27f91049256-kube-api-access-hp8sx\" (UniqueName: \"kubernetes.io/projected/5d11b295-8a7a-423c-812f-c27f91049256-kube-api-access-hp8sx\") pod \"5d11b295-8a7a-423c-812f-c27f91049256\" (UID: \"5d11b295-8a7a-423c-812f-c27f91049256\") " Oct 30 03:56:12.621454 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.621239 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d11b295-8a7a-423c-812f-c27f91049256-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/5d11b295-8a7a-423c-812f-c27f91049256-data-science-collector-ta-server-cert\") pod \"5d11b295-8a7a-423c-812f-c27f91049256\" (UID: \"5d11b295-8a7a-423c-812f-c27f91049256\") " Oct 30 03:56:12.621536 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.621512 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d11b295-8a7a-423c-812f-c27f91049256-ta-internal" pod "5d11b295-8a7a-423c-812f-c27f91049256" (UID: "5d11b295-8a7a-423c-812f-c27f91049256"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:12.623692 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.623661 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d11b295-8a7a-423c-812f-c27f91049256-kube-api-access-hp8sx" pod "5d11b295-8a7a-423c-812f-c27f91049256" (UID: "5d11b295-8a7a-423c-812f-c27f91049256"). InnerVolumeSpecName "kube-api-access-hp8sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:12.623902 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.623880 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d11b295-8a7a-423c-812f-c27f91049256-data-science-collector-ta-server-cert" pod "5d11b295-8a7a-423c-812f-c27f91049256" (UID: "5d11b295-8a7a-423c-812f-c27f91049256"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:12.722272 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.722224 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/5d11b295-8a7a-423c-812f-c27f91049256-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.722272 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.722265 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/5d11b295-8a7a-423c-812f-c27f91049256-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.722272 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.722276 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp8sx\" (UniqueName: \"kubernetes.io/projected/5d11b295-8a7a-423c-812f-c27f91049256-kube-api-access-hp8sx\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.989637 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:12.989542 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6"] Oct 30 03:56:13.029777 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.029746 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:56:13.030218 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.029917 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="ad51ce29-2421-46e3-97cf-e5e2fbadf67d" containerName="perses" containerID="cri-o://2af460072686db74224dea9496d6d819f4773ac16265751b412b9e0d8305c0d7" gracePeriod=30 Oct 30 03:56:13.065963 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.065934 5882 generic.go:356] "Generic (PLEG): container finished" podID="3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" containerID="59489c0d247cd6c31a81ba7c637048a8553ff57cf1595fce97cec2d168ab6992" exitCode=0 Oct 30 03:56:13.066095 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.066013 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7","Type":"ContainerDied","Data":"59489c0d247cd6c31a81ba7c637048a8553ff57cf1595fce97cec2d168ab6992"} Oct 30 03:56:13.067259 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.067235 5882 generic.go:356] "Generic (PLEG): container finished" podID="5d11b295-8a7a-423c-812f-c27f91049256" containerID="1ecd7b076c2d022ff7ec77315114c81e71916dff67933c1ded218a3cc0aacb5e" exitCode=0 Oct 30 03:56:13.067364 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.067278 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" Oct 30 03:56:13.067364 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.067312 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" event={"ID":"5d11b295-8a7a-423c-812f-c27f91049256","Type":"ContainerDied","Data":"1ecd7b076c2d022ff7ec77315114c81e71916dff67933c1ded218a3cc0aacb5e"} Oct 30 03:56:13.067364 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.067346 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd" event={"ID":"5d11b295-8a7a-423c-812f-c27f91049256","Type":"ContainerDied","Data":"48bfd1ab7cc11049002d27a23b9770b79ef942e5801b4ab2a97afbc4fc6e3a83"} Oct 30 03:56:13.067488 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.067363 5882 scope.go:122] "RemoveContainer" containerID="1ecd7b076c2d022ff7ec77315114c81e71916dff67933c1ded218a3cc0aacb5e" Oct 30 03:56:13.068756 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.068734 5882 generic.go:356] "Generic (PLEG): container finished" podID="b49d0601-a592-4f95-af65-26fbda077c59" containerID="1207645ce3ce1aeceb76a12b9cee9ab0296676525e4bb7109aaa63c8699295f7" exitCode=0 Oct 30 03:56:13.068838 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.068769 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp" Oct 30 03:56:13.068838 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.068766 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp" event={"ID":"b49d0601-a592-4f95-af65-26fbda077c59","Type":"ContainerDied","Data":"1207645ce3ce1aeceb76a12b9cee9ab0296676525e4bb7109aaa63c8699295f7"} Oct 30 03:56:13.068916 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.068869 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp" event={"ID":"b49d0601-a592-4f95-af65-26fbda077c59","Type":"ContainerDied","Data":"d2754e4a009fd062961fc05674afc1cce5cf5bfe15a7af768535ef55341d72ea"} Oct 30 03:56:13.141705 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.141682 5882 scope.go:122] "RemoveContainer" containerID="1ecd7b076c2d022ff7ec77315114c81e71916dff67933c1ded218a3cc0aacb5e" Oct 30 03:56:13.141971 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:56:13.141948 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ecd7b076c2d022ff7ec77315114c81e71916dff67933c1ded218a3cc0aacb5e\": container with ID starting with 1ecd7b076c2d022ff7ec77315114c81e71916dff67933c1ded218a3cc0aacb5e not found: ID does not exist" containerID="1ecd7b076c2d022ff7ec77315114c81e71916dff67933c1ded218a3cc0aacb5e" Oct 30 03:56:13.142096 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.141979 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ecd7b076c2d022ff7ec77315114c81e71916dff67933c1ded218a3cc0aacb5e"} err="failed to get container status \"1ecd7b076c2d022ff7ec77315114c81e71916dff67933c1ded218a3cc0aacb5e\": rpc error: code = NotFound desc = could not find container \"1ecd7b076c2d022ff7ec77315114c81e71916dff67933c1ded218a3cc0aacb5e\": container with ID starting with 1ecd7b076c2d022ff7ec77315114c81e71916dff67933c1ded218a3cc0aacb5e not found: ID does not exist" Oct 30 03:56:13.142096 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.142001 5882 scope.go:122] "RemoveContainer" containerID="1207645ce3ce1aeceb76a12b9cee9ab0296676525e4bb7109aaa63c8699295f7" Oct 30 03:56:13.146069 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.146049 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd"] Oct 30 03:56:13.151140 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.151120 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-657cd485df-7lgtd"] Oct 30 03:56:13.151204 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.151181 5882 scope.go:122] "RemoveContainer" containerID="1207645ce3ce1aeceb76a12b9cee9ab0296676525e4bb7109aaa63c8699295f7" Oct 30 03:56:13.151493 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:56:13.151475 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1207645ce3ce1aeceb76a12b9cee9ab0296676525e4bb7109aaa63c8699295f7\": container with ID starting with 1207645ce3ce1aeceb76a12b9cee9ab0296676525e4bb7109aaa63c8699295f7 not found: ID does not exist" containerID="1207645ce3ce1aeceb76a12b9cee9ab0296676525e4bb7109aaa63c8699295f7" Oct 30 03:56:13.151534 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.151499 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1207645ce3ce1aeceb76a12b9cee9ab0296676525e4bb7109aaa63c8699295f7"} err="failed to get container status \"1207645ce3ce1aeceb76a12b9cee9ab0296676525e4bb7109aaa63c8699295f7\": rpc error: code = NotFound desc = could not find container \"1207645ce3ce1aeceb76a12b9cee9ab0296676525e4bb7109aaa63c8699295f7\": container with ID starting with 1207645ce3ce1aeceb76a12b9cee9ab0296676525e4bb7109aaa63c8699295f7 not found: ID does not exist" Oct 30 03:56:13.188204 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.188164 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp"] Oct 30 03:56:13.192440 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.192411 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bhvkp"] Oct 30 03:56:13.226407 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:56:13.226373 5882 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-targetallocator: configmap "data-science-collector-targetallocator" not found Oct 30 03:56:13.226562 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:56:13.226463 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dd0218b-734c-48b5-9126-c13bc93a9a64-ta-internal podName:7dd0218b-734c-48b5-9126-c13bc93a9a64 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:13.726439671 +0000 UTC m=+4485.840754638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ta-internal" (UniqueName: "kubernetes.io/configmap/7dd0218b-734c-48b5-9126-c13bc93a9a64-ta-internal") pod "data-science-collector-targetallocator-5888c49b67-lrbc6" (UID: "7dd0218b-734c-48b5-9126-c13bc93a9a64") : configmap "data-science-collector-targetallocator" not found Oct 30 03:56:13.278900 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.278868 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:13.428296 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.428259 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-config\" (UniqueName: \"kubernetes.io/configmap/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-config\") pod \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\" (UID: \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\") " Oct 30 03:56:13.428461 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.428329 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-plugins\" (UniqueName: \"kubernetes.io/empty-dir/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-plugins\") pod \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\" (UID: \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\") " Oct 30 03:56:13.428461 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.428357 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-kube-api-access-b7scg\" (UniqueName: \"kubernetes.io/projected/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-kube-api-access-b7scg\") pod \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\" (UID: \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\") " Oct 30 03:56:13.428540 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.428476 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\" (UID: \"ad51ce29-2421-46e3-97cf-e5e2fbadf67d\") " Oct 30 03:56:13.428661 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.428637 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-config" pod "ad51ce29-2421-46e3-97cf-e5e2fbadf67d" (UID: "ad51ce29-2421-46e3-97cf-e5e2fbadf67d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:13.428768 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.428751 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:56:13.430656 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.430596 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-kube-api-access-b7scg" pod "ad51ce29-2421-46e3-97cf-e5e2fbadf67d" (UID: "ad51ce29-2421-46e3-97cf-e5e2fbadf67d"). InnerVolumeSpecName "kube-api-access-b7scg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:13.431103 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.431080 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e" pod "ad51ce29-2421-46e3-97cf-e5e2fbadf67d" (UID: "ad51ce29-2421-46e3-97cf-e5e2fbadf67d"). InnerVolumeSpecName "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 03:56:13.530039 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.530008 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7scg\" (UniqueName: \"kubernetes.io/projected/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-kube-api-access-b7scg\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:56:13.530172 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.530051 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 03:56:13.543394 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.543373 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e") on node "ip-10-0-141-200.ec2.internal" Oct 30 03:56:13.576221 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.576192 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-plugins" pod "ad51ce29-2421-46e3-97cf-e5e2fbadf67d" (UID: "ad51ce29-2421-46e3-97cf-e5e2fbadf67d"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:13.631398 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.631360 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" Oct 30 03:56:13.631398 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:13.631393 5882 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/ad51ce29-2421-46e3-97cf-e5e2fbadf67d-plugins\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:56:13.732463 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:56:13.732429 5882 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-targetallocator: configmap "data-science-collector-targetallocator" not found Oct 30 03:56:13.732641 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:56:13.732497 5882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dd0218b-734c-48b5-9126-c13bc93a9a64-ta-internal podName:7dd0218b-734c-48b5-9126-c13bc93a9a64 nodeName:}" failed. No retries permitted until 2026-10-30 03:56:14.732483429 +0000 UTC m=+4486.846798395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "ta-internal" (UniqueName: "kubernetes.io/configmap/7dd0218b-734c-48b5-9126-c13bc93a9a64-ta-internal") pod "data-science-collector-targetallocator-5888c49b67-lrbc6" (UID: "7dd0218b-734c-48b5-9126-c13bc93a9a64") : configmap "data-science-collector-targetallocator" not found Oct 30 03:56:14.011566 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.011535 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d11b295-8a7a-423c-812f-c27f91049256" path="/var/lib/kubelet/pods/5d11b295-8a7a-423c-812f-c27f91049256/volumes" Oct 30 03:56:14.011962 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.011944 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49d0601-a592-4f95-af65-26fbda077c59" path="/var/lib/kubelet/pods/b49d0601-a592-4f95-af65-26fbda077c59/volumes" Oct 30 03:56:14.076252 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.076224 5882 generic.go:356] "Generic (PLEG): container finished" podID="ad51ce29-2421-46e3-97cf-e5e2fbadf67d" containerID="2af460072686db74224dea9496d6d819f4773ac16265751b412b9e0d8305c0d7" exitCode=0 Oct 30 03:56:14.076709 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.076277 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:14.076709 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.076305 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"ad51ce29-2421-46e3-97cf-e5e2fbadf67d","Type":"ContainerDied","Data":"2af460072686db74224dea9496d6d819f4773ac16265751b412b9e0d8305c0d7"} Oct 30 03:56:14.076709 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.076351 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"ad51ce29-2421-46e3-97cf-e5e2fbadf67d","Type":"ContainerDied","Data":"2ba7bd2380c658436eeef5f656cbb54097d47e1757b11840aad010117905ff0f"} Oct 30 03:56:14.076709 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.076373 5882 scope.go:122] "RemoveContainer" containerID="2af460072686db74224dea9496d6d819f4773ac16265751b412b9e0d8305c0d7" Oct 30 03:56:14.077641 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.077519 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" podUID="7dd0218b-734c-48b5-9126-c13bc93a9a64" containerName="ta-container" containerID="cri-o://8be4c1d12fc5823c201ce9d5c9dcb9b18f6c77a4a4ec00eaee5aa4a4558a8927" gracePeriod=30 Oct 30 03:56:14.086376 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.086353 5882 scope.go:122] "RemoveContainer" containerID="2af460072686db74224dea9496d6d819f4773ac16265751b412b9e0d8305c0d7" Oct 30 03:56:14.086697 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:56:14.086676 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2af460072686db74224dea9496d6d819f4773ac16265751b412b9e0d8305c0d7\": container with ID starting with 2af460072686db74224dea9496d6d819f4773ac16265751b412b9e0d8305c0d7 not found: ID does not exist" containerID="2af460072686db74224dea9496d6d819f4773ac16265751b412b9e0d8305c0d7" Oct 30 03:56:14.086752 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.086707 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2af460072686db74224dea9496d6d819f4773ac16265751b412b9e0d8305c0d7"} err="failed to get container status \"2af460072686db74224dea9496d6d819f4773ac16265751b412b9e0d8305c0d7\": rpc error: code = NotFound desc = could not find container \"2af460072686db74224dea9496d6d819f4773ac16265751b412b9e0d8305c0d7\": container with ID starting with 2af460072686db74224dea9496d6d819f4773ac16265751b412b9e0d8305c0d7 not found: ID does not exist" Oct 30 03:56:14.131807 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.131781 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:56:14.133993 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.133974 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:56:14.317624 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.317588 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" Oct 30 03:56:14.438401 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.438368 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dd0218b-734c-48b5-9126-c13bc93a9a64-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7dd0218b-734c-48b5-9126-c13bc93a9a64-data-science-collector-ta-server-cert\") pod \"7dd0218b-734c-48b5-9126-c13bc93a9a64\" (UID: \"7dd0218b-734c-48b5-9126-c13bc93a9a64\") " Oct 30 03:56:14.438594 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.438453 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dd0218b-734c-48b5-9126-c13bc93a9a64-ta-internal\" (UniqueName: \"kubernetes.io/configmap/7dd0218b-734c-48b5-9126-c13bc93a9a64-ta-internal\") pod \"7dd0218b-734c-48b5-9126-c13bc93a9a64\" (UID: \"7dd0218b-734c-48b5-9126-c13bc93a9a64\") " Oct 30 03:56:14.438594 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.438547 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dd0218b-734c-48b5-9126-c13bc93a9a64-kube-api-access-2jrdd\" (UniqueName: \"kubernetes.io/projected/7dd0218b-734c-48b5-9126-c13bc93a9a64-kube-api-access-2jrdd\") pod \"7dd0218b-734c-48b5-9126-c13bc93a9a64\" (UID: \"7dd0218b-734c-48b5-9126-c13bc93a9a64\") " Oct 30 03:56:14.438838 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.438815 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd0218b-734c-48b5-9126-c13bc93a9a64-ta-internal" pod "7dd0218b-734c-48b5-9126-c13bc93a9a64" (UID: "7dd0218b-734c-48b5-9126-c13bc93a9a64"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:14.440503 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.440482 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd0218b-734c-48b5-9126-c13bc93a9a64-data-science-collector-ta-server-cert" pod "7dd0218b-734c-48b5-9126-c13bc93a9a64" (UID: "7dd0218b-734c-48b5-9126-c13bc93a9a64"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:14.440558 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.440522 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd0218b-734c-48b5-9126-c13bc93a9a64-kube-api-access-2jrdd" pod "7dd0218b-734c-48b5-9126-c13bc93a9a64" (UID: "7dd0218b-734c-48b5-9126-c13bc93a9a64"). InnerVolumeSpecName "kube-api-access-2jrdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:14.539170 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.539142 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7dd0218b-734c-48b5-9126-c13bc93a9a64-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:56:14.539170 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.539168 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7dd0218b-734c-48b5-9126-c13bc93a9a64-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:56:14.539312 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:14.539178 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jrdd\" (UniqueName: \"kubernetes.io/projected/7dd0218b-734c-48b5-9126-c13bc93a9a64-kube-api-access-2jrdd\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:56:15.085183 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:15.085143 5882 generic.go:356] "Generic (PLEG): container finished" podID="7dd0218b-734c-48b5-9126-c13bc93a9a64" containerID="8be4c1d12fc5823c201ce9d5c9dcb9b18f6c77a4a4ec00eaee5aa4a4558a8927" exitCode=0 Oct 30 03:56:15.085597 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:15.085189 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" Oct 30 03:56:15.085597 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:15.085211 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" event={"ID":"7dd0218b-734c-48b5-9126-c13bc93a9a64","Type":"ContainerDied","Data":"8be4c1d12fc5823c201ce9d5c9dcb9b18f6c77a4a4ec00eaee5aa4a4558a8927"} Oct 30 03:56:15.085597 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:15.085248 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6" event={"ID":"7dd0218b-734c-48b5-9126-c13bc93a9a64","Type":"ContainerDied","Data":"2b516b4134088d8d315a6f3d845cc2cd769110eb2d3989bd9bfa1972bfe43321"} Oct 30 03:56:15.085597 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:15.085270 5882 scope.go:122] "RemoveContainer" containerID="8be4c1d12fc5823c201ce9d5c9dcb9b18f6c77a4a4ec00eaee5aa4a4558a8927" Oct 30 03:56:15.097988 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:15.097968 5882 scope.go:122] "RemoveContainer" containerID="8be4c1d12fc5823c201ce9d5c9dcb9b18f6c77a4a4ec00eaee5aa4a4558a8927" Oct 30 03:56:15.098416 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:56:15.098391 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be4c1d12fc5823c201ce9d5c9dcb9b18f6c77a4a4ec00eaee5aa4a4558a8927\": container with ID starting with 8be4c1d12fc5823c201ce9d5c9dcb9b18f6c77a4a4ec00eaee5aa4a4558a8927 not found: ID does not exist" containerID="8be4c1d12fc5823c201ce9d5c9dcb9b18f6c77a4a4ec00eaee5aa4a4558a8927" Oct 30 03:56:15.098486 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:15.098425 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be4c1d12fc5823c201ce9d5c9dcb9b18f6c77a4a4ec00eaee5aa4a4558a8927"} err="failed to get container status \"8be4c1d12fc5823c201ce9d5c9dcb9b18f6c77a4a4ec00eaee5aa4a4558a8927\": rpc error: code = NotFound desc = could not find container \"8be4c1d12fc5823c201ce9d5c9dcb9b18f6c77a4a4ec00eaee5aa4a4558a8927\": container with ID starting with 8be4c1d12fc5823c201ce9d5c9dcb9b18f6c77a4a4ec00eaee5aa4a4558a8927 not found: ID does not exist" Oct 30 03:56:15.140389 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:15.140359 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6"] Oct 30 03:56:15.154544 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:15.154516 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5888c49b67-lrbc6"] Oct 30 03:56:15.859202 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:15.859162 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.42:9093/-/ready\": dial tcp 10.134.0.42:9093: connect: connection refused" Oct 30 03:56:16.009344 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:16.009308 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dd0218b-734c-48b5-9126-c13bc93a9a64" path="/var/lib/kubelet/pods/7dd0218b-734c-48b5-9126-c13bc93a9a64/volumes" Oct 30 03:56:16.009847 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:16.009825 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad51ce29-2421-46e3-97cf-e5e2fbadf67d" path="/var/lib/kubelet/pods/ad51ce29-2421-46e3-97cf-e5e2fbadf67d/volumes" Oct 30 03:56:18.938782 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:18.938749 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:56:18.939611 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:18.939585 5882 cpu_manager.go:414] "containerMap: removing container" podUID="b49d0601-a592-4f95-af65-26fbda077c59" containerName="thanos-querier" Oct 30 03:56:18.939611 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:18.939608 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49d0601-a592-4f95-af65-26fbda077c59" containerName="thanos-querier" Oct 30 03:56:18.939796 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:18.939638 5882 cpu_manager.go:414] "containerMap: removing container" podUID="5d11b295-8a7a-423c-812f-c27f91049256" containerName="ta-container" Oct 30 03:56:18.939796 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:18.939644 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d11b295-8a7a-423c-812f-c27f91049256" containerName="ta-container" Oct 30 03:56:18.939796 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:18.939652 5882 cpu_manager.go:414] "containerMap: removing container" podUID="7dd0218b-734c-48b5-9126-c13bc93a9a64" containerName="ta-container" Oct 30 03:56:18.939796 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:18.939657 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd0218b-734c-48b5-9126-c13bc93a9a64" containerName="ta-container" Oct 30 03:56:18.939796 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:18.939667 5882 cpu_manager.go:414] "containerMap: removing container" podUID="ad51ce29-2421-46e3-97cf-e5e2fbadf67d" containerName="perses" Oct 30 03:56:18.939796 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:18.939671 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad51ce29-2421-46e3-97cf-e5e2fbadf67d" containerName="perses" Oct 30 03:56:18.939796 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:18.939753 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49d0601-a592-4f95-af65-26fbda077c59" containerName="thanos-querier" Oct 30 03:56:18.939796 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:18.939765 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d11b295-8a7a-423c-812f-c27f91049256" containerName="ta-container" Oct 30 03:56:18.939796 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:18.939774 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd0218b-734c-48b5-9126-c13bc93a9a64" containerName="ta-container" Oct 30 03:56:18.939796 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:18.939783 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad51ce29-2421-46e3-97cf-e5e2fbadf67d" containerName="perses" Oct 30 03:56:18.944463 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:18.944445 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:18.959097 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:18.959077 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:56:19.010111 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:19.010075 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 30 03:56:19.010245 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:19.010150 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 03:56:19.078323 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:19.078286 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6634a735-5012-4a14-92db-7ef215b34cb0-config\") pod \"data-science-perses-0\" (UID: \"6634a735-5012-4a14-92db-7ef215b34cb0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:19.078464 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:19.078375 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fcx4\" (UniqueName: \"kubernetes.io/projected/6634a735-5012-4a14-92db-7ef215b34cb0-kube-api-access-4fcx4\") pod \"data-science-perses-0\" (UID: \"6634a735-5012-4a14-92db-7ef215b34cb0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:19.078464 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:19.078406 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/6634a735-5012-4a14-92db-7ef215b34cb0-plugins\") pod \"data-science-perses-0\" (UID: \"6634a735-5012-4a14-92db-7ef215b34cb0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:19.179513 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:19.179487 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fcx4\" (UniqueName: \"kubernetes.io/projected/6634a735-5012-4a14-92db-7ef215b34cb0-kube-api-access-4fcx4\") pod \"data-science-perses-0\" (UID: \"6634a735-5012-4a14-92db-7ef215b34cb0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:19.179690 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:19.179530 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/6634a735-5012-4a14-92db-7ef215b34cb0-plugins\") pod \"data-science-perses-0\" (UID: \"6634a735-5012-4a14-92db-7ef215b34cb0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:19.179690 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:19.179583 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6634a735-5012-4a14-92db-7ef215b34cb0-config\") pod \"data-science-perses-0\" (UID: \"6634a735-5012-4a14-92db-7ef215b34cb0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:19.179935 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:19.179914 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/6634a735-5012-4a14-92db-7ef215b34cb0-plugins\") pod \"data-science-perses-0\" (UID: \"6634a735-5012-4a14-92db-7ef215b34cb0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:19.180225 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:19.180207 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6634a735-5012-4a14-92db-7ef215b34cb0-config\") pod \"data-science-perses-0\" (UID: \"6634a735-5012-4a14-92db-7ef215b34cb0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:19.206847 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:19.206789 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fcx4\" (UniqueName: \"kubernetes.io/projected/6634a735-5012-4a14-92db-7ef215b34cb0-kube-api-access-4fcx4\") pod \"data-science-perses-0\" (UID: \"6634a735-5012-4a14-92db-7ef215b34cb0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:20.859715 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:20.859672 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.42:9093/-/ready\": dial tcp 10.134.0.42:9093: connect: connection refused" Oct 30 03:56:21.397644 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:21.397596 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"6634a735-5012-4a14-92db-7ef215b34cb0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:21.444450 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:21.444414 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"6634a735-5012-4a14-92db-7ef215b34cb0\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:21.498856 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:21.498826 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"6634a735-5012-4a14-92db-7ef215b34cb0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:21.499017 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:21.498935 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"6634a735-5012-4a14-92db-7ef215b34cb0\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:21.515118 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:21.515095 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"6634a735-5012-4a14-92db-7ef215b34cb0\") DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:21.619781 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:21.619751 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"6634a735-5012-4a14-92db-7ef215b34cb0\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2c422b71c1e0b8fdf34659f2cef7846e06b4353ed88027dc99ade5d7fc29e9d6/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:21.651209 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:21.651151 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"6634a735-5012-4a14-92db-7ef215b34cb0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:21.964275 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:21.964198 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:22.259239 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.259205 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:56:22.312645 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:56:22.312592 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6634a735_5012_4a14_92db_7ef215b34cb0.slice/crio-cdd0f2440ff633ee280ec437722c0fbfda0fdfbe3cf48b80b87ca09253b8acea WatchSource:0}: Error finding container cdd0f2440ff633ee280ec437722c0fbfda0fdfbe3cf48b80b87ca09253b8acea: Status 404 returned error can't find the container with id cdd0f2440ff633ee280ec437722c0fbfda0fdfbe3cf48b80b87ca09253b8acea Oct 30 03:56:22.485561 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.485532 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:56:22.607948 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.607905 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-kube-api-access-m7fl6\" (UniqueName: \"kubernetes.io/projected/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-kube-api-access-m7fl6\") pod \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " Oct 30 03:56:22.608085 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.607958 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-web-config\" (UniqueName: \"kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-web-config\") pod \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " Oct 30 03:56:22.608085 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.608038 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-tls-assets\" (UniqueName: \"kubernetes.io/projected/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-tls-assets\") pod \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " Oct 30 03:56:22.608211 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.608084 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-config-volume\" (UniqueName: \"kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-config-volume\") pod \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " Oct 30 03:56:22.608211 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.608145 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-alertmanager-data-science-monitoringstack-db\") pod \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " Oct 30 03:56:22.608667 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.608638 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-alertmanager-data-science-monitoringstack-db" pod "3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" (UID: "3a64a0a4-a619-46cc-aa53-c3b5f46b87f7"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:22.608755 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.608701 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-config-out\") pod \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " Oct 30 03:56:22.608820 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.608759 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-cluster-tls-config\") pod \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\" (UID: \"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7\") " Oct 30 03:56:22.609070 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.609049 5882 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:56:22.610548 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.610521 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-kube-api-access-m7fl6" pod "3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" (UID: "3a64a0a4-a619-46cc-aa53-c3b5f46b87f7"). InnerVolumeSpecName "kube-api-access-m7fl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:22.610837 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.610804 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-tls-assets" pod "3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" (UID: "3a64a0a4-a619-46cc-aa53-c3b5f46b87f7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:22.611069 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.611045 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-config-out" pod "3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" (UID: "3a64a0a4-a619-46cc-aa53-c3b5f46b87f7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:22.611827 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.611803 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-config-volume" pod "3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" (UID: "3a64a0a4-a619-46cc-aa53-c3b5f46b87f7"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:22.617340 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.617312 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-cluster-tls-config" pod "3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" (UID: "3a64a0a4-a619-46cc-aa53-c3b5f46b87f7"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:22.623277 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.623188 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-web-config" pod "3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" (UID: "3a64a0a4-a619-46cc-aa53-c3b5f46b87f7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:22.710124 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.710088 5882 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-tls-assets\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:56:22.710124 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.710120 5882 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-config-volume\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:56:22.710124 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.710128 5882 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-config-out\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:56:22.710344 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.710137 5882 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-cluster-tls-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:56:22.710344 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.710146 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7fl6\" (UniqueName: \"kubernetes.io/projected/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-kube-api-access-m7fl6\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:56:22.710344 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:22.710155 5882 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7-web-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:56:23.129252 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:23.129219 5882 generic.go:356] "Generic (PLEG): container finished" podID="3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" containerID="f9898b1c1ab8fa1f0702685e306c91131cd81332df2dfbbb8a0b0daec7f3a6fb" exitCode=0 Oct 30 03:56:23.129759 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:23.129289 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 03:56:23.129759 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:23.129303 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7","Type":"ContainerDied","Data":"f9898b1c1ab8fa1f0702685e306c91131cd81332df2dfbbb8a0b0daec7f3a6fb"} Oct 30 03:56:23.129759 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:23.129383 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3a64a0a4-a619-46cc-aa53-c3b5f46b87f7","Type":"ContainerDied","Data":"e9aa5bf41d32f4ed194a6d278430d227404676efe4734e37241f0e8ab2996cea"} Oct 30 03:56:23.129759 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:23.129418 5882 scope.go:122] "RemoveContainer" containerID="59489c0d247cd6c31a81ba7c637048a8553ff57cf1595fce97cec2d168ab6992" Oct 30 03:56:23.133775 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:23.133749 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"6634a735-5012-4a14-92db-7ef215b34cb0","Type":"ContainerStarted","Data":"442ca00fb24f479871d66b32a68251627911b44555fda4e6de856c7d8217a459"} Oct 30 03:56:23.133910 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:23.133794 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:23.133910 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:23.133807 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"6634a735-5012-4a14-92db-7ef215b34cb0","Type":"ContainerStarted","Data":"cdd0f2440ff633ee280ec437722c0fbfda0fdfbe3cf48b80b87ca09253b8acea"} Oct 30 03:56:23.139630 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:23.139603 5882 scope.go:122] "RemoveContainer" containerID="f9898b1c1ab8fa1f0702685e306c91131cd81332df2dfbbb8a0b0daec7f3a6fb" Oct 30 03:56:23.148159 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:23.148144 5882 scope.go:122] "RemoveContainer" containerID="9ceaf71601ca5b306f02aa9d00a2513184be1c350be1672210a3ce2f685c0b2c" Oct 30 03:56:23.156864 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:23.156843 5882 scope.go:122] "RemoveContainer" containerID="59489c0d247cd6c31a81ba7c637048a8553ff57cf1595fce97cec2d168ab6992" Oct 30 03:56:23.157131 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:56:23.157113 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59489c0d247cd6c31a81ba7c637048a8553ff57cf1595fce97cec2d168ab6992\": container with ID starting with 59489c0d247cd6c31a81ba7c637048a8553ff57cf1595fce97cec2d168ab6992 not found: ID does not exist" containerID="59489c0d247cd6c31a81ba7c637048a8553ff57cf1595fce97cec2d168ab6992" Oct 30 03:56:23.157176 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:23.157140 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59489c0d247cd6c31a81ba7c637048a8553ff57cf1595fce97cec2d168ab6992"} err="failed to get container status \"59489c0d247cd6c31a81ba7c637048a8553ff57cf1595fce97cec2d168ab6992\": rpc error: code = NotFound desc = could not find container \"59489c0d247cd6c31a81ba7c637048a8553ff57cf1595fce97cec2d168ab6992\": container with ID starting with 59489c0d247cd6c31a81ba7c637048a8553ff57cf1595fce97cec2d168ab6992 not found: ID does not exist" Oct 30 03:56:23.157176 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:23.157158 5882 scope.go:122] "RemoveContainer" containerID="f9898b1c1ab8fa1f0702685e306c91131cd81332df2dfbbb8a0b0daec7f3a6fb" Oct 30 03:56:23.157375 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:56:23.157356 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9898b1c1ab8fa1f0702685e306c91131cd81332df2dfbbb8a0b0daec7f3a6fb\": container with ID starting with f9898b1c1ab8fa1f0702685e306c91131cd81332df2dfbbb8a0b0daec7f3a6fb not found: ID does not exist" containerID="f9898b1c1ab8fa1f0702685e306c91131cd81332df2dfbbb8a0b0daec7f3a6fb" Oct 30 03:56:23.157441 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:23.157385 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9898b1c1ab8fa1f0702685e306c91131cd81332df2dfbbb8a0b0daec7f3a6fb"} err="failed to get container status \"f9898b1c1ab8fa1f0702685e306c91131cd81332df2dfbbb8a0b0daec7f3a6fb\": rpc error: code = NotFound desc = could not find container \"f9898b1c1ab8fa1f0702685e306c91131cd81332df2dfbbb8a0b0daec7f3a6fb\": container with ID starting with f9898b1c1ab8fa1f0702685e306c91131cd81332df2dfbbb8a0b0daec7f3a6fb not found: ID does not exist" Oct 30 03:56:23.157441 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:23.157406 5882 scope.go:122] "RemoveContainer" containerID="9ceaf71601ca5b306f02aa9d00a2513184be1c350be1672210a3ce2f685c0b2c" Oct 30 03:56:23.157686 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:56:23.157668 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ceaf71601ca5b306f02aa9d00a2513184be1c350be1672210a3ce2f685c0b2c\": container with ID starting with 9ceaf71601ca5b306f02aa9d00a2513184be1c350be1672210a3ce2f685c0b2c not found: ID does not exist" containerID="9ceaf71601ca5b306f02aa9d00a2513184be1c350be1672210a3ce2f685c0b2c" Oct 30 03:56:23.157742 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:23.157692 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ceaf71601ca5b306f02aa9d00a2513184be1c350be1672210a3ce2f685c0b2c"} err="failed to get container status \"9ceaf71601ca5b306f02aa9d00a2513184be1c350be1672210a3ce2f685c0b2c\": rpc error: code = NotFound desc = could not find container \"9ceaf71601ca5b306f02aa9d00a2513184be1c350be1672210a3ce2f685c0b2c\": container with ID starting with 9ceaf71601ca5b306f02aa9d00a2513184be1c350be1672210a3ce2f685c0b2c not found: ID does not exist" Oct 30 03:56:23.201103 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:23.201068 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 03:56:23.214844 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:23.214807 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 03:56:23.247950 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:23.247896 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=5.24788362 podStartE2EDuration="5.24788362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:23.247382481 +0000 UTC m=+4495.361697456" watchObservedRunningTime="2026-10-30 03:56:23.24788362 +0000 UTC m=+4495.362198594" Oct 30 03:56:23.944061 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:56:23.944025 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:56:24.010673 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:24.010637 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" path="/var/lib/kubelet/pods/3a64a0a4-a619-46cc-aa53-c3b5f46b87f7/volumes" Oct 30 03:56:28.440546 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:28.440430 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:56:28.463076 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:28.453468 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 03:56:34.144419 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:56:34.144388 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:56:53.942548 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:56:53.942510 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:57:14.565985 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:14.565948 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:57:14.566601 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:14.566197 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="6634a735-5012-4a14-92db-7ef215b34cb0" containerName="perses" containerID="cri-o://442ca00fb24f479871d66b32a68251627911b44555fda4e6de856c7d8217a459" gracePeriod=30 Oct 30 03:57:14.810515 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:14.810490 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:14.963768 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:14.963683 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6634a735-5012-4a14-92db-7ef215b34cb0-plugins\" (UniqueName: \"kubernetes.io/empty-dir/6634a735-5012-4a14-92db-7ef215b34cb0-plugins\") pod \"6634a735-5012-4a14-92db-7ef215b34cb0\" (UID: \"6634a735-5012-4a14-92db-7ef215b34cb0\") " Oct 30 03:57:14.963768 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:14.963752 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6634a735-5012-4a14-92db-7ef215b34cb0-config\" (UniqueName: \"kubernetes.io/configmap/6634a735-5012-4a14-92db-7ef215b34cb0-config\") pod \"6634a735-5012-4a14-92db-7ef215b34cb0\" (UID: \"6634a735-5012-4a14-92db-7ef215b34cb0\") " Oct 30 03:57:14.963962 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:14.963864 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"6634a735-5012-4a14-92db-7ef215b34cb0\" (UID: \"6634a735-5012-4a14-92db-7ef215b34cb0\") " Oct 30 03:57:14.963962 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:14.963902 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6634a735-5012-4a14-92db-7ef215b34cb0-kube-api-access-4fcx4\" (UniqueName: \"kubernetes.io/projected/6634a735-5012-4a14-92db-7ef215b34cb0-kube-api-access-4fcx4\") pod \"6634a735-5012-4a14-92db-7ef215b34cb0\" (UID: \"6634a735-5012-4a14-92db-7ef215b34cb0\") " Oct 30 03:57:14.964150 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:14.964124 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6634a735-5012-4a14-92db-7ef215b34cb0-config" pod "6634a735-5012-4a14-92db-7ef215b34cb0" (UID: "6634a735-5012-4a14-92db-7ef215b34cb0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:57:14.966071 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:14.966045 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6634a735-5012-4a14-92db-7ef215b34cb0-kube-api-access-4fcx4" pod "6634a735-5012-4a14-92db-7ef215b34cb0" (UID: "6634a735-5012-4a14-92db-7ef215b34cb0"). InnerVolumeSpecName "kube-api-access-4fcx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:57:14.966603 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:14.966581 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e" pod "6634a735-5012-4a14-92db-7ef215b34cb0" (UID: "6634a735-5012-4a14-92db-7ef215b34cb0"). InnerVolumeSpecName "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 03:57:15.064514 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:15.064481 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6634a735-5012-4a14-92db-7ef215b34cb0-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:57:15.064692 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:15.064531 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 03:57:15.064692 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:15.064543 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fcx4\" (UniqueName: \"kubernetes.io/projected/6634a735-5012-4a14-92db-7ef215b34cb0-kube-api-access-4fcx4\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:57:15.082596 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:15.082570 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e") on node "ip-10-0-141-200.ec2.internal" Oct 30 03:57:15.117847 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:15.117810 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6634a735-5012-4a14-92db-7ef215b34cb0-plugins" pod "6634a735-5012-4a14-92db-7ef215b34cb0" (UID: "6634a735-5012-4a14-92db-7ef215b34cb0"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:57:15.165252 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:15.165218 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" Oct 30 03:57:15.165252 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:15.165247 5882 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/6634a735-5012-4a14-92db-7ef215b34cb0-plugins\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:57:15.397755 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:15.397720 5882 generic.go:356] "Generic (PLEG): container finished" podID="6634a735-5012-4a14-92db-7ef215b34cb0" containerID="442ca00fb24f479871d66b32a68251627911b44555fda4e6de856c7d8217a459" exitCode=0 Oct 30 03:57:15.397928 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:15.397763 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:15.397928 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:15.397804 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"6634a735-5012-4a14-92db-7ef215b34cb0","Type":"ContainerDied","Data":"442ca00fb24f479871d66b32a68251627911b44555fda4e6de856c7d8217a459"} Oct 30 03:57:15.397928 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:15.397848 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"6634a735-5012-4a14-92db-7ef215b34cb0","Type":"ContainerDied","Data":"cdd0f2440ff633ee280ec437722c0fbfda0fdfbe3cf48b80b87ca09253b8acea"} Oct 30 03:57:15.397928 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:15.397866 5882 scope.go:122] "RemoveContainer" containerID="442ca00fb24f479871d66b32a68251627911b44555fda4e6de856c7d8217a459" Oct 30 03:57:15.406986 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:15.406969 5882 scope.go:122] "RemoveContainer" containerID="442ca00fb24f479871d66b32a68251627911b44555fda4e6de856c7d8217a459" Oct 30 03:57:15.407231 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:57:15.407214 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"442ca00fb24f479871d66b32a68251627911b44555fda4e6de856c7d8217a459\": container with ID starting with 442ca00fb24f479871d66b32a68251627911b44555fda4e6de856c7d8217a459 not found: ID does not exist" containerID="442ca00fb24f479871d66b32a68251627911b44555fda4e6de856c7d8217a459" Oct 30 03:57:15.407275 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:15.407239 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"442ca00fb24f479871d66b32a68251627911b44555fda4e6de856c7d8217a459"} err="failed to get container status \"442ca00fb24f479871d66b32a68251627911b44555fda4e6de856c7d8217a459\": rpc error: code = NotFound desc = could not find container \"442ca00fb24f479871d66b32a68251627911b44555fda4e6de856c7d8217a459\": container with ID starting with 442ca00fb24f479871d66b32a68251627911b44555fda4e6de856c7d8217a459 not found: ID does not exist" Oct 30 03:57:15.457803 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:15.457770 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:57:15.460050 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:15.460029 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:57:16.010824 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:16.010794 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6634a735-5012-4a14-92db-7ef215b34cb0" path="/var/lib/kubelet/pods/6634a735-5012-4a14-92db-7ef215b34cb0/volumes" Oct 30 03:57:18.070872 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.070838 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:57:18.071244 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.071219 5882 cpu_manager.go:414] "containerMap: removing container" podUID="3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" containerName="alertmanager" Oct 30 03:57:18.071244 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.071234 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" containerName="alertmanager" Oct 30 03:57:18.071244 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.071244 5882 cpu_manager.go:414] "containerMap: removing container" podUID="3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" containerName="init-config-reloader" Oct 30 03:57:18.071347 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.071249 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" containerName="init-config-reloader" Oct 30 03:57:18.071347 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.071265 5882 cpu_manager.go:414] "containerMap: removing container" podUID="6634a735-5012-4a14-92db-7ef215b34cb0" containerName="perses" Oct 30 03:57:18.071347 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.071270 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6634a735-5012-4a14-92db-7ef215b34cb0" containerName="perses" Oct 30 03:57:18.071347 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.071280 5882 cpu_manager.go:414] "containerMap: removing container" podUID="3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" containerName="config-reloader" Oct 30 03:57:18.071347 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.071284 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" containerName="config-reloader" Oct 30 03:57:18.071494 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.071361 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" containerName="alertmanager" Oct 30 03:57:18.071494 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.071371 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6634a735-5012-4a14-92db-7ef215b34cb0" containerName="perses" Oct 30 03:57:18.071494 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.071383 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a64a0a4-a619-46cc-aa53-c3b5f46b87f7" containerName="config-reloader" Oct 30 03:57:18.094691 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.094648 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:57:18.094849 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.094766 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:18.116841 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.116814 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 03:57:18.125270 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.125253 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 03:57:18.125601 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.125580 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 30 03:57:18.125724 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.125709 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 03:57:18.192554 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.192525 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-config\") pod \"data-science-perses-0\" (UID: \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:18.192746 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.192565 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-plugins\") pod \"data-science-perses-0\" (UID: \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:18.192746 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.192598 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwvw5\" (UniqueName: \"kubernetes.io/projected/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-kube-api-access-lwvw5\") pod \"data-science-perses-0\" (UID: \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:18.293490 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.293457 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-plugins\") pod \"data-science-perses-0\" (UID: \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:18.293689 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.293502 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwvw5\" (UniqueName: \"kubernetes.io/projected/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-kube-api-access-lwvw5\") pod \"data-science-perses-0\" (UID: \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:18.293689 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.293558 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-config\") pod \"data-science-perses-0\" (UID: \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:18.293932 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.293910 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-plugins\") pod \"data-science-perses-0\" (UID: \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:18.294154 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.294136 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-config\") pod \"data-science-perses-0\" (UID: \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:18.318650 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:18.318627 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwvw5\" (UniqueName: \"kubernetes.io/projected/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-kube-api-access-lwvw5\") pod \"data-science-perses-0\" (UID: \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:22.933333 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:22.933300 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:22.966318 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:22.966291 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:23.034440 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:23.034413 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:23.034584 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:23.034520 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:23.050878 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:23.050852 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\") DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:23.154223 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:23.154193 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2c422b71c1e0b8fdf34659f2cef7846e06b4353ed88027dc99ade5d7fc29e9d6/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:23.185330 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:23.185267 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:23.225138 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:23.225105 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:23.519273 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:23.519239 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:57:23.520239 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:57:23.520212 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac8a0d98_b3b7_4601_a3a2_770a786ebb48.slice/crio-e557de72dd627e7415781a887986fd41cd02f855482b3147e885ff5dc815c22a WatchSource:0}: Error finding container e557de72dd627e7415781a887986fd41cd02f855482b3147e885ff5dc815c22a: Status 404 returned error can't find the container with id e557de72dd627e7415781a887986fd41cd02f855482b3147e885ff5dc815c22a Oct 30 03:57:23.944598 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:57:23.944509 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:57:24.446583 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:24.446536 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"ac8a0d98-b3b7-4601-a3a2-770a786ebb48","Type":"ContainerStarted","Data":"74dab3a0facda371cafde0a4145d363fb9634e796e074dd329ac4f8e4643c50e"} Oct 30 03:57:24.446771 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:24.446593 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"ac8a0d98-b3b7-4601-a3a2-770a786ebb48","Type":"ContainerStarted","Data":"e557de72dd627e7415781a887986fd41cd02f855482b3147e885ff5dc815c22a"} Oct 30 03:57:24.446771 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:24.446642 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:24.491404 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:24.491350 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=6.491336825 podStartE2EDuration="6.491336825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:24.491007784 +0000 UTC m=+4556.605322763" watchObservedRunningTime="2026-10-30 03:57:24.491336825 +0000 UTC m=+4556.605651796" Oct 30 03:57:28.701208 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:28.701171 5882 scope.go:122] "RemoveContainer" containerID="ed5ba24a3848cf9cecf66d4ec9df7da747622e6f3cdb8ca41967dfe3d0db793d" Oct 30 03:57:33.606194 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:33.606157 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:57:33.606777 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:33.606369 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="ac8a0d98-b3b7-4601-a3a2-770a786ebb48" containerName="perses" containerID="cri-o://74dab3a0facda371cafde0a4145d363fb9634e796e074dd329ac4f8e4643c50e" gracePeriod=30 Oct 30 03:57:33.608426 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:33.608401 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:33.854991 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:33.854967 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:33.926402 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:33.926323 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-plugins\" (UniqueName: \"kubernetes.io/empty-dir/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-plugins\") pod \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\" (UID: \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\") " Oct 30 03:57:33.926402 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:33.926378 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-config\" (UniqueName: \"kubernetes.io/configmap/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-config\") pod \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\" (UID: \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\") " Oct 30 03:57:33.926671 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:33.926415 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-kube-api-access-lwvw5\" (UniqueName: \"kubernetes.io/projected/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-kube-api-access-lwvw5\") pod \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\" (UID: \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\") " Oct 30 03:57:33.926671 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:33.926592 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\" (UID: \"ac8a0d98-b3b7-4601-a3a2-770a786ebb48\") " Oct 30 03:57:33.926790 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:33.926748 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-config" pod "ac8a0d98-b3b7-4601-a3a2-770a786ebb48" (UID: "ac8a0d98-b3b7-4601-a3a2-770a786ebb48"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:57:33.926865 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:33.926846 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:57:33.928924 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:33.928895 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-kube-api-access-lwvw5" pod "ac8a0d98-b3b7-4601-a3a2-770a786ebb48" (UID: "ac8a0d98-b3b7-4601-a3a2-770a786ebb48"). InnerVolumeSpecName "kube-api-access-lwvw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:57:33.930320 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:33.930295 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e" pod "ac8a0d98-b3b7-4601-a3a2-770a786ebb48" (UID: "ac8a0d98-b3b7-4601-a3a2-770a786ebb48"). InnerVolumeSpecName "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 03:57:34.028221 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:34.028183 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwvw5\" (UniqueName: \"kubernetes.io/projected/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-kube-api-access-lwvw5\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:57:34.028386 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:34.028239 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 03:57:34.042635 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:34.042599 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e") on node "ip-10-0-141-200.ec2.internal" Oct 30 03:57:34.070476 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:34.070441 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-plugins" pod "ac8a0d98-b3b7-4601-a3a2-770a786ebb48" (UID: "ac8a0d98-b3b7-4601-a3a2-770a786ebb48"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:57:34.129488 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:34.129454 5882 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/ac8a0d98-b3b7-4601-a3a2-770a786ebb48-plugins\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:57:34.129488 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:34.129485 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" Oct 30 03:57:34.494599 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:34.494566 5882 generic.go:356] "Generic (PLEG): container finished" podID="ac8a0d98-b3b7-4601-a3a2-770a786ebb48" containerID="74dab3a0facda371cafde0a4145d363fb9634e796e074dd329ac4f8e4643c50e" exitCode=0 Oct 30 03:57:34.494785 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:34.494633 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:34.494785 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:34.494651 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"ac8a0d98-b3b7-4601-a3a2-770a786ebb48","Type":"ContainerDied","Data":"74dab3a0facda371cafde0a4145d363fb9634e796e074dd329ac4f8e4643c50e"} Oct 30 03:57:34.494785 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:34.494691 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"ac8a0d98-b3b7-4601-a3a2-770a786ebb48","Type":"ContainerDied","Data":"e557de72dd627e7415781a887986fd41cd02f855482b3147e885ff5dc815c22a"} Oct 30 03:57:34.494785 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:34.494708 5882 scope.go:122] "RemoveContainer" containerID="74dab3a0facda371cafde0a4145d363fb9634e796e074dd329ac4f8e4643c50e" Oct 30 03:57:34.504161 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:34.504143 5882 scope.go:122] "RemoveContainer" containerID="74dab3a0facda371cafde0a4145d363fb9634e796e074dd329ac4f8e4643c50e" Oct 30 03:57:34.504441 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:57:34.504425 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74dab3a0facda371cafde0a4145d363fb9634e796e074dd329ac4f8e4643c50e\": container with ID starting with 74dab3a0facda371cafde0a4145d363fb9634e796e074dd329ac4f8e4643c50e not found: ID does not exist" containerID="74dab3a0facda371cafde0a4145d363fb9634e796e074dd329ac4f8e4643c50e" Oct 30 03:57:34.504482 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:34.504449 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74dab3a0facda371cafde0a4145d363fb9634e796e074dd329ac4f8e4643c50e"} err="failed to get container status \"74dab3a0facda371cafde0a4145d363fb9634e796e074dd329ac4f8e4643c50e\": rpc error: code = NotFound desc = could not find container \"74dab3a0facda371cafde0a4145d363fb9634e796e074dd329ac4f8e4643c50e\": container with ID starting with 74dab3a0facda371cafde0a4145d363fb9634e796e074dd329ac4f8e4643c50e not found: ID does not exist" Oct 30 03:57:34.549202 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:34.549175 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:57:34.563041 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:34.563016 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:57:36.009867 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:36.009834 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac8a0d98-b3b7-4601-a3a2-770a786ebb48" path="/var/lib/kubelet/pods/ac8a0d98-b3b7-4601-a3a2-770a786ebb48/volumes" Oct 30 03:57:36.936606 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:36.936575 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:57:36.936927 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:36.936915 5882 cpu_manager.go:414] "containerMap: removing container" podUID="ac8a0d98-b3b7-4601-a3a2-770a786ebb48" containerName="perses" Oct 30 03:57:36.936971 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:36.936929 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac8a0d98-b3b7-4601-a3a2-770a786ebb48" containerName="perses" Oct 30 03:57:36.937025 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:36.937014 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac8a0d98-b3b7-4601-a3a2-770a786ebb48" containerName="perses" Oct 30 03:57:36.957021 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:36.956993 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:36.957722 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:36.957694 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:57:36.991834 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:36.991807 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 30 03:57:36.991998 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:36.991955 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 03:57:36.992069 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:36.992054 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 03:57:36.992120 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:36.992066 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 03:57:37.157342 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:37.157313 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7924d92d-db24-4478-aa73-475f627481e4-config\") pod \"data-science-perses-0\" (UID: \"7924d92d-db24-4478-aa73-475f627481e4\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:37.157726 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:37.157357 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t6l8\" (UniqueName: \"kubernetes.io/projected/7924d92d-db24-4478-aa73-475f627481e4-kube-api-access-8t6l8\") pod \"data-science-perses-0\" (UID: \"7924d92d-db24-4478-aa73-475f627481e4\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:37.157726 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:37.157381 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/7924d92d-db24-4478-aa73-475f627481e4-plugins\") pod \"data-science-perses-0\" (UID: \"7924d92d-db24-4478-aa73-475f627481e4\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:37.258745 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:37.258712 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t6l8\" (UniqueName: \"kubernetes.io/projected/7924d92d-db24-4478-aa73-475f627481e4-kube-api-access-8t6l8\") pod \"data-science-perses-0\" (UID: \"7924d92d-db24-4478-aa73-475f627481e4\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:37.258897 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:37.258750 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/7924d92d-db24-4478-aa73-475f627481e4-plugins\") pod \"data-science-perses-0\" (UID: \"7924d92d-db24-4478-aa73-475f627481e4\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:37.258897 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:37.258816 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7924d92d-db24-4478-aa73-475f627481e4-config\") pod \"data-science-perses-0\" (UID: \"7924d92d-db24-4478-aa73-475f627481e4\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:37.259248 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:37.259227 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/7924d92d-db24-4478-aa73-475f627481e4-plugins\") pod \"data-science-perses-0\" (UID: \"7924d92d-db24-4478-aa73-475f627481e4\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:37.259462 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:37.259445 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7924d92d-db24-4478-aa73-475f627481e4-config\") pod \"data-science-perses-0\" (UID: \"7924d92d-db24-4478-aa73-475f627481e4\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:37.283767 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:37.283742 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t6l8\" (UniqueName: \"kubernetes.io/projected/7924d92d-db24-4478-aa73-475f627481e4-kube-api-access-8t6l8\") pod \"data-science-perses-0\" (UID: \"7924d92d-db24-4478-aa73-475f627481e4\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:43.509252 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:43.509206 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"7924d92d-db24-4478-aa73-475f627481e4\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:43.556297 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:43.556267 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"7924d92d-db24-4478-aa73-475f627481e4\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:43.609847 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:43.609817 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"7924d92d-db24-4478-aa73-475f627481e4\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:43.609947 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:43.609930 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"7924d92d-db24-4478-aa73-475f627481e4\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:43.626165 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:43.626137 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"7924d92d-db24-4478-aa73-475f627481e4\") DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:43.729833 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:43.729805 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"7924d92d-db24-4478-aa73-475f627481e4\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2c422b71c1e0b8fdf34659f2cef7846e06b4353ed88027dc99ade5d7fc29e9d6/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:43.761390 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:43.761325 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"7924d92d-db24-4478-aa73-475f627481e4\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:43.881779 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:43.881743 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:44.174960 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:44.174926 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:57:44.175888 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:57:44.175862 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7924d92d_db24_4478_aa73_475f627481e4.slice/crio-620c13a1609b8cb41bb26f4c15a479df2d8a405a11d80e3b8955c866dafd8cdb WatchSource:0}: Error finding container 620c13a1609b8cb41bb26f4c15a479df2d8a405a11d80e3b8955c866dafd8cdb: Status 404 returned error can't find the container with id 620c13a1609b8cb41bb26f4c15a479df2d8a405a11d80e3b8955c866dafd8cdb Oct 30 03:57:44.547189 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:44.547150 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"7924d92d-db24-4478-aa73-475f627481e4","Type":"ContainerStarted","Data":"519cb722e5af660a73a100e10e470ce2f0c5aa5c4d4c096e43ad22e3f775a30f"} Oct 30 03:57:44.547774 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:44.547750 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"7924d92d-db24-4478-aa73-475f627481e4","Type":"ContainerStarted","Data":"620c13a1609b8cb41bb26f4c15a479df2d8a405a11d80e3b8955c866dafd8cdb"} Oct 30 03:57:44.548216 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:44.548192 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:57:44.594602 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:44.594537 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=8.594518959 podStartE2EDuration="8.594518959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:44.593966685 +0000 UTC m=+4576.708281660" watchObservedRunningTime="2026-10-30 03:57:44.594518959 +0000 UTC m=+4576.708833931" Oct 30 03:57:53.942768 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:57:53.942731 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:57:56.561034 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:57:56.561005 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:58:23.942340 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:58:23.942257 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:58:24.686318 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:24.686286 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:58:24.686508 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:24.686473 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="7924d92d-db24-4478-aa73-475f627481e4" containerName="perses" containerID="cri-o://519cb722e5af660a73a100e10e470ce2f0c5aa5c4d4c096e43ad22e3f775a30f" gracePeriod=30 Oct 30 03:58:24.931346 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:24.931322 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:58:25.075256 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.075221 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"7924d92d-db24-4478-aa73-475f627481e4\" (UID: \"7924d92d-db24-4478-aa73-475f627481e4\") " Oct 30 03:58:25.075761 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.075294 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7924d92d-db24-4478-aa73-475f627481e4-plugins\" (UniqueName: \"kubernetes.io/empty-dir/7924d92d-db24-4478-aa73-475f627481e4-plugins\") pod \"7924d92d-db24-4478-aa73-475f627481e4\" (UID: \"7924d92d-db24-4478-aa73-475f627481e4\") " Oct 30 03:58:25.075761 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.075410 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7924d92d-db24-4478-aa73-475f627481e4-kube-api-access-8t6l8\" (UniqueName: \"kubernetes.io/projected/7924d92d-db24-4478-aa73-475f627481e4-kube-api-access-8t6l8\") pod \"7924d92d-db24-4478-aa73-475f627481e4\" (UID: \"7924d92d-db24-4478-aa73-475f627481e4\") " Oct 30 03:58:25.075761 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.075448 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7924d92d-db24-4478-aa73-475f627481e4-config\" (UniqueName: \"kubernetes.io/configmap/7924d92d-db24-4478-aa73-475f627481e4-config\") pod \"7924d92d-db24-4478-aa73-475f627481e4\" (UID: \"7924d92d-db24-4478-aa73-475f627481e4\") " Oct 30 03:58:25.076089 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.076057 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7924d92d-db24-4478-aa73-475f627481e4-config" pod "7924d92d-db24-4478-aa73-475f627481e4" (UID: "7924d92d-db24-4478-aa73-475f627481e4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:58:25.078314 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.078272 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e" pod "7924d92d-db24-4478-aa73-475f627481e4" (UID: "7924d92d-db24-4478-aa73-475f627481e4"). InnerVolumeSpecName "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 03:58:25.078423 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.078395 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7924d92d-db24-4478-aa73-475f627481e4-kube-api-access-8t6l8" pod "7924d92d-db24-4478-aa73-475f627481e4" (UID: "7924d92d-db24-4478-aa73-475f627481e4"). InnerVolumeSpecName "kube-api-access-8t6l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:58:25.178658 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.177391 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t6l8\" (UniqueName: \"kubernetes.io/projected/7924d92d-db24-4478-aa73-475f627481e4-kube-api-access-8t6l8\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:58:25.178658 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.177429 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7924d92d-db24-4478-aa73-475f627481e4-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:58:25.178658 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.177469 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 03:58:25.196344 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.196318 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e") on node "ip-10-0-141-200.ec2.internal" Oct 30 03:58:25.231266 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.231225 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7924d92d-db24-4478-aa73-475f627481e4-plugins" pod "7924d92d-db24-4478-aa73-475f627481e4" (UID: "7924d92d-db24-4478-aa73-475f627481e4"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:58:25.278730 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.278697 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" Oct 30 03:58:25.278885 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.278756 5882 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/7924d92d-db24-4478-aa73-475f627481e4-plugins\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 03:58:25.753877 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.753831 5882 generic.go:356] "Generic (PLEG): container finished" podID="7924d92d-db24-4478-aa73-475f627481e4" containerID="519cb722e5af660a73a100e10e470ce2f0c5aa5c4d4c096e43ad22e3f775a30f" exitCode=0 Oct 30 03:58:25.754044 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.753897 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:58:25.754044 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.753916 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"7924d92d-db24-4478-aa73-475f627481e4","Type":"ContainerDied","Data":"519cb722e5af660a73a100e10e470ce2f0c5aa5c4d4c096e43ad22e3f775a30f"} Oct 30 03:58:25.754044 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.753960 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"7924d92d-db24-4478-aa73-475f627481e4","Type":"ContainerDied","Data":"620c13a1609b8cb41bb26f4c15a479df2d8a405a11d80e3b8955c866dafd8cdb"} Oct 30 03:58:25.754044 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.753982 5882 scope.go:122] "RemoveContainer" containerID="519cb722e5af660a73a100e10e470ce2f0c5aa5c4d4c096e43ad22e3f775a30f" Oct 30 03:58:25.763283 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.763263 5882 scope.go:122] "RemoveContainer" containerID="519cb722e5af660a73a100e10e470ce2f0c5aa5c4d4c096e43ad22e3f775a30f" Oct 30 03:58:25.763542 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:58:25.763522 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"519cb722e5af660a73a100e10e470ce2f0c5aa5c4d4c096e43ad22e3f775a30f\": container with ID starting with 519cb722e5af660a73a100e10e470ce2f0c5aa5c4d4c096e43ad22e3f775a30f not found: ID does not exist" containerID="519cb722e5af660a73a100e10e470ce2f0c5aa5c4d4c096e43ad22e3f775a30f" Oct 30 03:58:25.763604 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.763553 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519cb722e5af660a73a100e10e470ce2f0c5aa5c4d4c096e43ad22e3f775a30f"} err="failed to get container status \"519cb722e5af660a73a100e10e470ce2f0c5aa5c4d4c096e43ad22e3f775a30f\": rpc error: code = NotFound desc = could not find container \"519cb722e5af660a73a100e10e470ce2f0c5aa5c4d4c096e43ad22e3f775a30f\": container with ID starting with 519cb722e5af660a73a100e10e470ce2f0c5aa5c4d4c096e43ad22e3f775a30f not found: ID does not exist" Oct 30 03:58:25.808793 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.808764 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:58:25.822761 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:25.822733 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:58:26.009034 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:26.008999 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7924d92d-db24-4478-aa73-475f627481e4" path="/var/lib/kubelet/pods/7924d92d-db24-4478-aa73-475f627481e4/volumes" Oct 30 03:58:28.085264 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.085233 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 30 03:58:28.085672 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.085576 5882 cpu_manager.go:414] "containerMap: removing container" podUID="7924d92d-db24-4478-aa73-475f627481e4" containerName="perses" Oct 30 03:58:28.085672 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.085586 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7924d92d-db24-4478-aa73-475f627481e4" containerName="perses" Oct 30 03:58:28.085672 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.085662 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7924d92d-db24-4478-aa73-475f627481e4" containerName="perses" Oct 30 03:58:28.108124 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.108095 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 30 03:58:28.108275 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.108233 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs" Oct 30 03:58:28.125133 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.125103 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 03:58:28.125465 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.125438 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 03:58:28.138982 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.138960 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 03:58:28.203745 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.203718 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/df33968b-21ed-4b18-88a3-2b57f73faf58-data\") pod \"seaweedfs\" (UID: \"df33968b-21ed-4b18-88a3-2b57f73faf58\") " pod="redhat-ods-monitoring/seaweedfs" Oct 30 03:58:28.203891 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.203773 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr5xj\" (UniqueName: \"kubernetes.io/projected/df33968b-21ed-4b18-88a3-2b57f73faf58-kube-api-access-jr5xj\") pod \"seaweedfs\" (UID: \"df33968b-21ed-4b18-88a3-2b57f73faf58\") " pod="redhat-ods-monitoring/seaweedfs" Oct 30 03:58:28.304664 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.304604 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/df33968b-21ed-4b18-88a3-2b57f73faf58-data\") pod \"seaweedfs\" (UID: \"df33968b-21ed-4b18-88a3-2b57f73faf58\") " pod="redhat-ods-monitoring/seaweedfs" Oct 30 03:58:28.304847 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.304716 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr5xj\" (UniqueName: \"kubernetes.io/projected/df33968b-21ed-4b18-88a3-2b57f73faf58-kube-api-access-jr5xj\") pod \"seaweedfs\" (UID: \"df33968b-21ed-4b18-88a3-2b57f73faf58\") " pod="redhat-ods-monitoring/seaweedfs" Oct 30 03:58:28.304975 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.304953 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/df33968b-21ed-4b18-88a3-2b57f73faf58-data\") pod \"seaweedfs\" (UID: \"df33968b-21ed-4b18-88a3-2b57f73faf58\") " pod="redhat-ods-monitoring/seaweedfs" Oct 30 03:58:28.329949 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.329923 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr5xj\" (UniqueName: \"kubernetes.io/projected/df33968b-21ed-4b18-88a3-2b57f73faf58-kube-api-access-jr5xj\") pod \"seaweedfs\" (UID: \"df33968b-21ed-4b18-88a3-2b57f73faf58\") " pod="redhat-ods-monitoring/seaweedfs" Oct 30 03:58:28.445837 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.445741 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs" Oct 30 03:58:28.724975 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.724897 5882 scope.go:122] "RemoveContainer" containerID="fc198c0f2c8336bfd96d2fec338675aa7a5684fd021a164006ff44f169f51b52" Oct 30 03:58:28.734441 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.734115 5882 scope.go:122] "RemoveContainer" containerID="290a9ad22f964bfcdb29b58308f5ad18d2b766ecd22b3b27120f18a8c2f2b34d" Oct 30 03:58:28.735462 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.735439 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 30 03:58:28.737144 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:58:28.737119 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf33968b_21ed_4b18_88a3_2b57f73faf58.slice/crio-d02257e810fec3c06ae85691f0c6c763d45fbd0a812de13665c2ef92dbf1e9ab WatchSource:0}: Error finding container d02257e810fec3c06ae85691f0c6c763d45fbd0a812de13665c2ef92dbf1e9ab: Status 404 returned error can't find the container with id d02257e810fec3c06ae85691f0c6c763d45fbd0a812de13665c2ef92dbf1e9ab Oct 30 03:58:28.738870 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.738853 5882 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:58:28.747041 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.747020 5882 scope.go:122] "RemoveContainer" containerID="498842a5e30a1c5ad9dac1b8e9a38088c6b5c726170cba53ea506429afe187c2" Oct 30 03:58:28.754543 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.754520 5882 scope.go:122] "RemoveContainer" containerID="58ace56a3734018779814aec900038b296e31bd506ff400bff57248fa8114565" Oct 30 03:58:28.777891 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:28.777853 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs" event={"ID":"df33968b-21ed-4b18-88a3-2b57f73faf58","Type":"ContainerStarted","Data":"d02257e810fec3c06ae85691f0c6c763d45fbd0a812de13665c2ef92dbf1e9ab"} Oct 30 03:58:33.810559 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:33.810519 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs" event={"ID":"df33968b-21ed-4b18-88a3-2b57f73faf58","Type":"ContainerStarted","Data":"ae92b66bb40e6555c662b24210872816674845ec065cb056d57e167e4de3b356"} Oct 30 03:58:33.811024 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:33.810579 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/seaweedfs" Oct 30 03:58:33.852971 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:33.852924 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/seaweedfs" podStartSLOduration=1.207167837 podStartE2EDuration="5.852910125s" totalImagesPullingTime="4.645742288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:58:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:58:33.85255232 +0000 UTC m=+4625.966867295" watchObservedRunningTime="2026-10-30 03:58:33.852910125 +0000 UTC m=+4625.967225094" Oct 30 03:58:44.818314 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:44.818280 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/seaweedfs" Oct 30 03:58:53.942781 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:58:53.942741 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:58:56.394098 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:56.394068 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:58:56.397464 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:56.397445 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:58:56.416664 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:56.416636 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:58:56.477353 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:56.477321 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 30 03:58:56.538731 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:56.538693 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/334d4bb9-f380-4378-81a4-dd46ea1c377d-config\") pod \"data-science-perses-0\" (UID: \"334d4bb9-f380-4378-81a4-dd46ea1c377d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:58:56.538923 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:56.538846 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww9nz\" (UniqueName: \"kubernetes.io/projected/334d4bb9-f380-4378-81a4-dd46ea1c377d-kube-api-access-ww9nz\") pod \"data-science-perses-0\" (UID: \"334d4bb9-f380-4378-81a4-dd46ea1c377d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:58:56.538923 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:56.538898 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/334d4bb9-f380-4378-81a4-dd46ea1c377d-plugins\") pod \"data-science-perses-0\" (UID: \"334d4bb9-f380-4378-81a4-dd46ea1c377d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:58:56.640222 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:56.640187 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww9nz\" (UniqueName: \"kubernetes.io/projected/334d4bb9-f380-4378-81a4-dd46ea1c377d-kube-api-access-ww9nz\") pod \"data-science-perses-0\" (UID: \"334d4bb9-f380-4378-81a4-dd46ea1c377d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:58:56.640404 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:56.640232 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/334d4bb9-f380-4378-81a4-dd46ea1c377d-plugins\") pod \"data-science-perses-0\" (UID: \"334d4bb9-f380-4378-81a4-dd46ea1c377d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:58:56.640404 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:56.640277 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/334d4bb9-f380-4378-81a4-dd46ea1c377d-config\") pod \"data-science-perses-0\" (UID: \"334d4bb9-f380-4378-81a4-dd46ea1c377d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:58:56.640747 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:56.640722 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/334d4bb9-f380-4378-81a4-dd46ea1c377d-plugins\") pod \"data-science-perses-0\" (UID: \"334d4bb9-f380-4378-81a4-dd46ea1c377d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:58:56.640879 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:56.640862 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/334d4bb9-f380-4378-81a4-dd46ea1c377d-config\") pod \"data-science-perses-0\" (UID: \"334d4bb9-f380-4378-81a4-dd46ea1c377d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:58:56.665218 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:58:56.665157 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww9nz\" (UniqueName: \"kubernetes.io/projected/334d4bb9-f380-4378-81a4-dd46ea1c377d-kube-api-access-ww9nz\") pod \"data-science-perses-0\" (UID: \"334d4bb9-f380-4378-81a4-dd46ea1c377d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:59:05.314765 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:59:05.314722 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"334d4bb9-f380-4378-81a4-dd46ea1c377d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:59:05.348221 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:59:05.348190 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"334d4bb9-f380-4378-81a4-dd46ea1c377d\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:59:05.415712 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:59:05.415671 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"334d4bb9-f380-4378-81a4-dd46ea1c377d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:59:05.415892 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:59:05.415828 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"334d4bb9-f380-4378-81a4-dd46ea1c377d\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:59:05.432182 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:59:05.432153 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"334d4bb9-f380-4378-81a4-dd46ea1c377d\") DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:59:05.537138 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:59:05.537106 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"334d4bb9-f380-4378-81a4-dd46ea1c377d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2c422b71c1e0b8fdf34659f2cef7846e06b4353ed88027dc99ade5d7fc29e9d6/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:59:05.569023 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:59:05.568964 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"334d4bb9-f380-4378-81a4-dd46ea1c377d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:59:05.710097 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:59:05.710061 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:59:06.002004 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:59:06.001974 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 03:59:06.002543 ip-10-0-141-200 kubenswrapper[5882]: W1030 03:59:06.002514 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod334d4bb9_f380_4378_81a4_dd46ea1c377d.slice/crio-33ce8add22149f3d58865c1414f9e7b4d8fad0340ba3de0485ea2005882915aa WatchSource:0}: Error finding container 33ce8add22149f3d58865c1414f9e7b4d8fad0340ba3de0485ea2005882915aa: Status 404 returned error can't find the container with id 33ce8add22149f3d58865c1414f9e7b4d8fad0340ba3de0485ea2005882915aa Oct 30 03:59:06.981412 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:59:06.981374 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"334d4bb9-f380-4378-81a4-dd46ea1c377d","Type":"ContainerStarted","Data":"5823080e1218652968da82a278c5ba970284bd88c2106778ea0279433145d828"} Oct 30 03:59:06.981902 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:59:06.981420 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"334d4bb9-f380-4378-81a4-dd46ea1c377d","Type":"ContainerStarted","Data":"33ce8add22149f3d58865c1414f9e7b4d8fad0340ba3de0485ea2005882915aa"} Oct 30 03:59:06.981902 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:59:06.981455 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:59:07.025272 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:59:07.025220 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=11.025207044 podStartE2EDuration="11.025207044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:59:07.024937649 +0000 UTC m=+4659.139252623" watchObservedRunningTime="2026-10-30 03:59:07.025207044 +0000 UTC m=+4659.139522012" Oct 30 03:59:17.990045 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:59:17.990010 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 03:59:23.942423 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:59:23.942384 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 03:59:28.796677 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:59:28.796640 5882 scope.go:122] "RemoveContainer" containerID="b77466ac0b5da2f9af6a99f1a5167f445310b206aae4c3cad531e351d0567cd8" Oct 30 03:59:28.805464 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:59:28.805443 5882 scope.go:122] "RemoveContainer" containerID="d7b67aea388f0a1676849178e92461484ce6a443e7f4dc2f445bc18874f75d62" Oct 30 03:59:28.812946 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:59:28.812918 5882 scope.go:122] "RemoveContainer" containerID="73bd312e8f7c08cddcdd4c17f3edaa50430d9323558b709cc821db7d4853fd7b" Oct 30 03:59:28.820537 ip-10-0-141-200 kubenswrapper[5882]: I1030 03:59:28.820516 5882 scope.go:122] "RemoveContainer" containerID="3ed7ab8720efc6aced60992b54743ea63d9c790fcff839b4d2c2fb9d8b38c022" Oct 30 03:59:53.943103 ip-10-0-141-200 kubenswrapper[5882]: E1030 03:59:53.943062 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:00:08.706910 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:08.706875 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:00:08.707402 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:08.707104 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="334d4bb9-f380-4378-81a4-dd46ea1c377d" containerName="perses" containerID="cri-o://5823080e1218652968da82a278c5ba970284bd88c2106778ea0279433145d828" gracePeriod=30 Oct 30 04:00:08.954342 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:08.954315 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:09.050347 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.050315 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334d4bb9-f380-4378-81a4-dd46ea1c377d-kube-api-access-ww9nz\" (UniqueName: \"kubernetes.io/projected/334d4bb9-f380-4378-81a4-dd46ea1c377d-kube-api-access-ww9nz\") pod \"334d4bb9-f380-4378-81a4-dd46ea1c377d\" (UID: \"334d4bb9-f380-4378-81a4-dd46ea1c377d\") " Oct 30 04:00:09.050555 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.050442 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"334d4bb9-f380-4378-81a4-dd46ea1c377d\" (UID: \"334d4bb9-f380-4378-81a4-dd46ea1c377d\") " Oct 30 04:00:09.050555 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.050540 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334d4bb9-f380-4378-81a4-dd46ea1c377d-plugins\" (UniqueName: \"kubernetes.io/empty-dir/334d4bb9-f380-4378-81a4-dd46ea1c377d-plugins\") pod \"334d4bb9-f380-4378-81a4-dd46ea1c377d\" (UID: \"334d4bb9-f380-4378-81a4-dd46ea1c377d\") " Oct 30 04:00:09.050691 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.050570 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/334d4bb9-f380-4378-81a4-dd46ea1c377d-config\" (UniqueName: \"kubernetes.io/configmap/334d4bb9-f380-4378-81a4-dd46ea1c377d-config\") pod \"334d4bb9-f380-4378-81a4-dd46ea1c377d\" (UID: \"334d4bb9-f380-4378-81a4-dd46ea1c377d\") " Oct 30 04:00:09.051495 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.051464 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334d4bb9-f380-4378-81a4-dd46ea1c377d-config" pod "334d4bb9-f380-4378-81a4-dd46ea1c377d" (UID: "334d4bb9-f380-4378-81a4-dd46ea1c377d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:09.056803 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.056772 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334d4bb9-f380-4378-81a4-dd46ea1c377d-kube-api-access-ww9nz" pod "334d4bb9-f380-4378-81a4-dd46ea1c377d" (UID: "334d4bb9-f380-4378-81a4-dd46ea1c377d"). InnerVolumeSpecName "kube-api-access-ww9nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:09.057222 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.057198 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e" pod "334d4bb9-f380-4378-81a4-dd46ea1c377d" (UID: "334d4bb9-f380-4378-81a4-dd46ea1c377d"). InnerVolumeSpecName "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:00:09.151608 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.151577 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/334d4bb9-f380-4378-81a4-dd46ea1c377d-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:00:09.151608 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.151605 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww9nz\" (UniqueName: \"kubernetes.io/projected/334d4bb9-f380-4378-81a4-dd46ea1c377d-kube-api-access-ww9nz\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:00:09.152137 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.151663 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:00:09.167827 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.167800 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:00:09.222368 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.222328 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334d4bb9-f380-4378-81a4-dd46ea1c377d-plugins" pod "334d4bb9-f380-4378-81a4-dd46ea1c377d" (UID: "334d4bb9-f380-4378-81a4-dd46ea1c377d"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:09.252702 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.252672 5882 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/334d4bb9-f380-4378-81a4-dd46ea1c377d-plugins\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:00:09.252702 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.252702 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" Oct 30 04:00:09.279272 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.279235 5882 generic.go:356] "Generic (PLEG): container finished" podID="334d4bb9-f380-4378-81a4-dd46ea1c377d" containerID="5823080e1218652968da82a278c5ba970284bd88c2106778ea0279433145d828" exitCode=0 Oct 30 04:00:09.279460 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.279272 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"334d4bb9-f380-4378-81a4-dd46ea1c377d","Type":"ContainerDied","Data":"5823080e1218652968da82a278c5ba970284bd88c2106778ea0279433145d828"} Oct 30 04:00:09.279460 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.279328 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"334d4bb9-f380-4378-81a4-dd46ea1c377d","Type":"ContainerDied","Data":"33ce8add22149f3d58865c1414f9e7b4d8fad0340ba3de0485ea2005882915aa"} Oct 30 04:00:09.279460 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.279351 5882 scope.go:122] "RemoveContainer" containerID="5823080e1218652968da82a278c5ba970284bd88c2106778ea0279433145d828" Oct 30 04:00:09.279460 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.279283 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:09.294648 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.294628 5882 scope.go:122] "RemoveContainer" containerID="5823080e1218652968da82a278c5ba970284bd88c2106778ea0279433145d828" Oct 30 04:00:09.294977 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:00:09.294950 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5823080e1218652968da82a278c5ba970284bd88c2106778ea0279433145d828\": container with ID starting with 5823080e1218652968da82a278c5ba970284bd88c2106778ea0279433145d828 not found: ID does not exist" containerID="5823080e1218652968da82a278c5ba970284bd88c2106778ea0279433145d828" Oct 30 04:00:09.295081 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.294987 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5823080e1218652968da82a278c5ba970284bd88c2106778ea0279433145d828"} err="failed to get container status \"5823080e1218652968da82a278c5ba970284bd88c2106778ea0279433145d828\": rpc error: code = NotFound desc = could not find container \"5823080e1218652968da82a278c5ba970284bd88c2106778ea0279433145d828\": container with ID starting with 5823080e1218652968da82a278c5ba970284bd88c2106778ea0279433145d828 not found: ID does not exist" Oct 30 04:00:09.336109 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.336031 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:00:09.342133 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.342103 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:00:09.738831 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.738799 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 30 04:00:09.739242 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:09.738986 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/seaweedfs" podUID="df33968b-21ed-4b18-88a3-2b57f73faf58" containerName="seaweedfs" containerID="cri-o://ae92b66bb40e6555c662b24210872816674845ec065cb056d57e167e4de3b356" gracePeriod=30 Oct 30 04:00:10.008909 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:10.008870 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="334d4bb9-f380-4378-81a4-dd46ea1c377d" path="/var/lib/kubelet/pods/334d4bb9-f380-4378-81a4-dd46ea1c377d/volumes" Oct 30 04:00:23.942161 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:00:23.942119 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:00:28.848586 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:28.848549 5882 scope.go:122] "RemoveContainer" containerID="873c2761acbf47c40ad8e9b7f4747e50d7c834e0fa6cbc0b7d9740366359b52c" Oct 30 04:00:28.856796 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:28.856774 5882 scope.go:122] "RemoveContainer" containerID="788dbed1297aca3a684a05e46215a32f0c006d063e7bc9f61a201ba4aff5a4d7" Oct 30 04:00:28.864186 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:28.864167 5882 scope.go:122] "RemoveContainer" containerID="23abf57f5bfc681e0e2dfbc606e7e41d914f13b28579a81f985d14fcbf762a58" Oct 30 04:00:28.871237 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:28.871217 5882 scope.go:122] "RemoveContainer" containerID="c59cd5bcfe4ca6c10564c1d309a483c451c7b42da05c04d90c3c0bebdf1f27a5" Oct 30 04:00:28.878208 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:28.878192 5882 scope.go:122] "RemoveContainer" containerID="d1a72a082f0b7c2e4472cf7db7be5d46dcbc551f34dec414191637768af44e3f" Oct 30 04:00:28.885125 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:28.885105 5882 scope.go:122] "RemoveContainer" containerID="de6bedfa62efb6fe45ab1a3b8f4d7f8d763b8654eca31a677ab09f149e8a1533" Oct 30 04:00:28.892130 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:28.892109 5882 scope.go:122] "RemoveContainer" containerID="5a1b93df1e6c403659cf1b7b7ec5825c47237a7bb9c7abc3d73b26bba4ea6910" Oct 30 04:00:28.899134 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:28.899115 5882 scope.go:122] "RemoveContainer" containerID="fe27740702cb261933154d1439bd7875f3749991b20479938592da1dc2c16fb0" Oct 30 04:00:28.906145 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:28.906124 5882 scope.go:122] "RemoveContainer" containerID="424dae49fb36c9381094d7be178ebe534542cd9f398353847813e8338a38fc6f" Oct 30 04:00:28.912974 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:28.912957 5882 scope.go:122] "RemoveContainer" containerID="ff055f6eb939e236d5015579139343b2c443c7b5af897f09c06c34b45c7a60e5" Oct 30 04:00:34.817331 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:34.817292 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/seaweedfs" podUID="df33968b-21ed-4b18-88a3-2b57f73faf58" containerName="seaweedfs" probeResult="failure" output="dial tcp 10.134.0.105:8333: connect: connection refused" Oct 30 04:00:34.983624 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:34.983584 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs" Oct 30 04:00:35.079359 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:35.079280 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df33968b-21ed-4b18-88a3-2b57f73faf58-data\" (UniqueName: \"kubernetes.io/empty-dir/df33968b-21ed-4b18-88a3-2b57f73faf58-data\") pod \"df33968b-21ed-4b18-88a3-2b57f73faf58\" (UID: \"df33968b-21ed-4b18-88a3-2b57f73faf58\") " Oct 30 04:00:35.079518 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:35.079388 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df33968b-21ed-4b18-88a3-2b57f73faf58-kube-api-access-jr5xj\" (UniqueName: \"kubernetes.io/projected/df33968b-21ed-4b18-88a3-2b57f73faf58-kube-api-access-jr5xj\") pod \"df33968b-21ed-4b18-88a3-2b57f73faf58\" (UID: \"df33968b-21ed-4b18-88a3-2b57f73faf58\") " Oct 30 04:00:35.080611 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:35.080583 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df33968b-21ed-4b18-88a3-2b57f73faf58-data" pod "df33968b-21ed-4b18-88a3-2b57f73faf58" (UID: "df33968b-21ed-4b18-88a3-2b57f73faf58"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:35.081425 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:35.081405 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df33968b-21ed-4b18-88a3-2b57f73faf58-kube-api-access-jr5xj" pod "df33968b-21ed-4b18-88a3-2b57f73faf58" (UID: "df33968b-21ed-4b18-88a3-2b57f73faf58"). InnerVolumeSpecName "kube-api-access-jr5xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:35.180971 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:35.180933 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr5xj\" (UniqueName: \"kubernetes.io/projected/df33968b-21ed-4b18-88a3-2b57f73faf58-kube-api-access-jr5xj\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:00:35.180971 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:35.180965 5882 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/df33968b-21ed-4b18-88a3-2b57f73faf58-data\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:00:35.408684 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:35.408591 5882 generic.go:356] "Generic (PLEG): container finished" podID="df33968b-21ed-4b18-88a3-2b57f73faf58" containerID="ae92b66bb40e6555c662b24210872816674845ec065cb056d57e167e4de3b356" exitCode=0 Oct 30 04:00:35.408684 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:35.408658 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs" event={"ID":"df33968b-21ed-4b18-88a3-2b57f73faf58","Type":"ContainerDied","Data":"ae92b66bb40e6555c662b24210872816674845ec065cb056d57e167e4de3b356"} Oct 30 04:00:35.408684 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:35.408672 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs" Oct 30 04:00:35.408889 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:35.408704 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs" event={"ID":"df33968b-21ed-4b18-88a3-2b57f73faf58","Type":"ContainerDied","Data":"d02257e810fec3c06ae85691f0c6c763d45fbd0a812de13665c2ef92dbf1e9ab"} Oct 30 04:00:35.408889 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:35.408722 5882 scope.go:122] "RemoveContainer" containerID="ae92b66bb40e6555c662b24210872816674845ec065cb056d57e167e4de3b356" Oct 30 04:00:35.417709 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:35.417694 5882 scope.go:122] "RemoveContainer" containerID="ae92b66bb40e6555c662b24210872816674845ec065cb056d57e167e4de3b356" Oct 30 04:00:35.417961 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:00:35.417942 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae92b66bb40e6555c662b24210872816674845ec065cb056d57e167e4de3b356\": container with ID starting with ae92b66bb40e6555c662b24210872816674845ec065cb056d57e167e4de3b356 not found: ID does not exist" containerID="ae92b66bb40e6555c662b24210872816674845ec065cb056d57e167e4de3b356" Oct 30 04:00:35.418019 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:35.417970 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae92b66bb40e6555c662b24210872816674845ec065cb056d57e167e4de3b356"} err="failed to get container status \"ae92b66bb40e6555c662b24210872816674845ec065cb056d57e167e4de3b356\": rpc error: code = NotFound desc = could not find container \"ae92b66bb40e6555c662b24210872816674845ec065cb056d57e167e4de3b356\": container with ID starting with ae92b66bb40e6555c662b24210872816674845ec065cb056d57e167e4de3b356 not found: ID does not exist" Oct 30 04:00:35.464717 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:35.464691 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 30 04:00:35.468016 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:35.467996 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 30 04:00:36.010913 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:36.010875 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df33968b-21ed-4b18-88a3-2b57f73faf58" path="/var/lib/kubelet/pods/df33968b-21ed-4b18-88a3-2b57f73faf58/volumes" Oct 30 04:00:38.327993 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:38.327963 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/fake-gcs-server"] Oct 30 04:00:38.328357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:38.328302 5882 cpu_manager.go:414] "containerMap: removing container" podUID="df33968b-21ed-4b18-88a3-2b57f73faf58" containerName="seaweedfs" Oct 30 04:00:38.328357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:38.328312 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df33968b-21ed-4b18-88a3-2b57f73faf58" containerName="seaweedfs" Oct 30 04:00:38.328357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:38.328330 5882 cpu_manager.go:414] "containerMap: removing container" podUID="334d4bb9-f380-4378-81a4-dd46ea1c377d" containerName="perses" Oct 30 04:00:38.328357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:38.328335 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="334d4bb9-f380-4378-81a4-dd46ea1c377d" containerName="perses" Oct 30 04:00:38.328499 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:38.328390 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="334d4bb9-f380-4378-81a4-dd46ea1c377d" containerName="perses" Oct 30 04:00:38.328499 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:38.328404 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="df33968b-21ed-4b18-88a3-2b57f73faf58" containerName="seaweedfs" Oct 30 04:00:38.332731 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:38.332715 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/fake-gcs-server" Oct 30 04:00:38.349841 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:38.349814 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/fake-gcs-server"] Oct 30 04:00:38.356515 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:38.356494 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 04:00:38.362885 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:38.362868 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 04:00:38.362976 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:38.362947 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 04:00:38.409397 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:38.409371 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cc56a994-26bb-4ca9-b0ee-81fa0a7b798c-data\") pod \"fake-gcs-server\" (UID: \"cc56a994-26bb-4ca9-b0ee-81fa0a7b798c\") " pod="redhat-ods-monitoring/fake-gcs-server" Oct 30 04:00:38.409530 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:38.409432 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bsgw\" (UniqueName: \"kubernetes.io/projected/cc56a994-26bb-4ca9-b0ee-81fa0a7b798c-kube-api-access-8bsgw\") pod \"fake-gcs-server\" (UID: \"cc56a994-26bb-4ca9-b0ee-81fa0a7b798c\") " pod="redhat-ods-monitoring/fake-gcs-server" Oct 30 04:00:38.510599 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:38.510564 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cc56a994-26bb-4ca9-b0ee-81fa0a7b798c-data\") pod \"fake-gcs-server\" (UID: \"cc56a994-26bb-4ca9-b0ee-81fa0a7b798c\") " pod="redhat-ods-monitoring/fake-gcs-server" Oct 30 04:00:38.510784 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:38.510662 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bsgw\" (UniqueName: \"kubernetes.io/projected/cc56a994-26bb-4ca9-b0ee-81fa0a7b798c-kube-api-access-8bsgw\") pod \"fake-gcs-server\" (UID: \"cc56a994-26bb-4ca9-b0ee-81fa0a7b798c\") " pod="redhat-ods-monitoring/fake-gcs-server" Oct 30 04:00:38.511040 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:38.511008 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cc56a994-26bb-4ca9-b0ee-81fa0a7b798c-data\") pod \"fake-gcs-server\" (UID: \"cc56a994-26bb-4ca9-b0ee-81fa0a7b798c\") " pod="redhat-ods-monitoring/fake-gcs-server" Oct 30 04:00:38.536280 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:38.536252 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bsgw\" (UniqueName: \"kubernetes.io/projected/cc56a994-26bb-4ca9-b0ee-81fa0a7b798c-kube-api-access-8bsgw\") pod \"fake-gcs-server\" (UID: \"cc56a994-26bb-4ca9-b0ee-81fa0a7b798c\") " pod="redhat-ods-monitoring/fake-gcs-server" Oct 30 04:00:38.652074 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:38.651994 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/fake-gcs-server" Oct 30 04:00:38.940786 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:38.940755 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/fake-gcs-server"] Oct 30 04:00:38.941488 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:00:38.941464 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc56a994_26bb_4ca9_b0ee_81fa0a7b798c.slice/crio-8937dc68a61f603c96fa6c0be609eb06cf0c231b4c9e8bc77d82621ae03b986f WatchSource:0}: Error finding container 8937dc68a61f603c96fa6c0be609eb06cf0c231b4c9e8bc77d82621ae03b986f: Status 404 returned error can't find the container with id 8937dc68a61f603c96fa6c0be609eb06cf0c231b4c9e8bc77d82621ae03b986f Oct 30 04:00:39.428476 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:39.428445 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/fake-gcs-server" event={"ID":"cc56a994-26bb-4ca9-b0ee-81fa0a7b798c","Type":"ContainerStarted","Data":"8937dc68a61f603c96fa6c0be609eb06cf0c231b4c9e8bc77d82621ae03b986f"} Oct 30 04:00:40.434543 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:40.434509 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/fake-gcs-server" event={"ID":"cc56a994-26bb-4ca9-b0ee-81fa0a7b798c","Type":"ContainerStarted","Data":"f4a73517c9fd42d48bbbf184a49d627acdbbfa580d061fb2968d4b6613df6a74"} Oct 30 04:00:40.435002 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:40.434641 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/fake-gcs-server" Oct 30 04:00:40.479357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:40.479265 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/fake-gcs-server" podStartSLOduration=1.224523287 podStartE2EDuration="2.479250661s" totalImagesPullingTime="1.254727374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:40.47891537 +0000 UTC m=+4752.593230344" watchObservedRunningTime="2026-10-30 04:00:40.479250661 +0000 UTC m=+4752.593565627" Oct 30 04:00:46.441778 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:46.441743 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/fake-gcs-server" Oct 30 04:00:53.942834 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:00:53.942787 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:00:54.529114 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:54.529081 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:00:54.535527 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:54.535489 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:54.549934 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:54.549903 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:00:54.579872 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:54.579841 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 30 04:00:54.651992 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:54.651958 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khp98\" (UniqueName: \"kubernetes.io/projected/d5d30c27-bf17-4684-a50f-98aebeb193dd-kube-api-access-khp98\") pod \"data-science-perses-0\" (UID: \"d5d30c27-bf17-4684-a50f-98aebeb193dd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:54.652121 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:54.652042 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/d5d30c27-bf17-4684-a50f-98aebeb193dd-plugins\") pod \"data-science-perses-0\" (UID: \"d5d30c27-bf17-4684-a50f-98aebeb193dd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:54.652121 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:54.652083 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5d30c27-bf17-4684-a50f-98aebeb193dd-config\") pod \"data-science-perses-0\" (UID: \"d5d30c27-bf17-4684-a50f-98aebeb193dd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:54.752793 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:54.752760 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khp98\" (UniqueName: \"kubernetes.io/projected/d5d30c27-bf17-4684-a50f-98aebeb193dd-kube-api-access-khp98\") pod \"data-science-perses-0\" (UID: \"d5d30c27-bf17-4684-a50f-98aebeb193dd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:54.752947 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:54.752818 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/d5d30c27-bf17-4684-a50f-98aebeb193dd-plugins\") pod \"data-science-perses-0\" (UID: \"d5d30c27-bf17-4684-a50f-98aebeb193dd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:54.752947 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:54.752846 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5d30c27-bf17-4684-a50f-98aebeb193dd-config\") pod \"data-science-perses-0\" (UID: \"d5d30c27-bf17-4684-a50f-98aebeb193dd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:54.753308 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:54.753285 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/d5d30c27-bf17-4684-a50f-98aebeb193dd-plugins\") pod \"data-science-perses-0\" (UID: \"d5d30c27-bf17-4684-a50f-98aebeb193dd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:54.753522 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:54.753505 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5d30c27-bf17-4684-a50f-98aebeb193dd-config\") pod \"data-science-perses-0\" (UID: \"d5d30c27-bf17-4684-a50f-98aebeb193dd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:54.777888 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:54.777853 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khp98\" (UniqueName: \"kubernetes.io/projected/d5d30c27-bf17-4684-a50f-98aebeb193dd-kube-api-access-khp98\") pod \"data-science-perses-0\" (UID: \"d5d30c27-bf17-4684-a50f-98aebeb193dd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:57.582004 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:57.581966 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"d5d30c27-bf17-4684-a50f-98aebeb193dd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:57.614634 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:57.614589 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"d5d30c27-bf17-4684-a50f-98aebeb193dd\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:57.683460 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:57.683416 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"d5d30c27-bf17-4684-a50f-98aebeb193dd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:57.683686 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:57.683564 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"d5d30c27-bf17-4684-a50f-98aebeb193dd\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:57.699594 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:57.699549 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"d5d30c27-bf17-4684-a50f-98aebeb193dd\") DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:57.803240 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:57.803209 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"d5d30c27-bf17-4684-a50f-98aebeb193dd\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2c422b71c1e0b8fdf34659f2cef7846e06b4353ed88027dc99ade5d7fc29e9d6/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:57.834532 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:57.834469 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"d5d30c27-bf17-4684-a50f-98aebeb193dd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:58.148035 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:58.147960 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:58.450332 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:58.450301 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:00:58.451037 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:00:58.451008 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5d30c27_bf17_4684_a50f_98aebeb193dd.slice/crio-9615cacf7ec56e40c0e44fbcc042763e7c0d047cc68eb5d38ceb5ce024b2291a WatchSource:0}: Error finding container 9615cacf7ec56e40c0e44fbcc042763e7c0d047cc68eb5d38ceb5ce024b2291a: Status 404 returned error can't find the container with id 9615cacf7ec56e40c0e44fbcc042763e7c0d047cc68eb5d38ceb5ce024b2291a Oct 30 04:00:58.515320 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:58.515283 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"d5d30c27-bf17-4684-a50f-98aebeb193dd","Type":"ContainerStarted","Data":"cd6061e574d15c0c2bb95e4b9e77fbf64c98ad8ed7532fcfac011ed2b7ba1f21"} Oct 30 04:00:58.515453 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:58.515335 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"d5d30c27-bf17-4684-a50f-98aebeb193dd","Type":"ContainerStarted","Data":"9615cacf7ec56e40c0e44fbcc042763e7c0d047cc68eb5d38ceb5ce024b2291a"} Oct 30 04:00:58.515453 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:58.515373 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:00:58.561985 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:00:58.561934 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=4.561921242 podStartE2EDuration="4.561921242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:58.560251744 +0000 UTC m=+4770.674566720" watchObservedRunningTime="2026-10-30 04:00:58.561921242 +0000 UTC m=+4770.676236216" Oct 30 04:01:09.522158 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:01:09.522079 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:01:23.942819 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:01:23.942783 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:01:28.471347 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:01:28.471238 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 04:01:28.487031 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:01:28.487007 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 04:01:28.972794 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:01:28.972761 5882 scope.go:122] "RemoveContainer" containerID="c44bc1ade1fb3b5535544e845aaca393d7a80d869872fc1eded73a65d1da35d2" Oct 30 04:01:28.980851 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:01:28.980829 5882 scope.go:122] "RemoveContainer" containerID="52efdbfa9decfe856bdbce472d292bd63135d01beaac4da2373a555804481bf9" Oct 30 04:01:28.988390 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:01:28.988366 5882 scope.go:122] "RemoveContainer" containerID="e2dbdc1bac6993055089bf572a3d07fec69bb8af6d944663059386db9caa643f" Oct 30 04:01:28.995930 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:01:28.995912 5882 scope.go:122] "RemoveContainer" containerID="5afcf24e9aae5ce2262bff3a5e89f1fe044621310fe58b32f7dc49fafb6dd0d2" Oct 30 04:01:29.002980 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:01:29.002960 5882 scope.go:122] "RemoveContainer" containerID="717d99a833370d31790639cb181e82d5b48c92d5562331b738fee5b56763b4ce" Oct 30 04:01:29.009798 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:01:29.009780 5882 scope.go:122] "RemoveContainer" containerID="3a67655c583a8baae360410848bb14196879bc8c5f2b3f55c1fc6fc4b37ac170" Oct 30 04:01:53.942792 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:01:53.942756 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:02:23.942495 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:02:23.942457 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:02:29.047156 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:29.047113 5882 scope.go:122] "RemoveContainer" containerID="4b1130e15ccc7f12e75147dc49d2c2fb79f2c9e33d30bfe202de9962a44201ab" Oct 30 04:02:29.055568 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:29.055542 5882 scope.go:122] "RemoveContainer" containerID="60b42b91f08d52fd5986bf71f3578614a0160faa38acab6956157d1879306686" Oct 30 04:02:29.062991 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:29.062969 5882 scope.go:122] "RemoveContainer" containerID="80ba224fb05e81ad14dab9704df102634d253b29f9b6acbc0a79a296c8687aeb" Oct 30 04:02:29.069772 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:29.069751 5882 scope.go:122] "RemoveContainer" containerID="e2eeca322707c3df3ce1e609bf00b170425816c143404bd107c8097590f6dc9d" Oct 30 04:02:43.038580 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.038542 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:02:43.038999 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.038789 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="d5d30c27-bf17-4684-a50f-98aebeb193dd" containerName="perses" containerID="cri-o://cd6061e574d15c0c2bb95e4b9e77fbf64c98ad8ed7532fcfac011ed2b7ba1f21" gracePeriod=30 Oct 30 04:02:43.301502 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.301437 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:43.355073 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.355042 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d30c27-bf17-4684-a50f-98aebeb193dd-plugins\" (UniqueName: \"kubernetes.io/empty-dir/d5d30c27-bf17-4684-a50f-98aebeb193dd-plugins\") pod \"d5d30c27-bf17-4684-a50f-98aebeb193dd\" (UID: \"d5d30c27-bf17-4684-a50f-98aebeb193dd\") " Oct 30 04:02:43.355247 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.355128 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5d30c27-bf17-4684-a50f-98aebeb193dd-config\" (UniqueName: \"kubernetes.io/configmap/d5d30c27-bf17-4684-a50f-98aebeb193dd-config\") pod \"d5d30c27-bf17-4684-a50f-98aebeb193dd\" (UID: \"d5d30c27-bf17-4684-a50f-98aebeb193dd\") " Oct 30 04:02:43.355247 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.355232 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"d5d30c27-bf17-4684-a50f-98aebeb193dd\" (UID: \"d5d30c27-bf17-4684-a50f-98aebeb193dd\") " Oct 30 04:02:43.355367 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.355312 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d30c27-bf17-4684-a50f-98aebeb193dd-kube-api-access-khp98\" (UniqueName: \"kubernetes.io/projected/d5d30c27-bf17-4684-a50f-98aebeb193dd-kube-api-access-khp98\") pod \"d5d30c27-bf17-4684-a50f-98aebeb193dd\" (UID: \"d5d30c27-bf17-4684-a50f-98aebeb193dd\") " Oct 30 04:02:43.355495 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.355468 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d30c27-bf17-4684-a50f-98aebeb193dd-config" pod "d5d30c27-bf17-4684-a50f-98aebeb193dd" (UID: "d5d30c27-bf17-4684-a50f-98aebeb193dd"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:02:43.355604 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.355590 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5d30c27-bf17-4684-a50f-98aebeb193dd-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:02:43.357703 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.357671 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d30c27-bf17-4684-a50f-98aebeb193dd-kube-api-access-khp98" pod "d5d30c27-bf17-4684-a50f-98aebeb193dd" (UID: "d5d30c27-bf17-4684-a50f-98aebeb193dd"). InnerVolumeSpecName "kube-api-access-khp98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:02:43.358138 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.358115 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e" pod "d5d30c27-bf17-4684-a50f-98aebeb193dd" (UID: "d5d30c27-bf17-4684-a50f-98aebeb193dd"). InnerVolumeSpecName "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:02:43.456817 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.456779 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:02:43.456817 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.456818 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khp98\" (UniqueName: \"kubernetes.io/projected/d5d30c27-bf17-4684-a50f-98aebeb193dd-kube-api-access-khp98\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:02:43.481892 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.481868 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:02:43.540712 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.540593 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d30c27-bf17-4684-a50f-98aebeb193dd-plugins" pod "d5d30c27-bf17-4684-a50f-98aebeb193dd" (UID: "d5d30c27-bf17-4684-a50f-98aebeb193dd"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:02:43.557277 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.557208 5882 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/d5d30c27-bf17-4684-a50f-98aebeb193dd-plugins\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:02:43.557277 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.557237 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" Oct 30 04:02:43.981311 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.981225 5882 generic.go:356] "Generic (PLEG): container finished" podID="d5d30c27-bf17-4684-a50f-98aebeb193dd" containerID="cd6061e574d15c0c2bb95e4b9e77fbf64c98ad8ed7532fcfac011ed2b7ba1f21" exitCode=0 Oct 30 04:02:43.981311 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.981291 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:43.981494 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.981308 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"d5d30c27-bf17-4684-a50f-98aebeb193dd","Type":"ContainerDied","Data":"cd6061e574d15c0c2bb95e4b9e77fbf64c98ad8ed7532fcfac011ed2b7ba1f21"} Oct 30 04:02:43.981494 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.981352 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"d5d30c27-bf17-4684-a50f-98aebeb193dd","Type":"ContainerDied","Data":"9615cacf7ec56e40c0e44fbcc042763e7c0d047cc68eb5d38ceb5ce024b2291a"} Oct 30 04:02:43.981494 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.981372 5882 scope.go:122] "RemoveContainer" containerID="cd6061e574d15c0c2bb95e4b9e77fbf64c98ad8ed7532fcfac011ed2b7ba1f21" Oct 30 04:02:43.990403 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.990386 5882 scope.go:122] "RemoveContainer" containerID="cd6061e574d15c0c2bb95e4b9e77fbf64c98ad8ed7532fcfac011ed2b7ba1f21" Oct 30 04:02:43.990694 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:02:43.990675 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd6061e574d15c0c2bb95e4b9e77fbf64c98ad8ed7532fcfac011ed2b7ba1f21\": container with ID starting with cd6061e574d15c0c2bb95e4b9e77fbf64c98ad8ed7532fcfac011ed2b7ba1f21 not found: ID does not exist" containerID="cd6061e574d15c0c2bb95e4b9e77fbf64c98ad8ed7532fcfac011ed2b7ba1f21" Oct 30 04:02:43.990791 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:43.990703 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6061e574d15c0c2bb95e4b9e77fbf64c98ad8ed7532fcfac011ed2b7ba1f21"} err="failed to get container status \"cd6061e574d15c0c2bb95e4b9e77fbf64c98ad8ed7532fcfac011ed2b7ba1f21\": rpc error: code = NotFound desc = could not find container \"cd6061e574d15c0c2bb95e4b9e77fbf64c98ad8ed7532fcfac011ed2b7ba1f21\": container with ID starting with cd6061e574d15c0c2bb95e4b9e77fbf64c98ad8ed7532fcfac011ed2b7ba1f21 not found: ID does not exist" Oct 30 04:02:44.039184 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:44.039153 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:02:44.053386 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:44.053358 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:02:44.083640 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:44.083602 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/fake-gcs-server"] Oct 30 04:02:44.083793 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:44.083773 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/fake-gcs-server" podUID="cc56a994-26bb-4ca9-b0ee-81fa0a7b798c" containerName="fake-gcs-server" containerID="cri-o://f4a73517c9fd42d48bbbf184a49d627acdbbfa580d061fb2968d4b6613df6a74" gracePeriod=30 Oct 30 04:02:44.324581 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:44.324555 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/fake-gcs-server" Oct 30 04:02:44.364445 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:44.364415 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc56a994-26bb-4ca9-b0ee-81fa0a7b798c-data\" (UniqueName: \"kubernetes.io/empty-dir/cc56a994-26bb-4ca9-b0ee-81fa0a7b798c-data\") pod \"cc56a994-26bb-4ca9-b0ee-81fa0a7b798c\" (UID: \"cc56a994-26bb-4ca9-b0ee-81fa0a7b798c\") " Oct 30 04:02:44.364660 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:44.364548 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc56a994-26bb-4ca9-b0ee-81fa0a7b798c-kube-api-access-8bsgw\" (UniqueName: \"kubernetes.io/projected/cc56a994-26bb-4ca9-b0ee-81fa0a7b798c-kube-api-access-8bsgw\") pod \"cc56a994-26bb-4ca9-b0ee-81fa0a7b798c\" (UID: \"cc56a994-26bb-4ca9-b0ee-81fa0a7b798c\") " Oct 30 04:02:44.364847 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:44.364819 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc56a994-26bb-4ca9-b0ee-81fa0a7b798c-data" pod "cc56a994-26bb-4ca9-b0ee-81fa0a7b798c" (UID: "cc56a994-26bb-4ca9-b0ee-81fa0a7b798c"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:02:44.365162 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:44.365142 5882 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cc56a994-26bb-4ca9-b0ee-81fa0a7b798c-data\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:02:44.367063 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:44.367038 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc56a994-26bb-4ca9-b0ee-81fa0a7b798c-kube-api-access-8bsgw" pod "cc56a994-26bb-4ca9-b0ee-81fa0a7b798c" (UID: "cc56a994-26bb-4ca9-b0ee-81fa0a7b798c"). InnerVolumeSpecName "kube-api-access-8bsgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:02:44.465883 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:44.465847 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bsgw\" (UniqueName: \"kubernetes.io/projected/cc56a994-26bb-4ca9-b0ee-81fa0a7b798c-kube-api-access-8bsgw\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:02:44.987093 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:44.987057 5882 generic.go:356] "Generic (PLEG): container finished" podID="cc56a994-26bb-4ca9-b0ee-81fa0a7b798c" containerID="f4a73517c9fd42d48bbbf184a49d627acdbbfa580d061fb2968d4b6613df6a74" exitCode=0 Oct 30 04:02:44.987273 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:44.987107 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/fake-gcs-server" Oct 30 04:02:44.987273 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:44.987122 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/fake-gcs-server" event={"ID":"cc56a994-26bb-4ca9-b0ee-81fa0a7b798c","Type":"ContainerDied","Data":"f4a73517c9fd42d48bbbf184a49d627acdbbfa580d061fb2968d4b6613df6a74"} Oct 30 04:02:44.987273 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:44.987150 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/fake-gcs-server" event={"ID":"cc56a994-26bb-4ca9-b0ee-81fa0a7b798c","Type":"ContainerDied","Data":"8937dc68a61f603c96fa6c0be609eb06cf0c231b4c9e8bc77d82621ae03b986f"} Oct 30 04:02:44.987273 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:44.987166 5882 scope.go:122] "RemoveContainer" containerID="f4a73517c9fd42d48bbbf184a49d627acdbbfa580d061fb2968d4b6613df6a74" Oct 30 04:02:44.995957 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:44.995941 5882 scope.go:122] "RemoveContainer" containerID="f4a73517c9fd42d48bbbf184a49d627acdbbfa580d061fb2968d4b6613df6a74" Oct 30 04:02:44.996205 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:02:44.996188 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4a73517c9fd42d48bbbf184a49d627acdbbfa580d061fb2968d4b6613df6a74\": container with ID starting with f4a73517c9fd42d48bbbf184a49d627acdbbfa580d061fb2968d4b6613df6a74 not found: ID does not exist" containerID="f4a73517c9fd42d48bbbf184a49d627acdbbfa580d061fb2968d4b6613df6a74" Oct 30 04:02:44.996249 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:44.996211 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a73517c9fd42d48bbbf184a49d627acdbbfa580d061fb2968d4b6613df6a74"} err="failed to get container status \"f4a73517c9fd42d48bbbf184a49d627acdbbfa580d061fb2968d4b6613df6a74\": rpc error: code = NotFound desc = could not find container \"f4a73517c9fd42d48bbbf184a49d627acdbbfa580d061fb2968d4b6613df6a74\": container with ID starting with f4a73517c9fd42d48bbbf184a49d627acdbbfa580d061fb2968d4b6613df6a74 not found: ID does not exist" Oct 30 04:02:45.042648 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:45.042605 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/fake-gcs-server"] Oct 30 04:02:45.046789 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:45.046769 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/fake-gcs-server"] Oct 30 04:02:46.009103 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:46.009068 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc56a994-26bb-4ca9-b0ee-81fa0a7b798c" path="/var/lib/kubelet/pods/cc56a994-26bb-4ca9-b0ee-81fa0a7b798c/volumes" Oct 30 04:02:46.009471 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:46.009459 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5d30c27-bf17-4684-a50f-98aebeb193dd" path="/var/lib/kubelet/pods/d5d30c27-bf17-4684-a50f-98aebeb193dd/volumes" Oct 30 04:02:48.220567 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.220532 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:02:48.221068 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.221049 5882 cpu_manager.go:414] "containerMap: removing container" podUID="d5d30c27-bf17-4684-a50f-98aebeb193dd" containerName="perses" Oct 30 04:02:48.221136 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.221070 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d30c27-bf17-4684-a50f-98aebeb193dd" containerName="perses" Oct 30 04:02:48.221136 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.221107 5882 cpu_manager.go:414] "containerMap: removing container" podUID="cc56a994-26bb-4ca9-b0ee-81fa0a7b798c" containerName="fake-gcs-server" Oct 30 04:02:48.221136 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.221116 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc56a994-26bb-4ca9-b0ee-81fa0a7b798c" containerName="fake-gcs-server" Oct 30 04:02:48.221283 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.221209 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d30c27-bf17-4684-a50f-98aebeb193dd" containerName="perses" Oct 30 04:02:48.221283 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.221225 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc56a994-26bb-4ca9-b0ee-81fa0a7b798c" containerName="fake-gcs-server" Oct 30 04:02:48.225444 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.225421 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:48.243741 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.243718 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:02:48.257635 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.257572 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 04:02:48.257635 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.257584 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 04:02:48.257817 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.257744 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 04:02:48.257817 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.257748 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 30 04:02:48.397730 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.397694 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcdg2\" (UniqueName: \"kubernetes.io/projected/75816a3b-0d3b-4f73-af12-4052bcc742fb-kube-api-access-jcdg2\") pod \"data-science-perses-0\" (UID: \"75816a3b-0d3b-4f73-af12-4052bcc742fb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:48.397902 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.397738 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/75816a3b-0d3b-4f73-af12-4052bcc742fb-plugins\") pod \"data-science-perses-0\" (UID: \"75816a3b-0d3b-4f73-af12-4052bcc742fb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:48.397902 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.397881 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/75816a3b-0d3b-4f73-af12-4052bcc742fb-config\") pod \"data-science-perses-0\" (UID: \"75816a3b-0d3b-4f73-af12-4052bcc742fb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:48.499232 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.499200 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcdg2\" (UniqueName: \"kubernetes.io/projected/75816a3b-0d3b-4f73-af12-4052bcc742fb-kube-api-access-jcdg2\") pod \"data-science-perses-0\" (UID: \"75816a3b-0d3b-4f73-af12-4052bcc742fb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:48.499395 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.499345 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/75816a3b-0d3b-4f73-af12-4052bcc742fb-plugins\") pod \"data-science-perses-0\" (UID: \"75816a3b-0d3b-4f73-af12-4052bcc742fb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:48.499488 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.499466 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/75816a3b-0d3b-4f73-af12-4052bcc742fb-config\") pod \"data-science-perses-0\" (UID: \"75816a3b-0d3b-4f73-af12-4052bcc742fb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:48.499771 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.499749 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/75816a3b-0d3b-4f73-af12-4052bcc742fb-plugins\") pod \"data-science-perses-0\" (UID: \"75816a3b-0d3b-4f73-af12-4052bcc742fb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:48.500062 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.500043 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/75816a3b-0d3b-4f73-af12-4052bcc742fb-config\") pod \"data-science-perses-0\" (UID: \"75816a3b-0d3b-4f73-af12-4052bcc742fb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:48.524603 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:48.524573 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcdg2\" (UniqueName: \"kubernetes.io/projected/75816a3b-0d3b-4f73-af12-4052bcc742fb-kube-api-access-jcdg2\") pod \"data-science-perses-0\" (UID: \"75816a3b-0d3b-4f73-af12-4052bcc742fb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:50.415145 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:50.415091 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"75816a3b-0d3b-4f73-af12-4052bcc742fb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:50.462589 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:50.462551 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"75816a3b-0d3b-4f73-af12-4052bcc742fb\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:50.515907 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:50.515858 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"75816a3b-0d3b-4f73-af12-4052bcc742fb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:50.516083 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:50.516008 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"75816a3b-0d3b-4f73-af12-4052bcc742fb\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:50.532348 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:50.532326 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"75816a3b-0d3b-4f73-af12-4052bcc742fb\") DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:50.636784 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:50.636757 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"75816a3b-0d3b-4f73-af12-4052bcc742fb\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2c422b71c1e0b8fdf34659f2cef7846e06b4353ed88027dc99ade5d7fc29e9d6/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:50.667906 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:50.667848 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"75816a3b-0d3b-4f73-af12-4052bcc742fb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:50.949508 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:50.949421 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:51.239650 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:51.239600 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:02:51.240432 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:02:51.240402 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75816a3b_0d3b_4f73_af12_4052bcc742fb.slice/crio-ceb8436312d600baaf30042c97ab0e700d9ce6002d471ed9fdf1a14b21a38342 WatchSource:0}: Error finding container ceb8436312d600baaf30042c97ab0e700d9ce6002d471ed9fdf1a14b21a38342: Status 404 returned error can't find the container with id ceb8436312d600baaf30042c97ab0e700d9ce6002d471ed9fdf1a14b21a38342 Oct 30 04:02:52.021220 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:52.021172 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"75816a3b-0d3b-4f73-af12-4052bcc742fb","Type":"ContainerStarted","Data":"70a4eb3b21f4e22ff1ced6c2ab47d4f7d038b9d17bc356c32412d736b2ccf92b"} Oct 30 04:02:52.021637 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:52.021233 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"75816a3b-0d3b-4f73-af12-4052bcc742fb","Type":"ContainerStarted","Data":"ceb8436312d600baaf30042c97ab0e700d9ce6002d471ed9fdf1a14b21a38342"} Oct 30 04:02:52.021637 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:52.021313 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:02:52.065484 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:02:52.065430 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=4.06541712 podStartE2EDuration="4.06541712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:02:52.065138873 +0000 UTC m=+4884.179453860" watchObservedRunningTime="2026-10-30 04:02:52.06541712 +0000 UTC m=+4884.179732092" Oct 30 04:02:53.942520 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:02:53.942481 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:03:03.029671 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:03.029641 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:05.875394 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:05.875356 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:03:05.875885 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:05.875519 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="75816a3b-0d3b-4f73-af12-4052bcc742fb" containerName="perses" containerID="cri-o://70a4eb3b21f4e22ff1ced6c2ab47d4f7d038b9d17bc356c32412d736b2ccf92b" gracePeriod=30 Oct 30 04:03:06.080458 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:06.080421 5882 generic.go:356] "Generic (PLEG): container finished" podID="75816a3b-0d3b-4f73-af12-4052bcc742fb" containerID="70a4eb3b21f4e22ff1ced6c2ab47d4f7d038b9d17bc356c32412d736b2ccf92b" exitCode=0 Oct 30 04:03:06.080636 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:06.080473 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"75816a3b-0d3b-4f73-af12-4052bcc742fb","Type":"ContainerDied","Data":"70a4eb3b21f4e22ff1ced6c2ab47d4f7d038b9d17bc356c32412d736b2ccf92b"} Oct 30 04:03:06.126245 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:06.126190 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:06.153170 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:06.153140 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"75816a3b-0d3b-4f73-af12-4052bcc742fb\" (UID: \"75816a3b-0d3b-4f73-af12-4052bcc742fb\") " Oct 30 04:03:06.153311 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:06.153201 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75816a3b-0d3b-4f73-af12-4052bcc742fb-kube-api-access-jcdg2\" (UniqueName: \"kubernetes.io/projected/75816a3b-0d3b-4f73-af12-4052bcc742fb-kube-api-access-jcdg2\") pod \"75816a3b-0d3b-4f73-af12-4052bcc742fb\" (UID: \"75816a3b-0d3b-4f73-af12-4052bcc742fb\") " Oct 30 04:03:06.153311 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:06.153233 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75816a3b-0d3b-4f73-af12-4052bcc742fb-plugins\" (UniqueName: \"kubernetes.io/empty-dir/75816a3b-0d3b-4f73-af12-4052bcc742fb-plugins\") pod \"75816a3b-0d3b-4f73-af12-4052bcc742fb\" (UID: \"75816a3b-0d3b-4f73-af12-4052bcc742fb\") " Oct 30 04:03:06.153311 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:06.153283 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75816a3b-0d3b-4f73-af12-4052bcc742fb-config\" (UniqueName: \"kubernetes.io/configmap/75816a3b-0d3b-4f73-af12-4052bcc742fb-config\") pod \"75816a3b-0d3b-4f73-af12-4052bcc742fb\" (UID: \"75816a3b-0d3b-4f73-af12-4052bcc742fb\") " Oct 30 04:03:06.153734 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:06.153698 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75816a3b-0d3b-4f73-af12-4052bcc742fb-config" pod "75816a3b-0d3b-4f73-af12-4052bcc742fb" (UID: "75816a3b-0d3b-4f73-af12-4052bcc742fb"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:03:06.155461 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:06.155438 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75816a3b-0d3b-4f73-af12-4052bcc742fb-kube-api-access-jcdg2" pod "75816a3b-0d3b-4f73-af12-4052bcc742fb" (UID: "75816a3b-0d3b-4f73-af12-4052bcc742fb"). InnerVolumeSpecName "kube-api-access-jcdg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:03:06.155819 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:06.155795 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e" pod "75816a3b-0d3b-4f73-af12-4052bcc742fb" (UID: "75816a3b-0d3b-4f73-af12-4052bcc742fb"). InnerVolumeSpecName "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:03:06.258747 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:06.258414 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcdg2\" (UniqueName: \"kubernetes.io/projected/75816a3b-0d3b-4f73-af12-4052bcc742fb-kube-api-access-jcdg2\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:03:06.258747 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:06.258456 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/75816a3b-0d3b-4f73-af12-4052bcc742fb-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:03:06.258747 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:06.258502 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:03:06.278176 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:06.278148 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:03:06.317150 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:06.317111 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75816a3b-0d3b-4f73-af12-4052bcc742fb-plugins" pod "75816a3b-0d3b-4f73-af12-4052bcc742fb" (UID: "75816a3b-0d3b-4f73-af12-4052bcc742fb"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:03:06.359425 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:06.359390 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" Oct 30 04:03:06.359576 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:06.359441 5882 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/75816a3b-0d3b-4f73-af12-4052bcc742fb-plugins\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:03:07.085717 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:07.085677 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"75816a3b-0d3b-4f73-af12-4052bcc742fb","Type":"ContainerDied","Data":"ceb8436312d600baaf30042c97ab0e700d9ce6002d471ed9fdf1a14b21a38342"} Oct 30 04:03:07.086128 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:07.085739 5882 scope.go:122] "RemoveContainer" containerID="70a4eb3b21f4e22ff1ced6c2ab47d4f7d038b9d17bc356c32412d736b2ccf92b" Oct 30 04:03:07.086128 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:07.085698 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:07.141354 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:07.141324 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:03:07.155218 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:07.155192 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:03:08.009847 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:08.009818 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75816a3b-0d3b-4f73-af12-4052bcc742fb" path="/var/lib/kubelet/pods/75816a3b-0d3b-4f73-af12-4052bcc742fb/volumes" Oct 30 04:03:10.752956 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:10.752924 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l"] Oct 30 04:03:10.753532 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:10.753461 5882 cpu_manager.go:414] "containerMap: removing container" podUID="75816a3b-0d3b-4f73-af12-4052bcc742fb" containerName="perses" Oct 30 04:03:10.753532 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:10.753481 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75816a3b-0d3b-4f73-af12-4052bcc742fb" containerName="perses" Oct 30 04:03:10.753689 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:10.753559 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="75816a3b-0d3b-4f73-af12-4052bcc742fb" containerName="perses" Oct 30 04:03:10.760641 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:10.760601 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l" Oct 30 04:03:10.777805 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:10.777782 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 04:03:10.777805 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:10.777798 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 04:03:10.778010 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:10.777985 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 04:03:10.778266 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:10.778244 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l"] Oct 30 04:03:10.798841 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:10.798819 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4cbq\" (UniqueName: \"kubernetes.io/projected/51a2c83e-1fd8-4a9c-81c7-95d6dd672e32-kube-api-access-x4cbq\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l\" (UID: \"51a2c83e-1fd8-4a9c-81c7-95d6dd672e32\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l" Oct 30 04:03:10.900227 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:10.900196 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4cbq\" (UniqueName: \"kubernetes.io/projected/51a2c83e-1fd8-4a9c-81c7-95d6dd672e32-kube-api-access-x4cbq\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l\" (UID: \"51a2c83e-1fd8-4a9c-81c7-95d6dd672e32\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l" Oct 30 04:03:10.925688 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:10.925655 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4cbq\" (UniqueName: \"kubernetes.io/projected/51a2c83e-1fd8-4a9c-81c7-95d6dd672e32-kube-api-access-x4cbq\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l\" (UID: \"51a2c83e-1fd8-4a9c-81c7-95d6dd672e32\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l" Oct 30 04:03:11.084434 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:11.084357 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l" Oct 30 04:03:11.373436 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:11.373406 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l"] Oct 30 04:03:11.374259 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:03:11.374231 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51a2c83e_1fd8_4a9c_81c7_95d6dd672e32.slice/crio-e9a8b214ae49a57fe6d86d4e13004dec5b7ef5ad41cbcc919a50a33c01adcca4 WatchSource:0}: Error finding container e9a8b214ae49a57fe6d86d4e13004dec5b7ef5ad41cbcc919a50a33c01adcca4: Status 404 returned error can't find the container with id e9a8b214ae49a57fe6d86d4e13004dec5b7ef5ad41cbcc919a50a33c01adcca4 Oct 30 04:03:11.831770 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:11.831739 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:03:11.835111 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:11.835090 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:11.858958 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:11.858928 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:03:11.906086 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:11.906054 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 30 04:03:11.926686 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:11.921343 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9"] Oct 30 04:03:11.928954 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:11.928931 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" Oct 30 04:03:11.944223 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:11.944197 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9"] Oct 30 04:03:11.947676 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:11.947651 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-lt7qd\"" Oct 30 04:03:11.947803 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:11.947727 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 30 04:03:12.006859 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.006824 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 30 04:03:12.011284 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.011258 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/555eacfa-6f9e-4efd-8fec-8b38c8c50283-config\") pod \"data-science-perses-0\" (UID: \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:12.011416 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.011314 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcb42\" (UniqueName: \"kubernetes.io/projected/555eacfa-6f9e-4efd-8fec-8b38c8c50283-kube-api-access-rcb42\") pod \"data-science-perses-0\" (UID: \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:12.011416 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.011405 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/555eacfa-6f9e-4efd-8fec-8b38c8c50283-plugins\") pod \"data-science-perses-0\" (UID: \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:12.108760 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.108680 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l" event={"ID":"51a2c83e-1fd8-4a9c-81c7-95d6dd672e32","Type":"ContainerStarted","Data":"e7d5e0f77212b66d29010e5052e26178d45ceeaf37adde40b5570fafe4170cda"} Oct 30 04:03:12.108760 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.108725 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l" event={"ID":"51a2c83e-1fd8-4a9c-81c7-95d6dd672e32","Type":"ContainerStarted","Data":"e9a8b214ae49a57fe6d86d4e13004dec5b7ef5ad41cbcc919a50a33c01adcca4"} Oct 30 04:03:12.112210 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.112186 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/555eacfa-6f9e-4efd-8fec-8b38c8c50283-plugins\") pod \"data-science-perses-0\" (UID: \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:12.112272 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.112236 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q22hj\" (UniqueName: \"kubernetes.io/projected/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-kube-api-access-q22hj\") pod \"data-science-collector-targetallocator-74d89c7b7b-hc6s9\" (UID: \"09e2fa8d-a3bd-41fc-9820-cb4e52dcf810\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" Oct 30 04:03:12.112316 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.112298 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/555eacfa-6f9e-4efd-8fec-8b38c8c50283-config\") pod \"data-science-perses-0\" (UID: \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:12.112352 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.112329 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-74d89c7b7b-hc6s9\" (UID: \"09e2fa8d-a3bd-41fc-9820-cb4e52dcf810\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" Oct 30 04:03:12.112557 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.112535 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcb42\" (UniqueName: \"kubernetes.io/projected/555eacfa-6f9e-4efd-8fec-8b38c8c50283-kube-api-access-rcb42\") pod \"data-science-perses-0\" (UID: \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:12.112601 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.112581 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-ta-internal\") pod \"data-science-collector-targetallocator-74d89c7b7b-hc6s9\" (UID: \"09e2fa8d-a3bd-41fc-9820-cb4e52dcf810\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" Oct 30 04:03:12.112662 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.112599 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/555eacfa-6f9e-4efd-8fec-8b38c8c50283-plugins\") pod \"data-science-perses-0\" (UID: \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:12.113004 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.112987 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/555eacfa-6f9e-4efd-8fec-8b38c8c50283-config\") pod \"data-science-perses-0\" (UID: \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:12.138483 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.138459 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcb42\" (UniqueName: \"kubernetes.io/projected/555eacfa-6f9e-4efd-8fec-8b38c8c50283-kube-api-access-rcb42\") pod \"data-science-perses-0\" (UID: \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:12.153059 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.153013 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l" podStartSLOduration=2.152997465 podStartE2EDuration="2.152997465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:12.152359746 +0000 UTC m=+4904.266674725" watchObservedRunningTime="2026-10-30 04:03:12.152997465 +0000 UTC m=+4904.267312441" Oct 30 04:03:12.213493 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.213448 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-ta-internal\") pod \"data-science-collector-targetallocator-74d89c7b7b-hc6s9\" (UID: \"09e2fa8d-a3bd-41fc-9820-cb4e52dcf810\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" Oct 30 04:03:12.213686 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.213553 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q22hj\" (UniqueName: \"kubernetes.io/projected/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-kube-api-access-q22hj\") pod \"data-science-collector-targetallocator-74d89c7b7b-hc6s9\" (UID: \"09e2fa8d-a3bd-41fc-9820-cb4e52dcf810\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" Oct 30 04:03:12.213686 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.213655 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-74d89c7b7b-hc6s9\" (UID: \"09e2fa8d-a3bd-41fc-9820-cb4e52dcf810\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" Oct 30 04:03:12.214137 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.214108 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-ta-internal\") pod \"data-science-collector-targetallocator-74d89c7b7b-hc6s9\" (UID: \"09e2fa8d-a3bd-41fc-9820-cb4e52dcf810\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" Oct 30 04:03:12.216031 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.216009 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-74d89c7b7b-hc6s9\" (UID: \"09e2fa8d-a3bd-41fc-9820-cb4e52dcf810\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" Oct 30 04:03:12.239022 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.238993 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q22hj\" (UniqueName: \"kubernetes.io/projected/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-kube-api-access-q22hj\") pod \"data-science-collector-targetallocator-74d89c7b7b-hc6s9\" (UID: \"09e2fa8d-a3bd-41fc-9820-cb4e52dcf810\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" Oct 30 04:03:12.240806 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.240778 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" Oct 30 04:03:12.533428 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.533397 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9"] Oct 30 04:03:12.534139 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:03:12.534117 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09e2fa8d_a3bd_41fc_9820_cb4e52dcf810.slice/crio-30631222dec6e83b49660047401836e8a07e2e80c6956461304ded00acae6c40 WatchSource:0}: Error finding container 30631222dec6e83b49660047401836e8a07e2e80c6956461304ded00acae6c40: Status 404 returned error can't find the container with id 30631222dec6e83b49660047401836e8a07e2e80c6956461304ded00acae6c40 Oct 30 04:03:12.536525 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.536484 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:03:12.536609 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:12.536544 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:03:13.114343 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:13.114307 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" event={"ID":"09e2fa8d-a3bd-41fc-9820-cb4e52dcf810","Type":"ContainerStarted","Data":"c7a08305fd356598793240d29e870f49e27185d34fc3f928f21f8022aa306194"} Oct 30 04:03:13.114767 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:13.114427 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" Oct 30 04:03:13.114767 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:13.114444 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" event={"ID":"09e2fa8d-a3bd-41fc-9820-cb4e52dcf810","Type":"ContainerStarted","Data":"30631222dec6e83b49660047401836e8a07e2e80c6956461304ded00acae6c40"} Oct 30 04:03:13.116283 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:13.116260 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" Oct 30 04:03:13.159672 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:13.159605 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" podStartSLOduration=2.159591365 podStartE2EDuration="2.159591365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:13.158304422 +0000 UTC m=+4905.272619423" watchObservedRunningTime="2026-10-30 04:03:13.159591365 +0000 UTC m=+4905.273906340" Oct 30 04:03:14.569964 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:14.569926 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7"] Oct 30 04:03:14.573463 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:14.573441 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7" Oct 30 04:03:14.592844 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:14.592816 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7"] Oct 30 04:03:14.630962 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:14.630932 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n965d\" (UniqueName: \"kubernetes.io/projected/f4dc26eb-1fe7-451e-aa4b-5380b08875ee-kube-api-access-n965d\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7\" (UID: \"f4dc26eb-1fe7-451e-aa4b-5380b08875ee\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7" Oct 30 04:03:14.731420 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:14.731374 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n965d\" (UniqueName: \"kubernetes.io/projected/f4dc26eb-1fe7-451e-aa4b-5380b08875ee-kube-api-access-n965d\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7\" (UID: \"f4dc26eb-1fe7-451e-aa4b-5380b08875ee\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7" Oct 30 04:03:14.756643 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:14.756590 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n965d\" (UniqueName: \"kubernetes.io/projected/f4dc26eb-1fe7-451e-aa4b-5380b08875ee-kube-api-access-n965d\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7\" (UID: \"f4dc26eb-1fe7-451e-aa4b-5380b08875ee\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7" Oct 30 04:03:14.885191 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:14.885103 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7" Oct 30 04:03:15.174609 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.174571 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7"] Oct 30 04:03:15.175073 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:03:15.175042 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4dc26eb_1fe7_451e_aa4b_5380b08875ee.slice/crio-5df0676ea2cdfd6c45b8392b23a73857c5170651d9de3d97faf39f3de4460f94 WatchSource:0}: Error finding container 5df0676ea2cdfd6c45b8392b23a73857c5170651d9de3d97faf39f3de4460f94: Status 404 returned error can't find the container with id 5df0676ea2cdfd6c45b8392b23a73857c5170651d9de3d97faf39f3de4460f94 Oct 30 04:03:15.424195 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.424107 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:03:15.428206 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.428179 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.436235 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.436207 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.436368 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.436273 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.436368 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.436348 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05dd6cb8-8908-4609-937d-80bb026a7937-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.436453 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.436380 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvk8s\" (UniqueName: \"kubernetes.io/projected/05dd6cb8-8908-4609-937d-80bb026a7937-kube-api-access-zvk8s\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.436453 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.436402 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05dd6cb8-8908-4609-937d-80bb026a7937-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.436541 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.436515 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/05dd6cb8-8908-4609-937d-80bb026a7937-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.436590 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.436565 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.452047 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.452018 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:03:15.517859 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.517818 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 04:03:15.518036 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.517984 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 04:03:15.518036 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.517987 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:03:15.518036 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.517989 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 04:03:15.518159 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.518039 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-hnkwv\"" Oct 30 04:03:15.537535 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.537505 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05dd6cb8-8908-4609-937d-80bb026a7937-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.537743 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.537545 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvk8s\" (UniqueName: \"kubernetes.io/projected/05dd6cb8-8908-4609-937d-80bb026a7937-kube-api-access-zvk8s\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.537743 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.537568 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05dd6cb8-8908-4609-937d-80bb026a7937-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.537743 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.537603 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/05dd6cb8-8908-4609-937d-80bb026a7937-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.537743 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.537657 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.537743 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.537685 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.537743 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.537729 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.538207 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.538173 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/05dd6cb8-8908-4609-937d-80bb026a7937-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.539977 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.539952 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05dd6cb8-8908-4609-937d-80bb026a7937-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.540163 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.540137 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.540255 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.540193 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.540437 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.540416 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05dd6cb8-8908-4609-937d-80bb026a7937-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.540437 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.540420 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.562320 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.562293 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvk8s\" (UniqueName: \"kubernetes.io/projected/05dd6cb8-8908-4609-937d-80bb026a7937-kube-api-access-zvk8s\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:15.748683 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:15.748648 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:16.063174 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:16.063143 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:03:16.063915 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:03:16.063884 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05dd6cb8_8908_4609_937d_80bb026a7937.slice/crio-4fb7a461e30c88137dfca19583a7533570c1e976d15749eec09d8eedae107656 WatchSource:0}: Error finding container 4fb7a461e30c88137dfca19583a7533570c1e976d15749eec09d8eedae107656: Status 404 returned error can't find the container with id 4fb7a461e30c88137dfca19583a7533570c1e976d15749eec09d8eedae107656 Oct 30 04:03:16.128837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:16.128804 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7" event={"ID":"f4dc26eb-1fe7-451e-aa4b-5380b08875ee","Type":"ContainerStarted","Data":"d47c9da69add40f72bd5bd52581bc941486d1b9f64e0fa2d4b6716a564ec4e12"} Oct 30 04:03:16.128977 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:16.128853 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7" event={"ID":"f4dc26eb-1fe7-451e-aa4b-5380b08875ee","Type":"ContainerStarted","Data":"5df0676ea2cdfd6c45b8392b23a73857c5170651d9de3d97faf39f3de4460f94"} Oct 30 04:03:16.129926 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:16.129903 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"05dd6cb8-8908-4609-937d-80bb026a7937","Type":"ContainerStarted","Data":"4fb7a461e30c88137dfca19583a7533570c1e976d15749eec09d8eedae107656"} Oct 30 04:03:16.171285 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:16.171234 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7" podStartSLOduration=2.171220941 podStartE2EDuration="2.171220941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:16.170387056 +0000 UTC m=+4908.284702030" watchObservedRunningTime="2026-10-30 04:03:16.171220941 +0000 UTC m=+4908.285535983" Oct 30 04:03:16.197406 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:16.197374 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l"] Oct 30 04:03:16.197571 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:16.197544 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l" podUID="51a2c83e-1fd8-4a9c-81c7-95d6dd672e32" containerName="thanos-querier" containerID="cri-o://e7d5e0f77212b66d29010e5052e26178d45ceeaf37adde40b5570fafe4170cda" gracePeriod=30 Oct 30 04:03:16.436018 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:16.435992 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l" Oct 30 04:03:16.446213 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:16.446191 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a2c83e-1fd8-4a9c-81c7-95d6dd672e32-kube-api-access-x4cbq\" (UniqueName: \"kubernetes.io/projected/51a2c83e-1fd8-4a9c-81c7-95d6dd672e32-kube-api-access-x4cbq\") pod \"51a2c83e-1fd8-4a9c-81c7-95d6dd672e32\" (UID: \"51a2c83e-1fd8-4a9c-81c7-95d6dd672e32\") " Oct 30 04:03:16.448129 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:16.448106 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a2c83e-1fd8-4a9c-81c7-95d6dd672e32-kube-api-access-x4cbq" pod "51a2c83e-1fd8-4a9c-81c7-95d6dd672e32" (UID: "51a2c83e-1fd8-4a9c-81c7-95d6dd672e32"). InnerVolumeSpecName "kube-api-access-x4cbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:03:16.547588 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:16.547558 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4cbq\" (UniqueName: \"kubernetes.io/projected/51a2c83e-1fd8-4a9c-81c7-95d6dd672e32-kube-api-access-x4cbq\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:03:17.135537 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:17.135499 5882 generic.go:356] "Generic (PLEG): container finished" podID="51a2c83e-1fd8-4a9c-81c7-95d6dd672e32" containerID="e7d5e0f77212b66d29010e5052e26178d45ceeaf37adde40b5570fafe4170cda" exitCode=0 Oct 30 04:03:17.135987 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:17.135546 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l" Oct 30 04:03:17.135987 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:17.135589 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l" event={"ID":"51a2c83e-1fd8-4a9c-81c7-95d6dd672e32","Type":"ContainerDied","Data":"e7d5e0f77212b66d29010e5052e26178d45ceeaf37adde40b5570fafe4170cda"} Oct 30 04:03:17.135987 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:17.135646 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l" event={"ID":"51a2c83e-1fd8-4a9c-81c7-95d6dd672e32","Type":"ContainerDied","Data":"e9a8b214ae49a57fe6d86d4e13004dec5b7ef5ad41cbcc919a50a33c01adcca4"} Oct 30 04:03:17.135987 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:17.135664 5882 scope.go:122] "RemoveContainer" containerID="e7d5e0f77212b66d29010e5052e26178d45ceeaf37adde40b5570fafe4170cda" Oct 30 04:03:17.137309 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:17.137281 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"05dd6cb8-8908-4609-937d-80bb026a7937","Type":"ContainerStarted","Data":"1db668d90a12f794e32357fb5a0e1402633a4f13679b7e6132d5e60a9c2df932"} Oct 30 04:03:17.145029 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:17.144804 5882 scope.go:122] "RemoveContainer" containerID="e7d5e0f77212b66d29010e5052e26178d45ceeaf37adde40b5570fafe4170cda" Oct 30 04:03:17.145106 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:03:17.145086 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7d5e0f77212b66d29010e5052e26178d45ceeaf37adde40b5570fafe4170cda\": container with ID starting with e7d5e0f77212b66d29010e5052e26178d45ceeaf37adde40b5570fafe4170cda not found: ID does not exist" containerID="e7d5e0f77212b66d29010e5052e26178d45ceeaf37adde40b5570fafe4170cda" Oct 30 04:03:17.145184 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:17.145166 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7d5e0f77212b66d29010e5052e26178d45ceeaf37adde40b5570fafe4170cda"} err="failed to get container status \"e7d5e0f77212b66d29010e5052e26178d45ceeaf37adde40b5570fafe4170cda\": rpc error: code = NotFound desc = could not find container \"e7d5e0f77212b66d29010e5052e26178d45ceeaf37adde40b5570fafe4170cda\": container with ID starting with e7d5e0f77212b66d29010e5052e26178d45ceeaf37adde40b5570fafe4170cda not found: ID does not exist" Oct 30 04:03:17.239935 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:17.239904 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l"] Oct 30 04:03:17.246120 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:17.246090 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-qkw2l"] Oct 30 04:03:18.008892 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:18.008861 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51a2c83e-1fd8-4a9c-81c7-95d6dd672e32" path="/var/lib/kubelet/pods/51a2c83e-1fd8-4a9c-81c7-95d6dd672e32/volumes" Oct 30 04:03:20.152333 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:20.152248 5882 generic.go:356] "Generic (PLEG): container finished" podID="05dd6cb8-8908-4609-937d-80bb026a7937" containerID="1db668d90a12f794e32357fb5a0e1402633a4f13679b7e6132d5e60a9c2df932" exitCode=0 Oct 30 04:03:20.152757 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:20.152324 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"05dd6cb8-8908-4609-937d-80bb026a7937","Type":"ContainerDied","Data":"1db668d90a12f794e32357fb5a0e1402633a4f13679b7e6132d5e60a9c2df932"} Oct 30 04:03:20.585086 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:20.585050 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:20.617864 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:20.617839 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:20.686072 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:20.686044 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:20.686197 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:20.686148 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:20.702244 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:20.702222 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\") DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:20.806396 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:20.806365 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2c422b71c1e0b8fdf34659f2cef7846e06b4353ed88027dc99ade5d7fc29e9d6/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:20.838042 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:20.837987 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:21.148668 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:21.148557 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:21.158599 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:21.158570 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"05dd6cb8-8908-4609-937d-80bb026a7937","Type":"ContainerStarted","Data":"1caf7eb6375cba7f47288ef8c74cd0ac29431c00f325e7018ba367429fd4735a"} Oct 30 04:03:21.158926 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:21.158665 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"05dd6cb8-8908-4609-937d-80bb026a7937","Type":"ContainerStarted","Data":"6f4d2d5a34244e9647294b45608893bda58876c135e5545e3a851212e49b3f2a"} Oct 30 04:03:21.158926 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:21.158884 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:21.211470 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:21.210709 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.823364578 podStartE2EDuration="6.210689062s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.387324484s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:21.209550406 +0000 UTC m=+4913.323865394" watchObservedRunningTime="2026-10-30 04:03:21.210689062 +0000 UTC m=+4913.325004037" Oct 30 04:03:21.441419 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:21.441391 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:03:21.441716 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:03:21.441691 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod555eacfa_6f9e_4efd_8fec_8b38c8c50283.slice/crio-743a0746af7dc19b1045ff8d59a727f4912962c1349296cd80796d8dc7814c8c WatchSource:0}: Error finding container 743a0746af7dc19b1045ff8d59a727f4912962c1349296cd80796d8dc7814c8c: Status 404 returned error can't find the container with id 743a0746af7dc19b1045ff8d59a727f4912962c1349296cd80796d8dc7814c8c Oct 30 04:03:22.165257 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:22.165222 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"555eacfa-6f9e-4efd-8fec-8b38c8c50283","Type":"ContainerStarted","Data":"bf8d2de4eda133b7faf8f8777b9261c576a56de2b72aa89ffcc3dee40e754ada"} Oct 30 04:03:22.165693 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:22.165272 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"555eacfa-6f9e-4efd-8fec-8b38c8c50283","Type":"ContainerStarted","Data":"743a0746af7dc19b1045ff8d59a727f4912962c1349296cd80796d8dc7814c8c"} Oct 30 04:03:22.213287 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:22.213234 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=11.213220456 podStartE2EDuration="11.213220456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:22.212581289 +0000 UTC m=+4914.326896266" watchObservedRunningTime="2026-10-30 04:03:22.213220456 +0000 UTC m=+4914.327535424" Oct 30 04:03:23.171010 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:23.170976 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:23.942914 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:03:23.942880 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:03:27.167331 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:27.167298 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:03:34.177208 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:34.177175 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:03:42.184964 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:42.184926 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp"] Oct 30 04:03:42.185420 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:42.185307 5882 cpu_manager.go:414] "containerMap: removing container" podUID="51a2c83e-1fd8-4a9c-81c7-95d6dd672e32" containerName="thanos-querier" Oct 30 04:03:42.185420 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:42.185318 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a2c83e-1fd8-4a9c-81c7-95d6dd672e32" containerName="thanos-querier" Oct 30 04:03:42.185420 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:42.185386 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a2c83e-1fd8-4a9c-81c7-95d6dd672e32" containerName="thanos-querier" Oct 30 04:03:42.187653 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:42.187634 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" Oct 30 04:03:42.219196 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:42.219164 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp"] Oct 30 04:03:42.318429 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:42.318394 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrqg5\" (UniqueName: \"kubernetes.io/projected/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-kube-api-access-wrqg5\") pod \"data-science-collector-targetallocator-78584cf67c-8b8sp\" (UID: \"7d8d9408-b95a-40f1-b63d-4c8f86d820e7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" Oct 30 04:03:42.318587 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:42.318446 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-78584cf67c-8b8sp\" (UID: \"7d8d9408-b95a-40f1-b63d-4c8f86d820e7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" Oct 30 04:03:42.318587 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:42.318472 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-ta-internal\") pod \"data-science-collector-targetallocator-78584cf67c-8b8sp\" (UID: \"7d8d9408-b95a-40f1-b63d-4c8f86d820e7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" Oct 30 04:03:42.419966 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:42.419931 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-78584cf67c-8b8sp\" (UID: \"7d8d9408-b95a-40f1-b63d-4c8f86d820e7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" Oct 30 04:03:42.419966 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:42.419966 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-ta-internal\") pod \"data-science-collector-targetallocator-78584cf67c-8b8sp\" (UID: \"7d8d9408-b95a-40f1-b63d-4c8f86d820e7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" Oct 30 04:03:42.420170 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:42.420026 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrqg5\" (UniqueName: \"kubernetes.io/projected/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-kube-api-access-wrqg5\") pod \"data-science-collector-targetallocator-78584cf67c-8b8sp\" (UID: \"7d8d9408-b95a-40f1-b63d-4c8f86d820e7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" Oct 30 04:03:42.420655 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:42.420629 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-ta-internal\") pod \"data-science-collector-targetallocator-78584cf67c-8b8sp\" (UID: \"7d8d9408-b95a-40f1-b63d-4c8f86d820e7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" Oct 30 04:03:42.422299 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:42.422276 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-78584cf67c-8b8sp\" (UID: \"7d8d9408-b95a-40f1-b63d-4c8f86d820e7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" Oct 30 04:03:42.444962 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:42.444898 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrqg5\" (UniqueName: \"kubernetes.io/projected/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-kube-api-access-wrqg5\") pod \"data-science-collector-targetallocator-78584cf67c-8b8sp\" (UID: \"7d8d9408-b95a-40f1-b63d-4c8f86d820e7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" Oct 30 04:03:42.498673 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:42.498639 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" Oct 30 04:03:42.790146 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:42.790119 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp"] Oct 30 04:03:42.790916 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:03:42.790887 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d8d9408_b95a_40f1_b63d_4c8f86d820e7.slice/crio-292c1ca79d7efa8c0f5879559908b37a2fc950e13c541ec24164e3cee37b5abf WatchSource:0}: Error finding container 292c1ca79d7efa8c0f5879559908b37a2fc950e13c541ec24164e3cee37b5abf: Status 404 returned error can't find the container with id 292c1ca79d7efa8c0f5879559908b37a2fc950e13c541ec24164e3cee37b5abf Oct 30 04:03:42.793103 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:42.793071 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:03:42.793163 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:42.793127 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:03:43.265909 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:43.265869 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" event={"ID":"7d8d9408-b95a-40f1-b63d-4c8f86d820e7","Type":"ContainerStarted","Data":"229b094f7d74d0111bcf2d89adbb33a4623283072cb2de791e9716368a5435db"} Oct 30 04:03:43.266449 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:43.265923 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" event={"ID":"7d8d9408-b95a-40f1-b63d-4c8f86d820e7","Type":"ContainerStarted","Data":"292c1ca79d7efa8c0f5879559908b37a2fc950e13c541ec24164e3cee37b5abf"} Oct 30 04:03:43.266449 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:43.266121 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" Oct 30 04:03:43.267930 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:43.267908 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" Oct 30 04:03:43.310280 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:43.310222 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" podStartSLOduration=1.310204641 podStartE2EDuration="1.310204641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:43.309531159 +0000 UTC m=+4935.423846134" watchObservedRunningTime="2026-10-30 04:03:43.310204641 +0000 UTC m=+4935.424519617" Oct 30 04:03:43.384746 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:43.384713 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9"] Oct 30 04:03:43.384925 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:43.384901 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" podUID="09e2fa8d-a3bd-41fc-9820-cb4e52dcf810" containerName="ta-container" containerID="cri-o://c7a08305fd356598793240d29e870f49e27185d34fc3f928f21f8022aa306194" gracePeriod=30 Oct 30 04:03:43.627273 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:43.627249 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" Oct 30 04:03:43.731927 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:43.731894 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-ta-internal\" (UniqueName: \"kubernetes.io/configmap/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-ta-internal\") pod \"09e2fa8d-a3bd-41fc-9820-cb4e52dcf810\" (UID: \"09e2fa8d-a3bd-41fc-9820-cb4e52dcf810\") " Oct 30 04:03:43.732094 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:43.731943 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-data-science-collector-ta-server-cert\") pod \"09e2fa8d-a3bd-41fc-9820-cb4e52dcf810\" (UID: \"09e2fa8d-a3bd-41fc-9820-cb4e52dcf810\") " Oct 30 04:03:43.732094 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:43.731985 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-kube-api-access-q22hj\" (UniqueName: \"kubernetes.io/projected/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-kube-api-access-q22hj\") pod \"09e2fa8d-a3bd-41fc-9820-cb4e52dcf810\" (UID: \"09e2fa8d-a3bd-41fc-9820-cb4e52dcf810\") " Oct 30 04:03:43.732300 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:43.732273 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-ta-internal" pod "09e2fa8d-a3bd-41fc-9820-cb4e52dcf810" (UID: "09e2fa8d-a3bd-41fc-9820-cb4e52dcf810"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:03:43.733971 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:43.733943 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-data-science-collector-ta-server-cert" pod "09e2fa8d-a3bd-41fc-9820-cb4e52dcf810" (UID: "09e2fa8d-a3bd-41fc-9820-cb4e52dcf810"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:03:43.734085 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:43.734056 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-kube-api-access-q22hj" pod "09e2fa8d-a3bd-41fc-9820-cb4e52dcf810" (UID: "09e2fa8d-a3bd-41fc-9820-cb4e52dcf810"). InnerVolumeSpecName "kube-api-access-q22hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:03:43.832881 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:43.832808 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:03:43.832881 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:43.832838 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:03:43.832881 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:43.832849 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q22hj\" (UniqueName: \"kubernetes.io/projected/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810-kube-api-access-q22hj\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:03:44.271473 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:44.271433 5882 generic.go:356] "Generic (PLEG): container finished" podID="09e2fa8d-a3bd-41fc-9820-cb4e52dcf810" containerID="c7a08305fd356598793240d29e870f49e27185d34fc3f928f21f8022aa306194" exitCode=0 Oct 30 04:03:44.271473 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:44.271475 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" Oct 30 04:03:44.272009 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:44.271473 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" event={"ID":"09e2fa8d-a3bd-41fc-9820-cb4e52dcf810","Type":"ContainerDied","Data":"c7a08305fd356598793240d29e870f49e27185d34fc3f928f21f8022aa306194"} Oct 30 04:03:44.272009 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:44.271521 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9" event={"ID":"09e2fa8d-a3bd-41fc-9820-cb4e52dcf810","Type":"ContainerDied","Data":"30631222dec6e83b49660047401836e8a07e2e80c6956461304ded00acae6c40"} Oct 30 04:03:44.272009 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:44.271543 5882 scope.go:122] "RemoveContainer" containerID="c7a08305fd356598793240d29e870f49e27185d34fc3f928f21f8022aa306194" Oct 30 04:03:44.279808 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:44.279789 5882 scope.go:122] "RemoveContainer" containerID="c7a08305fd356598793240d29e870f49e27185d34fc3f928f21f8022aa306194" Oct 30 04:03:44.280064 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:03:44.280043 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a08305fd356598793240d29e870f49e27185d34fc3f928f21f8022aa306194\": container with ID starting with c7a08305fd356598793240d29e870f49e27185d34fc3f928f21f8022aa306194 not found: ID does not exist" containerID="c7a08305fd356598793240d29e870f49e27185d34fc3f928f21f8022aa306194" Oct 30 04:03:44.280114 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:44.280072 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a08305fd356598793240d29e870f49e27185d34fc3f928f21f8022aa306194"} err="failed to get container status \"c7a08305fd356598793240d29e870f49e27185d34fc3f928f21f8022aa306194\": rpc error: code = NotFound desc = could not find container \"c7a08305fd356598793240d29e870f49e27185d34fc3f928f21f8022aa306194\": container with ID starting with c7a08305fd356598793240d29e870f49e27185d34fc3f928f21f8022aa306194 not found: ID does not exist" Oct 30 04:03:44.326240 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:44.326204 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9"] Oct 30 04:03:44.340674 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:44.340646 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-74d89c7b7b-hc6s9"] Oct 30 04:03:44.454537 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:44.454504 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-ingress-prometheus-18e332e25b8c3016"] Oct 30 04:03:44.454930 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:44.454910 5882 cpu_manager.go:414] "containerMap: removing container" podUID="09e2fa8d-a3bd-41fc-9820-cb4e52dcf810" containerName="ta-container" Oct 30 04:03:44.454983 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:44.454931 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e2fa8d-a3bd-41fc-9820-cb4e52dcf810" containerName="ta-container" Oct 30 04:03:44.455021 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:44.455008 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e2fa8d-a3bd-41fc-9820-cb4e52dcf810" containerName="ta-container" Oct 30 04:03:44.457859 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:44.457842 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-prometheus-18e332e25b8c3016" Oct 30 04:03:44.467464 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:44.467439 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-prometheus-18e332e25b8c3016" Oct 30 04:03:44.477748 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:44.477718 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-prometheus-18e332e25b8c3016"] Oct 30 04:03:44.760767 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:44.760739 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-prometheus-18e332e25b8c3016"] Oct 30 04:03:44.761255 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:03:44.761229 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5eb5707b_83ae_4525_8d1c_f2ef98a1b744.slice/crio-92ec39bf3e96a2e2c8f775f80d74a7ee316ce0c6b2732cf5b7f4e5a559e12c9e WatchSource:0}: Error finding container 92ec39bf3e96a2e2c8f775f80d74a7ee316ce0c6b2732cf5b7f4e5a559e12c9e: Status 404 returned error can't find the container with id 92ec39bf3e96a2e2c8f775f80d74a7ee316ce0c6b2732cf5b7f4e5a559e12c9e Oct 30 04:03:45.278679 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:45.278644 5882 generic.go:356] "Generic (PLEG): container finished" podID="5eb5707b-83ae-4525-8d1c-f2ef98a1b744" containerID="750aabaf98dfb1874010480f237b38347e552312bf901a0de7c3edfd1bf5465d" exitCode=0 Oct 30 04:03:45.279111 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:45.278761 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-ingress-prometheus-18e332e25b8c3016" event={"ID":"5eb5707b-83ae-4525-8d1c-f2ef98a1b744","Type":"ContainerDied","Data":"750aabaf98dfb1874010480f237b38347e552312bf901a0de7c3edfd1bf5465d"} Oct 30 04:03:45.279111 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:45.278797 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-ingress-prometheus-18e332e25b8c3016" event={"ID":"5eb5707b-83ae-4525-8d1c-f2ef98a1b744","Type":"ContainerStarted","Data":"92ec39bf3e96a2e2c8f775f80d74a7ee316ce0c6b2732cf5b7f4e5a559e12c9e"} Oct 30 04:03:46.009681 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:46.009649 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09e2fa8d-a3bd-41fc-9820-cb4e52dcf810" path="/var/lib/kubelet/pods/09e2fa8d-a3bd-41fc-9820-cb4e52dcf810/volumes" Oct 30 04:03:46.411721 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:46.411696 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-prometheus-18e332e25b8c3016" Oct 30 04:03:46.499220 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:46.499182 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-prometheus-18e332e25b8c3016"] Oct 30 04:03:46.503012 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:46.502986 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-prometheus-18e332e25b8c3016"] Oct 30 04:03:46.569261 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:46.569176 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-ingress-perses-18e332e2d9941097"] Oct 30 04:03:46.569653 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:46.569634 5882 cpu_manager.go:414] "containerMap: removing container" podUID="5eb5707b-83ae-4525-8d1c-f2ef98a1b744" containerName="curl" Oct 30 04:03:46.569758 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:46.569655 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb5707b-83ae-4525-8d1c-f2ef98a1b744" containerName="curl" Oct 30 04:03:46.569814 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:46.569786 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb5707b-83ae-4525-8d1c-f2ef98a1b744" containerName="curl" Oct 30 04:03:46.571940 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:46.571920 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-perses-18e332e2d9941097" Oct 30 04:03:46.581460 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:46.581442 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-perses-18e332e2d9941097" Oct 30 04:03:46.590977 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:46.590744 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-perses-18e332e2d9941097"] Oct 30 04:03:46.869739 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:46.869714 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-perses-18e332e2d9941097"] Oct 30 04:03:46.870562 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:03:46.870538 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91673968_0fca_4c0e_a02c_0be8bbd6c65f.slice/crio-abeff4e5d806e1542291d49b8c8b6be855ac30d0518c87d644cec508d13ccc36 WatchSource:0}: Error finding container abeff4e5d806e1542291d49b8c8b6be855ac30d0518c87d644cec508d13ccc36: Status 404 returned error can't find the container with id abeff4e5d806e1542291d49b8c8b6be855ac30d0518c87d644cec508d13ccc36 Oct 30 04:03:47.289520 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:47.289490 5882 generic.go:356] "Generic (PLEG): container finished" podID="91673968-0fca-4c0e-a02c-0be8bbd6c65f" containerID="9a2b2cd48c6c111448d7df6a4363a539a48f19c86cb49db527c70508c490b4e4" exitCode=0 Oct 30 04:03:47.289727 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:47.289580 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-ingress-perses-18e332e2d9941097" event={"ID":"91673968-0fca-4c0e-a02c-0be8bbd6c65f","Type":"ContainerDied","Data":"9a2b2cd48c6c111448d7df6a4363a539a48f19c86cb49db527c70508c490b4e4"} Oct 30 04:03:47.289727 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:47.289632 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-ingress-perses-18e332e2d9941097" event={"ID":"91673968-0fca-4c0e-a02c-0be8bbd6c65f","Type":"ContainerStarted","Data":"abeff4e5d806e1542291d49b8c8b6be855ac30d0518c87d644cec508d13ccc36"} Oct 30 04:03:47.290795 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:47.290776 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-prometheus-18e332e25b8c3016" Oct 30 04:03:47.290795 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:47.290786 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92ec39bf3e96a2e2c8f775f80d74a7ee316ce0c6b2732cf5b7f4e5a559e12c9e" Oct 30 04:03:47.346220 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:03:47.346188 5882 status_manager.go:1045] "Failed to get status for pod" err="pods \"thanos-ingress-prometheus-18e332e25b8c3016\" is forbidden: User \"system:node:ip-10-0-141-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"redhat-ods-monitoring\": no relationship found between node 'ip-10-0-141-200.ec2.internal' and this object" podUID="5eb5707b-83ae-4525-8d1c-f2ef98a1b744" pod="redhat-ods-monitoring/thanos-ingress-prometheus-18e332e25b8c3016" Oct 30 04:03:48.010948 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:48.010907 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb5707b-83ae-4525-8d1c-f2ef98a1b744" path="/var/lib/kubelet/pods/5eb5707b-83ae-4525-8d1c-f2ef98a1b744/volumes" Oct 30 04:03:48.023212 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:03:48.023177 5882 status_manager.go:1045] "Failed to get status for pod" err="pods \"thanos-ingress-prometheus-18e332e25b8c3016\" is forbidden: User \"system:node:ip-10-0-141-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"redhat-ods-monitoring\": no relationship found between node 'ip-10-0-141-200.ec2.internal' and this object" podUID="5eb5707b-83ae-4525-8d1c-f2ef98a1b744" pod="redhat-ods-monitoring/thanos-ingress-prometheus-18e332e25b8c3016" Oct 30 04:03:48.427178 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:48.427151 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-perses-18e332e2d9941097" Oct 30 04:03:48.614981 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:48.614893 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-perses-18e332e2d9941097"] Oct 30 04:03:48.618607 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:48.618583 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-perses-18e332e2d9941097"] Oct 30 04:03:48.683716 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:48.683687 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-ingress-korrel8r-18e332e357a96b77"] Oct 30 04:03:48.684026 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:48.684014 5882 cpu_manager.go:414] "containerMap: removing container" podUID="91673968-0fca-4c0e-a02c-0be8bbd6c65f" containerName="curl" Oct 30 04:03:48.684071 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:48.684027 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91673968-0fca-4c0e-a02c-0be8bbd6c65f" containerName="curl" Oct 30 04:03:48.684131 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:48.684122 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="91673968-0fca-4c0e-a02c-0be8bbd6c65f" containerName="curl" Oct 30 04:03:48.686095 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:48.686080 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-korrel8r-18e332e357a96b77" Oct 30 04:03:48.695474 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:48.695455 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-korrel8r-18e332e357a96b77" Oct 30 04:03:48.705126 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:48.705094 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-korrel8r-18e332e357a96b77"] Oct 30 04:03:48.988966 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:48.988937 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-korrel8r-18e332e357a96b77"] Oct 30 04:03:48.989484 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:03:48.989444 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab461c4e_d162_41b0_a38d_cf91512caba3.slice/crio-298368d48a4d7a6f0bd8990bfb9ac4a35160e8ef3fbe7fdf932b5cd3b583c2f2 WatchSource:0}: Error finding container 298368d48a4d7a6f0bd8990bfb9ac4a35160e8ef3fbe7fdf932b5cd3b583c2f2: Status 404 returned error can't find the container with id 298368d48a4d7a6f0bd8990bfb9ac4a35160e8ef3fbe7fdf932b5cd3b583c2f2 Oct 30 04:03:49.300795 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:49.300765 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-perses-18e332e2d9941097" Oct 30 04:03:49.301197 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:49.300813 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abeff4e5d806e1542291d49b8c8b6be855ac30d0518c87d644cec508d13ccc36" Oct 30 04:03:49.302381 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:49.302356 5882 generic.go:356] "Generic (PLEG): container finished" podID="ab461c4e-d162-41b0-a38d-cf91512caba3" containerID="5da9d414265dddbaa98db024d92d49def1a1af787f2e7006531d003729b0e7fb" exitCode=0 Oct 30 04:03:49.302511 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:49.302398 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-ingress-korrel8r-18e332e357a96b77" event={"ID":"ab461c4e-d162-41b0-a38d-cf91512caba3","Type":"ContainerDied","Data":"5da9d414265dddbaa98db024d92d49def1a1af787f2e7006531d003729b0e7fb"} Oct 30 04:03:49.302511 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:49.302427 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-ingress-korrel8r-18e332e357a96b77" event={"ID":"ab461c4e-d162-41b0-a38d-cf91512caba3","Type":"ContainerStarted","Data":"298368d48a4d7a6f0bd8990bfb9ac4a35160e8ef3fbe7fdf932b5cd3b583c2f2"} Oct 30 04:03:49.359303 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:03:49.359272 5882 status_manager.go:1045] "Failed to get status for pod" err="pods \"thanos-ingress-perses-18e332e2d9941097\" is forbidden: User \"system:node:ip-10-0-141-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"redhat-ods-monitoring\": no relationship found between node 'ip-10-0-141-200.ec2.internal' and this object" podUID="91673968-0fca-4c0e-a02c-0be8bbd6c65f" pod="redhat-ods-monitoring/thanos-ingress-perses-18e332e2d9941097" Oct 30 04:03:50.009741 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:50.009703 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91673968-0fca-4c0e-a02c-0be8bbd6c65f" path="/var/lib/kubelet/pods/91673968-0fca-4c0e-a02c-0be8bbd6c65f/volumes" Oct 30 04:03:50.440092 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:50.440067 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-korrel8r-18e332e357a96b77" Oct 30 04:03:50.729433 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:50.729354 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-korrel8r-18e332e357a96b77"] Oct 30 04:03:50.733640 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:50.733593 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-korrel8r-18e332e357a96b77"] Oct 30 04:03:50.799745 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:50.799713 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e332e3d5bd59af"] Oct 30 04:03:50.800076 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:50.800064 5882 cpu_manager.go:414] "containerMap: removing container" podUID="ab461c4e-d162-41b0-a38d-cf91512caba3" containerName="curl" Oct 30 04:03:50.800135 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:50.800077 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab461c4e-d162-41b0-a38d-cf91512caba3" containerName="curl" Oct 30 04:03:50.800172 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:50.800158 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab461c4e-d162-41b0-a38d-cf91512caba3" containerName="curl" Oct 30 04:03:50.802388 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:50.802371 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e332e3d5bd59af" Oct 30 04:03:50.812277 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:50.812256 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e332e3d5bd59af" Oct 30 04:03:50.821318 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:50.821285 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e332e3d5bd59af"] Oct 30 04:03:51.103223 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:51.103193 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e332e3d5bd59af"] Oct 30 04:03:51.104317 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:03:51.104295 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27e4943d_58a4_46b4_91c4_c563d2c7bd72.slice/crio-13528f66535aa7be0fcea99971661a4e194826b59495367b3571fd348f272ff2 WatchSource:0}: Error finding container 13528f66535aa7be0fcea99971661a4e194826b59495367b3571fd348f272ff2: Status 404 returned error can't find the container with id 13528f66535aa7be0fcea99971661a4e194826b59495367b3571fd348f272ff2 Oct 30 04:03:51.312308 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:51.312266 5882 generic.go:356] "Generic (PLEG): container finished" podID="27e4943d-58a4-46b4-91c4-c563d2c7bd72" containerID="cf1ea8207257a5f95369a1b252a2bbdee742d9245527643574a918e08fad7cf8" exitCode=0 Oct 30 04:03:51.312478 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:51.312342 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e332e3d5bd59af" event={"ID":"27e4943d-58a4-46b4-91c4-c563d2c7bd72","Type":"ContainerDied","Data":"cf1ea8207257a5f95369a1b252a2bbdee742d9245527643574a918e08fad7cf8"} Oct 30 04:03:51.312478 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:51.312385 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e332e3d5bd59af" event={"ID":"27e4943d-58a4-46b4-91c4-c563d2c7bd72","Type":"ContainerStarted","Data":"13528f66535aa7be0fcea99971661a4e194826b59495367b3571fd348f272ff2"} Oct 30 04:03:51.313687 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:51.313670 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="298368d48a4d7a6f0bd8990bfb9ac4a35160e8ef3fbe7fdf932b5cd3b583c2f2" Oct 30 04:03:51.313779 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:51.313693 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-korrel8r-18e332e357a96b77" Oct 30 04:03:51.369218 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:03:51.369150 5882 status_manager.go:1045] "Failed to get status for pod" err="pods \"thanos-ingress-korrel8r-18e332e357a96b77\" is forbidden: User \"system:node:ip-10-0-141-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"redhat-ods-monitoring\": no relationship found between node 'ip-10-0-141-200.ec2.internal' and this object" podUID="ab461c4e-d162-41b0-a38d-cf91512caba3" pod="redhat-ods-monitoring/thanos-ingress-korrel8r-18e332e357a96b77" Oct 30 04:03:52.009373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:52.009340 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab461c4e-d162-41b0-a38d-cf91512caba3" path="/var/lib/kubelet/pods/ab461c4e-d162-41b0-a38d-cf91512caba3/volumes" Oct 30 04:03:52.437949 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:52.437923 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e332e3d5bd59af" Oct 30 04:03:52.846165 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:52.846131 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e332e3d5bd59af"] Oct 30 04:03:52.849901 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:52.849875 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e332e3d5bd59af"] Oct 30 04:03:52.924650 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:52.924605 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["thanos-ingress-probe-18e332e452ed11f9/thanos-ingress-untrusted-18e332e45401a494"] Oct 30 04:03:52.925014 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:52.925001 5882 cpu_manager.go:414] "containerMap: removing container" podUID="27e4943d-58a4-46b4-91c4-c563d2c7bd72" containerName="curl" Oct 30 04:03:52.925060 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:52.925015 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e4943d-58a4-46b4-91c4-c563d2c7bd72" containerName="curl" Oct 30 04:03:52.925097 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:52.925068 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="27e4943d-58a4-46b4-91c4-c563d2c7bd72" containerName="curl" Oct 30 04:03:52.927102 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:52.927086 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="thanos-ingress-probe-18e332e452ed11f9/thanos-ingress-untrusted-18e332e45401a494" Oct 30 04:03:52.945750 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:52.945723 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["thanos-ingress-probe-18e332e452ed11f9/thanos-ingress-untrusted-18e332e45401a494"] Oct 30 04:03:52.945906 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:52.945799 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"thanos-ingress-probe-18e332e452ed11f9\"/\"default-dockercfg-k5pft\"" Oct 30 04:03:52.947856 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:52.947834 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="thanos-ingress-probe-18e332e452ed11f9/thanos-ingress-untrusted-18e332e45401a494" Oct 30 04:03:53.235222 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:53.235192 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["thanos-ingress-probe-18e332e452ed11f9/thanos-ingress-untrusted-18e332e45401a494"] Oct 30 04:03:53.236058 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:03:53.236034 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd8e2d2a_a6a4_4422_b82f_77cf96b658be.slice/crio-065d9e1d871642b5374e87a64858b6d5aeec34bb82bf59c6d18f3f9815dc8dd8 WatchSource:0}: Error finding container 065d9e1d871642b5374e87a64858b6d5aeec34bb82bf59c6d18f3f9815dc8dd8: Status 404 returned error can't find the container with id 065d9e1d871642b5374e87a64858b6d5aeec34bb82bf59c6d18f3f9815dc8dd8 Oct 30 04:03:53.323672 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:53.323648 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e332e3d5bd59af" Oct 30 04:03:53.323818 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:53.323647 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13528f66535aa7be0fcea99971661a4e194826b59495367b3571fd348f272ff2" Oct 30 04:03:53.325162 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:53.325138 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="thanos-ingress-probe-18e332e452ed11f9/thanos-ingress-untrusted-18e332e45401a494" event={"ID":"dd8e2d2a-a6a4-4422-b82f-77cf96b658be","Type":"ContainerStarted","Data":"5a6dcfb8edb5dcfb7bf284c026727347038dae408594f262e1c9fb89e06c7d32"} Oct 30 04:03:53.325273 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:53.325169 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="thanos-ingress-probe-18e332e452ed11f9/thanos-ingress-untrusted-18e332e45401a494" event={"ID":"dd8e2d2a-a6a4-4422-b82f-77cf96b658be","Type":"ContainerStarted","Data":"065d9e1d871642b5374e87a64858b6d5aeec34bb82bf59c6d18f3f9815dc8dd8"} Oct 30 04:03:53.379033 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:53.378979 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="thanos-ingress-probe-18e332e452ed11f9/thanos-ingress-untrusted-18e332e45401a494" podStartSLOduration=1.37896573 podStartE2EDuration="1.37896573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:53.378193441 +0000 UTC m=+4945.492508416" watchObservedRunningTime="2026-10-30 04:03:53.37896573 +0000 UTC m=+4945.493280704" Oct 30 04:03:53.393546 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:03:53.393514 5882 status_manager.go:1045] "Failed to get status for pod" err="pods \"thanos-ingress-authenticated-proxy-18e332e3d5bd59af\" is forbidden: User \"system:node:ip-10-0-141-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"redhat-ods-monitoring\": no relationship found between node 'ip-10-0-141-200.ec2.internal' and this object" podUID="27e4943d-58a4-46b4-91c4-c563d2c7bd72" pod="redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e332e3d5bd59af" Oct 30 04:03:53.942312 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:03:53.942280 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:03:54.009437 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:54.009406 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27e4943d-58a4-46b4-91c4-c563d2c7bd72" path="/var/lib/kubelet/pods/27e4943d-58a4-46b4-91c4-c563d2c7bd72/volumes" Oct 30 04:03:56.961965 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:56.961925 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk"] Oct 30 04:03:56.964810 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:56.964788 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" Oct 30 04:03:56.996026 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:56.995995 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk"] Oct 30 04:03:57.056715 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:57.056678 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-ta-internal\") pod \"data-science-collector-targetallocator-5ff6d99748-mhdkk\" (UID: \"6c7aeadf-793e-4a83-99ba-4090fc0c2ee0\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" Oct 30 04:03:57.056715 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:57.056718 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7hst\" (UniqueName: \"kubernetes.io/projected/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-kube-api-access-z7hst\") pod \"data-science-collector-targetallocator-5ff6d99748-mhdkk\" (UID: \"6c7aeadf-793e-4a83-99ba-4090fc0c2ee0\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" Oct 30 04:03:57.056925 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:57.056874 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5ff6d99748-mhdkk\" (UID: \"6c7aeadf-793e-4a83-99ba-4090fc0c2ee0\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" Oct 30 04:03:57.158132 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:57.158088 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5ff6d99748-mhdkk\" (UID: \"6c7aeadf-793e-4a83-99ba-4090fc0c2ee0\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" Oct 30 04:03:57.158310 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:57.158189 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-ta-internal\") pod \"data-science-collector-targetallocator-5ff6d99748-mhdkk\" (UID: \"6c7aeadf-793e-4a83-99ba-4090fc0c2ee0\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" Oct 30 04:03:57.158310 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:57.158223 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7hst\" (UniqueName: \"kubernetes.io/projected/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-kube-api-access-z7hst\") pod \"data-science-collector-targetallocator-5ff6d99748-mhdkk\" (UID: \"6c7aeadf-793e-4a83-99ba-4090fc0c2ee0\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" Oct 30 04:03:57.158928 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:57.158903 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-ta-internal\") pod \"data-science-collector-targetallocator-5ff6d99748-mhdkk\" (UID: \"6c7aeadf-793e-4a83-99ba-4090fc0c2ee0\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" Oct 30 04:03:57.160583 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:57.160564 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5ff6d99748-mhdkk\" (UID: \"6c7aeadf-793e-4a83-99ba-4090fc0c2ee0\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" Oct 30 04:03:57.183669 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:57.183636 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7hst\" (UniqueName: \"kubernetes.io/projected/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-kube-api-access-z7hst\") pod \"data-science-collector-targetallocator-5ff6d99748-mhdkk\" (UID: \"6c7aeadf-793e-4a83-99ba-4090fc0c2ee0\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" Oct 30 04:03:57.274977 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:57.274935 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" Oct 30 04:03:57.566160 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:57.566126 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk"] Oct 30 04:03:57.566629 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:03:57.566595 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c7aeadf_793e_4a83_99ba_4090fc0c2ee0.slice/crio-243abc03a5f8e33b582ba5e137727595ace4ed144021e2710f45887947723af0 WatchSource:0}: Error finding container 243abc03a5f8e33b582ba5e137727595ace4ed144021e2710f45887947723af0: Status 404 returned error can't find the container with id 243abc03a5f8e33b582ba5e137727595ace4ed144021e2710f45887947723af0 Oct 30 04:03:57.568726 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:57.568697 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:03:57.568828 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:57.568762 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:03:58.349021 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:58.348988 5882 generic.go:356] "Generic (PLEG): container finished" podID="dd8e2d2a-a6a4-4422-b82f-77cf96b658be" containerID="5a6dcfb8edb5dcfb7bf284c026727347038dae408594f262e1c9fb89e06c7d32" exitCode=0 Oct 30 04:03:58.349426 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:58.349062 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="thanos-ingress-probe-18e332e452ed11f9/thanos-ingress-untrusted-18e332e45401a494" event={"ID":"dd8e2d2a-a6a4-4422-b82f-77cf96b658be","Type":"ContainerDied","Data":"5a6dcfb8edb5dcfb7bf284c026727347038dae408594f262e1c9fb89e06c7d32"} Oct 30 04:03:58.350711 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:58.350682 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" event={"ID":"6c7aeadf-793e-4a83-99ba-4090fc0c2ee0","Type":"ContainerStarted","Data":"54fa6049557b9b058d8a42b2ece3b74823931d22e0c48869ff512f0f910df11d"} Oct 30 04:03:58.350820 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:58.350718 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" event={"ID":"6c7aeadf-793e-4a83-99ba-4090fc0c2ee0","Type":"ContainerStarted","Data":"243abc03a5f8e33b582ba5e137727595ace4ed144021e2710f45887947723af0"} Oct 30 04:03:58.350886 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:58.350870 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" Oct 30 04:03:58.352596 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:58.352575 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" Oct 30 04:03:58.436994 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:58.436939 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" podStartSLOduration=2.436925223 podStartE2EDuration="2.436925223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:58.435936908 +0000 UTC m=+4950.550251882" watchObservedRunningTime="2026-10-30 04:03:58.436925223 +0000 UTC m=+4950.551240192" Oct 30 04:03:58.469166 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:58.469132 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp"] Oct 30 04:03:58.469385 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:58.469360 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" podUID="7d8d9408-b95a-40f1-b63d-4c8f86d820e7" containerName="ta-container" containerID="cri-o://229b094f7d74d0111bcf2d89adbb33a4623283072cb2de791e9716368a5435db" gracePeriod=30 Oct 30 04:03:58.725253 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:58.725226 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" Oct 30 04:03:58.774161 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:58.774133 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-ta-internal\" (UniqueName: \"kubernetes.io/configmap/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-ta-internal\") pod \"7d8d9408-b95a-40f1-b63d-4c8f86d820e7\" (UID: \"7d8d9408-b95a-40f1-b63d-4c8f86d820e7\") " Oct 30 04:03:58.774311 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:58.774208 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-data-science-collector-ta-server-cert\") pod \"7d8d9408-b95a-40f1-b63d-4c8f86d820e7\" (UID: \"7d8d9408-b95a-40f1-b63d-4c8f86d820e7\") " Oct 30 04:03:58.774311 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:58.774244 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-kube-api-access-wrqg5\" (UniqueName: \"kubernetes.io/projected/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-kube-api-access-wrqg5\") pod \"7d8d9408-b95a-40f1-b63d-4c8f86d820e7\" (UID: \"7d8d9408-b95a-40f1-b63d-4c8f86d820e7\") " Oct 30 04:03:58.774550 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:58.774528 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-ta-internal" pod "7d8d9408-b95a-40f1-b63d-4c8f86d820e7" (UID: "7d8d9408-b95a-40f1-b63d-4c8f86d820e7"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:03:58.776366 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:58.776341 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-kube-api-access-wrqg5" pod "7d8d9408-b95a-40f1-b63d-4c8f86d820e7" (UID: "7d8d9408-b95a-40f1-b63d-4c8f86d820e7"). InnerVolumeSpecName "kube-api-access-wrqg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:03:58.776506 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:58.776486 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-data-science-collector-ta-server-cert" pod "7d8d9408-b95a-40f1-b63d-4c8f86d820e7" (UID: "7d8d9408-b95a-40f1-b63d-4c8f86d820e7"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:03:58.875189 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:58.875113 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:03:58.875189 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:58.875141 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrqg5\" (UniqueName: \"kubernetes.io/projected/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-kube-api-access-wrqg5\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:03:58.875189 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:58.875151 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7d8d9408-b95a-40f1-b63d-4c8f86d820e7-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:03:59.356793 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:59.356762 5882 generic.go:356] "Generic (PLEG): container finished" podID="7d8d9408-b95a-40f1-b63d-4c8f86d820e7" containerID="229b094f7d74d0111bcf2d89adbb33a4623283072cb2de791e9716368a5435db" exitCode=0 Oct 30 04:03:59.357212 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:59.356811 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" Oct 30 04:03:59.357212 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:59.356858 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" event={"ID":"7d8d9408-b95a-40f1-b63d-4c8f86d820e7","Type":"ContainerDied","Data":"229b094f7d74d0111bcf2d89adbb33a4623283072cb2de791e9716368a5435db"} Oct 30 04:03:59.357212 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:59.356905 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp" event={"ID":"7d8d9408-b95a-40f1-b63d-4c8f86d820e7","Type":"ContainerDied","Data":"292c1ca79d7efa8c0f5879559908b37a2fc950e13c541ec24164e3cee37b5abf"} Oct 30 04:03:59.357212 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:59.356926 5882 scope.go:122] "RemoveContainer" containerID="229b094f7d74d0111bcf2d89adbb33a4623283072cb2de791e9716368a5435db" Oct 30 04:03:59.365939 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:59.365914 5882 scope.go:122] "RemoveContainer" containerID="229b094f7d74d0111bcf2d89adbb33a4623283072cb2de791e9716368a5435db" Oct 30 04:03:59.366232 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:03:59.366213 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"229b094f7d74d0111bcf2d89adbb33a4623283072cb2de791e9716368a5435db\": container with ID starting with 229b094f7d74d0111bcf2d89adbb33a4623283072cb2de791e9716368a5435db not found: ID does not exist" containerID="229b094f7d74d0111bcf2d89adbb33a4623283072cb2de791e9716368a5435db" Oct 30 04:03:59.366289 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:59.366242 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"229b094f7d74d0111bcf2d89adbb33a4623283072cb2de791e9716368a5435db"} err="failed to get container status \"229b094f7d74d0111bcf2d89adbb33a4623283072cb2de791e9716368a5435db\": rpc error: code = NotFound desc = could not find container \"229b094f7d74d0111bcf2d89adbb33a4623283072cb2de791e9716368a5435db\": container with ID starting with 229b094f7d74d0111bcf2d89adbb33a4623283072cb2de791e9716368a5435db not found: ID does not exist" Oct 30 04:03:59.414847 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:59.412839 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp"] Oct 30 04:03:59.426415 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:59.426352 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-78584cf67c-8b8sp"] Oct 30 04:03:59.489200 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:03:59.489176 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="thanos-ingress-probe-18e332e452ed11f9/thanos-ingress-untrusted-18e332e45401a494" Oct 30 04:04:00.015516 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:00.015474 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d8d9408-b95a-40f1-b63d-4c8f86d820e7" path="/var/lib/kubelet/pods/7d8d9408-b95a-40f1-b63d-4c8f86d820e7/volumes" Oct 30 04:04:00.364207 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:00.364130 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="thanos-ingress-probe-18e332e452ed11f9/thanos-ingress-untrusted-18e332e45401a494" Oct 30 04:04:00.364207 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:00.364134 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="thanos-ingress-probe-18e332e452ed11f9/thanos-ingress-untrusted-18e332e45401a494" event={"ID":"dd8e2d2a-a6a4-4422-b82f-77cf96b658be","Type":"ContainerDied","Data":"065d9e1d871642b5374e87a64858b6d5aeec34bb82bf59c6d18f3f9815dc8dd8"} Oct 30 04:04:00.364207 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:00.364181 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="065d9e1d871642b5374e87a64858b6d5aeec34bb82bf59c6d18f3f9815dc8dd8" Oct 30 04:04:01.025975 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:01.025940 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["thanos-ingress-probe-18e332e452ed11f9/thanos-ingress-untrusted-18e332e45401a494"] Oct 30 04:04:01.032096 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:01.032071 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["thanos-ingress-probe-18e332e452ed11f9/thanos-ingress-untrusted-18e332e45401a494"] Oct 30 04:04:02.009659 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:02.009608 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd8e2d2a-a6a4-4422-b82f-77cf96b658be" path="/var/lib/kubelet/pods/dd8e2d2a-a6a4-4422-b82f-77cf96b658be/volumes" Oct 30 04:04:08.090729 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.090695 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26"] Oct 30 04:04:08.091137 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.091066 5882 cpu_manager.go:414] "containerMap: removing container" podUID="dd8e2d2a-a6a4-4422-b82f-77cf96b658be" containerName="curl" Oct 30 04:04:08.091137 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.091079 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd8e2d2a-a6a4-4422-b82f-77cf96b658be" containerName="curl" Oct 30 04:04:08.091137 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.091087 5882 cpu_manager.go:414] "containerMap: removing container" podUID="7d8d9408-b95a-40f1-b63d-4c8f86d820e7" containerName="ta-container" Oct 30 04:04:08.091137 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.091092 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d8d9408-b95a-40f1-b63d-4c8f86d820e7" containerName="ta-container" Oct 30 04:04:08.091280 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.091167 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd8e2d2a-a6a4-4422-b82f-77cf96b658be" containerName="curl" Oct 30 04:04:08.091280 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.091176 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d8d9408-b95a-40f1-b63d-4c8f86d820e7" containerName="ta-container" Oct 30 04:04:08.093271 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.093250 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" Oct 30 04:04:08.124362 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.124328 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26"] Oct 30 04:04:08.154717 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.154686 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv27l\" (UniqueName: \"kubernetes.io/projected/0f856657-3fc0-4339-9fb9-02e4e193d103-kube-api-access-zv27l\") pod \"data-science-collector-targetallocator-ff587cc8b-v9m26\" (UID: \"0f856657-3fc0-4339-9fb9-02e4e193d103\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" Oct 30 04:04:08.154868 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.154747 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/0f856657-3fc0-4339-9fb9-02e4e193d103-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-ff587cc8b-v9m26\" (UID: \"0f856657-3fc0-4339-9fb9-02e4e193d103\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" Oct 30 04:04:08.154868 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.154811 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/0f856657-3fc0-4339-9fb9-02e4e193d103-ta-internal\") pod \"data-science-collector-targetallocator-ff587cc8b-v9m26\" (UID: \"0f856657-3fc0-4339-9fb9-02e4e193d103\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" Oct 30 04:04:08.255744 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.255706 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv27l\" (UniqueName: \"kubernetes.io/projected/0f856657-3fc0-4339-9fb9-02e4e193d103-kube-api-access-zv27l\") pod \"data-science-collector-targetallocator-ff587cc8b-v9m26\" (UID: \"0f856657-3fc0-4339-9fb9-02e4e193d103\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" Oct 30 04:04:08.255909 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.255765 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/0f856657-3fc0-4339-9fb9-02e4e193d103-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-ff587cc8b-v9m26\" (UID: \"0f856657-3fc0-4339-9fb9-02e4e193d103\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" Oct 30 04:04:08.255909 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.255787 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/0f856657-3fc0-4339-9fb9-02e4e193d103-ta-internal\") pod \"data-science-collector-targetallocator-ff587cc8b-v9m26\" (UID: \"0f856657-3fc0-4339-9fb9-02e4e193d103\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" Oct 30 04:04:08.256420 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.256399 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/0f856657-3fc0-4339-9fb9-02e4e193d103-ta-internal\") pod \"data-science-collector-targetallocator-ff587cc8b-v9m26\" (UID: \"0f856657-3fc0-4339-9fb9-02e4e193d103\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" Oct 30 04:04:08.258254 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.258218 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/0f856657-3fc0-4339-9fb9-02e4e193d103-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-ff587cc8b-v9m26\" (UID: \"0f856657-3fc0-4339-9fb9-02e4e193d103\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" Oct 30 04:04:08.282962 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.282935 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv27l\" (UniqueName: \"kubernetes.io/projected/0f856657-3fc0-4339-9fb9-02e4e193d103-kube-api-access-zv27l\") pod \"data-science-collector-targetallocator-ff587cc8b-v9m26\" (UID: \"0f856657-3fc0-4339-9fb9-02e4e193d103\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" Oct 30 04:04:08.404003 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.403926 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" Oct 30 04:04:08.696355 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.696326 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26"] Oct 30 04:04:08.697581 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:04:08.697550 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f856657_3fc0_4339_9fb9_02e4e193d103.slice/crio-45f665c69df02f78523b63db4e86d25822c87fa3930a7da76106156fce7790bb WatchSource:0}: Error finding container 45f665c69df02f78523b63db4e86d25822c87fa3930a7da76106156fce7790bb: Status 404 returned error can't find the container with id 45f665c69df02f78523b63db4e86d25822c87fa3930a7da76106156fce7790bb Oct 30 04:04:08.700076 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.700047 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:08.700151 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:08.700106 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:09.404058 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:09.404022 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" event={"ID":"0f856657-3fc0-4339-9fb9-02e4e193d103","Type":"ContainerStarted","Data":"76e4a369d11386c9322caabf73c60b5e996a0d67fb06a3ec9aca15c7479811be"} Oct 30 04:04:09.404058 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:09.404061 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" event={"ID":"0f856657-3fc0-4339-9fb9-02e4e193d103","Type":"ContainerStarted","Data":"45f665c69df02f78523b63db4e86d25822c87fa3930a7da76106156fce7790bb"} Oct 30 04:04:09.404533 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:09.404174 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" Oct 30 04:04:09.405971 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:09.405949 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" Oct 30 04:04:09.449146 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:09.449100 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" podStartSLOduration=1.449087129 podStartE2EDuration="1.449087129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:09.447799417 +0000 UTC m=+4961.562114393" watchObservedRunningTime="2026-10-30 04:04:09.449087129 +0000 UTC m=+4961.563402100" Oct 30 04:04:09.526929 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:09.526892 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk"] Oct 30 04:04:09.527098 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:09.527057 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" podUID="6c7aeadf-793e-4a83-99ba-4090fc0c2ee0" containerName="ta-container" containerID="cri-o://54fa6049557b9b058d8a42b2ece3b74823931d22e0c48869ff512f0f910df11d" gracePeriod=30 Oct 30 04:04:09.766700 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:09.766674 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" Oct 30 04:04:09.872931 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:09.872896 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-kube-api-access-z7hst\" (UniqueName: \"kubernetes.io/projected/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-kube-api-access-z7hst\") pod \"6c7aeadf-793e-4a83-99ba-4090fc0c2ee0\" (UID: \"6c7aeadf-793e-4a83-99ba-4090fc0c2ee0\") " Oct 30 04:04:09.873093 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:09.872942 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-data-science-collector-ta-server-cert\") pod \"6c7aeadf-793e-4a83-99ba-4090fc0c2ee0\" (UID: \"6c7aeadf-793e-4a83-99ba-4090fc0c2ee0\") " Oct 30 04:04:09.873093 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:09.872989 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-ta-internal\" (UniqueName: \"kubernetes.io/configmap/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-ta-internal\") pod \"6c7aeadf-793e-4a83-99ba-4090fc0c2ee0\" (UID: \"6c7aeadf-793e-4a83-99ba-4090fc0c2ee0\") " Oct 30 04:04:09.873399 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:09.873369 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-ta-internal" pod "6c7aeadf-793e-4a83-99ba-4090fc0c2ee0" (UID: "6c7aeadf-793e-4a83-99ba-4090fc0c2ee0"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:09.874956 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:09.874928 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-data-science-collector-ta-server-cert" pod "6c7aeadf-793e-4a83-99ba-4090fc0c2ee0" (UID: "6c7aeadf-793e-4a83-99ba-4090fc0c2ee0"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:09.875047 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:09.874938 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-kube-api-access-z7hst" pod "6c7aeadf-793e-4a83-99ba-4090fc0c2ee0" (UID: "6c7aeadf-793e-4a83-99ba-4090fc0c2ee0"). InnerVolumeSpecName "kube-api-access-z7hst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:09.973667 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:09.973575 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:09.973667 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:09.973641 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7hst\" (UniqueName: \"kubernetes.io/projected/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-kube-api-access-z7hst\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:09.973667 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:09.973659 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:10.410759 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:10.410722 5882 generic.go:356] "Generic (PLEG): container finished" podID="6c7aeadf-793e-4a83-99ba-4090fc0c2ee0" containerID="54fa6049557b9b058d8a42b2ece3b74823931d22e0c48869ff512f0f910df11d" exitCode=0 Oct 30 04:04:10.411172 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:10.410762 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" Oct 30 04:04:10.411172 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:10.410802 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" event={"ID":"6c7aeadf-793e-4a83-99ba-4090fc0c2ee0","Type":"ContainerDied","Data":"54fa6049557b9b058d8a42b2ece3b74823931d22e0c48869ff512f0f910df11d"} Oct 30 04:04:10.411172 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:10.410841 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk" event={"ID":"6c7aeadf-793e-4a83-99ba-4090fc0c2ee0","Type":"ContainerDied","Data":"243abc03a5f8e33b582ba5e137727595ace4ed144021e2710f45887947723af0"} Oct 30 04:04:10.411172 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:10.410862 5882 scope.go:122] "RemoveContainer" containerID="54fa6049557b9b058d8a42b2ece3b74823931d22e0c48869ff512f0f910df11d" Oct 30 04:04:10.419440 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:10.419421 5882 scope.go:122] "RemoveContainer" containerID="54fa6049557b9b058d8a42b2ece3b74823931d22e0c48869ff512f0f910df11d" Oct 30 04:04:10.419819 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:04:10.419799 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54fa6049557b9b058d8a42b2ece3b74823931d22e0c48869ff512f0f910df11d\": container with ID starting with 54fa6049557b9b058d8a42b2ece3b74823931d22e0c48869ff512f0f910df11d not found: ID does not exist" containerID="54fa6049557b9b058d8a42b2ece3b74823931d22e0c48869ff512f0f910df11d" Oct 30 04:04:10.419912 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:10.419825 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54fa6049557b9b058d8a42b2ece3b74823931d22e0c48869ff512f0f910df11d"} err="failed to get container status \"54fa6049557b9b058d8a42b2ece3b74823931d22e0c48869ff512f0f910df11d\": rpc error: code = NotFound desc = could not find container \"54fa6049557b9b058d8a42b2ece3b74823931d22e0c48869ff512f0f910df11d\": container with ID starting with 54fa6049557b9b058d8a42b2ece3b74823931d22e0c48869ff512f0f910df11d not found: ID does not exist" Oct 30 04:04:10.465900 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:10.465874 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk"] Oct 30 04:04:10.479967 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:10.479946 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5ff6d99748-mhdkk"] Oct 30 04:04:11.101427 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.101393 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:11.101802 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.101789 5882 cpu_manager.go:414] "containerMap: removing container" podUID="6c7aeadf-793e-4a83-99ba-4090fc0c2ee0" containerName="ta-container" Oct 30 04:04:11.101865 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.101803 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c7aeadf-793e-4a83-99ba-4090fc0c2ee0" containerName="ta-container" Oct 30 04:04:11.101865 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.101857 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c7aeadf-793e-4a83-99ba-4090fc0c2ee0" containerName="ta-container" Oct 30 04:04:11.103807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.103788 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:11.134478 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.134449 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:04:11.134478 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.134450 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:04:11.134716 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.134541 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:04:11.134716 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.134581 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-45ch8\"" Oct 30 04:04:11.148710 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.148534 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-72c79cac\"" Oct 30 04:04:11.150352 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.150330 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:11.183663 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.183635 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:11.183776 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.183748 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8rlf\" (UniqueName: \"kubernetes.io/projected/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-kube-api-access-h8rlf\") pod \"data-science-collector-collector-1\" (UID: \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:11.183827 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.183812 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:11.183861 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.183841 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:11.183892 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.183859 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:11.284758 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.284727 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8rlf\" (UniqueName: \"kubernetes.io/projected/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-kube-api-access-h8rlf\") pod \"data-science-collector-collector-1\" (UID: \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:11.284923 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.284770 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:11.284923 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.284792 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:11.284923 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.284841 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:11.285077 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.284995 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:11.285529 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.285501 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:11.287163 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.287144 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:11.287246 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.287183 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:11.287386 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.287362 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:11.309853 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.309828 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8rlf\" (UniqueName: \"kubernetes.io/projected/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-kube-api-access-h8rlf\") pod \"data-science-collector-collector-1\" (UID: \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:11.414980 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.414898 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:11.708497 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:04:11.708464 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod331e60cc_fbf6_4c1d_ab57_9c710fac3e99.slice/crio-ab1fb7ae951ff6acce936ac3deb81f4d209d98bd4f5d322b36b55f6b46860372 WatchSource:0}: Error finding container ab1fb7ae951ff6acce936ac3deb81f4d209d98bd4f5d322b36b55f6b46860372: Status 404 returned error can't find the container with id ab1fb7ae951ff6acce936ac3deb81f4d209d98bd4f5d322b36b55f6b46860372 Oct 30 04:04:11.708610 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.708587 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:11.710775 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.710748 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:11.710846 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:11.710802 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:12.009960 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:12.009916 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c7aeadf-793e-4a83-99ba-4090fc0c2ee0" path="/var/lib/kubelet/pods/6c7aeadf-793e-4a83-99ba-4090fc0c2ee0/volumes" Oct 30 04:04:12.422031 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:12.421952 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"331e60cc-fbf6-4c1d-ab57-9c710fac3e99","Type":"ContainerStarted","Data":"a0dd85289c5aaebb787f34557ccb0b6832393491a65f56a6ec56ee6f6c7df94c"} Oct 30 04:04:12.422031 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:12.422002 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"331e60cc-fbf6-4c1d-ab57-9c710fac3e99","Type":"ContainerStarted","Data":"ab1fb7ae951ff6acce936ac3deb81f4d209d98bd4f5d322b36b55f6b46860372"} Oct 30 04:04:12.468472 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:12.468420 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.468408194 podStartE2EDuration="1.468408194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:12.467569936 +0000 UTC m=+4964.581884934" watchObservedRunningTime="2026-10-30 04:04:12.468408194 +0000 UTC m=+4964.582723163" Oct 30 04:04:16.637108 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:16.637070 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:16.637730 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:16.637285 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="331e60cc-fbf6-4c1d-ab57-9c710fac3e99" containerName="otc-container" containerID="cri-o://a0dd85289c5aaebb787f34557ccb0b6832393491a65f56a6ec56ee6f6c7df94c" gracePeriod=30 Oct 30 04:04:16.882910 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:16.882879 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:16.898862 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:16.898791 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k"] Oct 30 04:04:16.899194 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:16.899178 5882 cpu_manager.go:414] "containerMap: removing container" podUID="331e60cc-fbf6-4c1d-ab57-9c710fac3e99" containerName="otc-container" Oct 30 04:04:16.899267 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:16.899194 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="331e60cc-fbf6-4c1d-ab57-9c710fac3e99" containerName="otc-container" Oct 30 04:04:16.899329 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:16.899281 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="331e60cc-fbf6-4c1d-ab57-9c710fac3e99" containerName="otc-container" Oct 30 04:04:16.901156 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:16.901136 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" Oct 30 04:04:16.924369 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:16.924342 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k"] Oct 30 04:04:16.939089 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:16.939058 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-monitoring-tls-certs\") pod \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\" (UID: \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\") " Oct 30 04:04:16.939264 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:16.939118 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-otc-internal\" (UniqueName: \"kubernetes.io/configmap/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-otc-internal\") pod \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\" (UID: \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\") " Oct 30 04:04:16.939264 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:16.939186 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-kube-api-access-h8rlf\" (UniqueName: \"kubernetes.io/projected/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-kube-api-access-h8rlf\") pod \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\" (UID: \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\") " Oct 30 04:04:16.939264 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:16.939259 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-data-science-collector-ta-client-cert\") pod \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\" (UID: \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\") " Oct 30 04:04:16.939451 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:16.939357 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-tls-certs\" (UniqueName: \"kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-tls-certs\") pod \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\" (UID: \"331e60cc-fbf6-4c1d-ab57-9c710fac3e99\") " Oct 30 04:04:16.939524 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:16.939501 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-otc-internal" pod "331e60cc-fbf6-4c1d-ab57-9c710fac3e99" (UID: "331e60cc-fbf6-4c1d-ab57-9c710fac3e99"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:16.939721 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:16.939700 5882 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-otc-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:16.941393 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:16.941367 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-data-science-collector-ta-client-cert" pod "331e60cc-fbf6-4c1d-ab57-9c710fac3e99" (UID: "331e60cc-fbf6-4c1d-ab57-9c710fac3e99"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:16.941573 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:16.941554 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-tls-certs" pod "331e60cc-fbf6-4c1d-ab57-9c710fac3e99" (UID: "331e60cc-fbf6-4c1d-ab57-9c710fac3e99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:16.941776 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:16.941756 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-monitoring-tls-certs" pod "331e60cc-fbf6-4c1d-ab57-9c710fac3e99" (UID: "331e60cc-fbf6-4c1d-ab57-9c710fac3e99"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:16.941847 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:16.941808 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-kube-api-access-h8rlf" pod "331e60cc-fbf6-4c1d-ab57-9c710fac3e99" (UID: "331e60cc-fbf6-4c1d-ab57-9c710fac3e99"). InnerVolumeSpecName "kube-api-access-h8rlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:17.040550 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.040496 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b1826f77-7ce4-4e29-bd49-23a730e22587-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6967c5f799-4d55k\" (UID: \"b1826f77-7ce4-4e29-bd49-23a730e22587\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" Oct 30 04:04:17.040744 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.040645 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b1826f77-7ce4-4e29-bd49-23a730e22587-ta-internal\") pod \"data-science-collector-targetallocator-6967c5f799-4d55k\" (UID: \"b1826f77-7ce4-4e29-bd49-23a730e22587\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" Oct 30 04:04:17.040744 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.040699 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnxkx\" (UniqueName: \"kubernetes.io/projected/b1826f77-7ce4-4e29-bd49-23a730e22587-kube-api-access-gnxkx\") pod \"data-science-collector-targetallocator-6967c5f799-4d55k\" (UID: \"b1826f77-7ce4-4e29-bd49-23a730e22587\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" Oct 30 04:04:17.040830 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.040783 5882 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-monitoring-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:17.040830 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.040797 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8rlf\" (UniqueName: \"kubernetes.io/projected/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-kube-api-access-h8rlf\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:17.040830 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.040808 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:17.040830 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.040818 5882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/331e60cc-fbf6-4c1d-ab57-9c710fac3e99-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:17.141346 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.141306 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnxkx\" (UniqueName: \"kubernetes.io/projected/b1826f77-7ce4-4e29-bd49-23a730e22587-kube-api-access-gnxkx\") pod \"data-science-collector-targetallocator-6967c5f799-4d55k\" (UID: \"b1826f77-7ce4-4e29-bd49-23a730e22587\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" Oct 30 04:04:17.141520 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.141407 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b1826f77-7ce4-4e29-bd49-23a730e22587-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6967c5f799-4d55k\" (UID: \"b1826f77-7ce4-4e29-bd49-23a730e22587\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" Oct 30 04:04:17.141520 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.141457 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b1826f77-7ce4-4e29-bd49-23a730e22587-ta-internal\") pod \"data-science-collector-targetallocator-6967c5f799-4d55k\" (UID: \"b1826f77-7ce4-4e29-bd49-23a730e22587\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" Oct 30 04:04:17.142132 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.142112 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b1826f77-7ce4-4e29-bd49-23a730e22587-ta-internal\") pod \"data-science-collector-targetallocator-6967c5f799-4d55k\" (UID: \"b1826f77-7ce4-4e29-bd49-23a730e22587\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" Oct 30 04:04:17.143820 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.143799 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b1826f77-7ce4-4e29-bd49-23a730e22587-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6967c5f799-4d55k\" (UID: \"b1826f77-7ce4-4e29-bd49-23a730e22587\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" Oct 30 04:04:17.167527 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.167470 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnxkx\" (UniqueName: \"kubernetes.io/projected/b1826f77-7ce4-4e29-bd49-23a730e22587-kube-api-access-gnxkx\") pod \"data-science-collector-targetallocator-6967c5f799-4d55k\" (UID: \"b1826f77-7ce4-4e29-bd49-23a730e22587\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" Oct 30 04:04:17.210990 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.210961 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" Oct 30 04:04:17.445546 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.445446 5882 generic.go:356] "Generic (PLEG): container finished" podID="331e60cc-fbf6-4c1d-ab57-9c710fac3e99" containerID="a0dd85289c5aaebb787f34557ccb0b6832393491a65f56a6ec56ee6f6c7df94c" exitCode=0 Oct 30 04:04:17.445546 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.445497 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:17.445546 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.445538 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"331e60cc-fbf6-4c1d-ab57-9c710fac3e99","Type":"ContainerDied","Data":"a0dd85289c5aaebb787f34557ccb0b6832393491a65f56a6ec56ee6f6c7df94c"} Oct 30 04:04:17.445852 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.445584 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"331e60cc-fbf6-4c1d-ab57-9c710fac3e99","Type":"ContainerDied","Data":"ab1fb7ae951ff6acce936ac3deb81f4d209d98bd4f5d322b36b55f6b46860372"} Oct 30 04:04:17.445852 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.445606 5882 scope.go:122] "RemoveContainer" containerID="a0dd85289c5aaebb787f34557ccb0b6832393491a65f56a6ec56ee6f6c7df94c" Oct 30 04:04:17.454423 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.454403 5882 scope.go:122] "RemoveContainer" containerID="a0dd85289c5aaebb787f34557ccb0b6832393491a65f56a6ec56ee6f6c7df94c" Oct 30 04:04:17.454710 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:04:17.454687 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0dd85289c5aaebb787f34557ccb0b6832393491a65f56a6ec56ee6f6c7df94c\": container with ID starting with a0dd85289c5aaebb787f34557ccb0b6832393491a65f56a6ec56ee6f6c7df94c not found: ID does not exist" containerID="a0dd85289c5aaebb787f34557ccb0b6832393491a65f56a6ec56ee6f6c7df94c" Oct 30 04:04:17.454771 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.454721 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0dd85289c5aaebb787f34557ccb0b6832393491a65f56a6ec56ee6f6c7df94c"} err="failed to get container status \"a0dd85289c5aaebb787f34557ccb0b6832393491a65f56a6ec56ee6f6c7df94c\": rpc error: code = NotFound desc = could not find container \"a0dd85289c5aaebb787f34557ccb0b6832393491a65f56a6ec56ee6f6c7df94c\": container with ID starting with a0dd85289c5aaebb787f34557ccb0b6832393491a65f56a6ec56ee6f6c7df94c not found: ID does not exist" Oct 30 04:04:17.504546 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.504519 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:17.507542 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.507517 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k"] Oct 30 04:04:17.508423 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:04:17.508399 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1826f77_7ce4_4e29_bd49_23a730e22587.slice/crio-1f2c934540daf61712824e48e290c0e4cbafff0fa64626c4755cd7e946d6eb5f WatchSource:0}: Error finding container 1f2c934540daf61712824e48e290c0e4cbafff0fa64626c4755cd7e946d6eb5f: Status 404 returned error can't find the container with id 1f2c934540daf61712824e48e290c0e4cbafff0fa64626c4755cd7e946d6eb5f Oct 30 04:04:17.510633 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.510595 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:17.510718 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.510656 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:17.518325 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.518301 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:17.580724 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.580694 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:17.583179 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.583161 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:17.607250 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.607215 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:04:17.607595 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.607570 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:04:17.607736 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.607669 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-45ch8\"" Oct 30 04:04:17.607804 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.607740 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:04:17.607886 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.607866 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:04:17.612248 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.612220 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:17.645753 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.645717 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:17.646089 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.645761 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:17.646089 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.645881 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:17.646089 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.645920 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29fsx\" (UniqueName: \"kubernetes.io/projected/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-kube-api-access-29fsx\") pod \"data-science-collector-collector-1\" (UID: \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:17.646089 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.645983 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:17.746854 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.746819 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:17.747063 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.746887 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29fsx\" (UniqueName: \"kubernetes.io/projected/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-kube-api-access-29fsx\") pod \"data-science-collector-collector-1\" (UID: \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:17.747063 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.746941 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:17.747203 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.747168 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:17.747277 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.747229 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:17.747661 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.747635 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:17.749422 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.749393 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:17.749586 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.749565 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:17.749841 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.749825 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:17.771964 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.771941 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29fsx\" (UniqueName: \"kubernetes.io/projected/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-kube-api-access-29fsx\") pod \"data-science-collector-collector-1\" (UID: \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:17.899263 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:17.899227 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:18.010579 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.010501 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="331e60cc-fbf6-4c1d-ab57-9c710fac3e99" path="/var/lib/kubelet/pods/331e60cc-fbf6-4c1d-ab57-9c710fac3e99/volumes" Oct 30 04:04:18.194308 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.194283 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:18.196027 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:04:18.195993 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a6ce70_10ab_464d_8ac7_e253eac8a64b.slice/crio-bcb5e32f74667a5a571858851e98cb30aef840aee4b99407b5c83937f0aa59ab WatchSource:0}: Error finding container bcb5e32f74667a5a571858851e98cb30aef840aee4b99407b5c83937f0aa59ab: Status 404 returned error can't find the container with id bcb5e32f74667a5a571858851e98cb30aef840aee4b99407b5c83937f0aa59ab Oct 30 04:04:18.198282 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.198253 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:18.198345 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.198311 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:18.452559 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.452468 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" event={"ID":"b1826f77-7ce4-4e29-bd49-23a730e22587","Type":"ContainerStarted","Data":"44910d2cfdf061d7db3c7f3ada2a18328840edb14e1ce4dbd0e4f8b5f3bda998"} Oct 30 04:04:18.452559 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.452517 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" event={"ID":"b1826f77-7ce4-4e29-bd49-23a730e22587","Type":"ContainerStarted","Data":"1f2c934540daf61712824e48e290c0e4cbafff0fa64626c4755cd7e946d6eb5f"} Oct 30 04:04:18.452811 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.452655 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" Oct 30 04:04:18.454330 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.454308 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" Oct 30 04:04:18.455602 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.455578 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d8a6ce70-10ab-464d-8ac7-e253eac8a64b","Type":"ContainerStarted","Data":"4a92021207e7bfc8978fad23a4299a16381387fcb2f85c57161fb654695b735f"} Oct 30 04:04:18.455731 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.455641 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d8a6ce70-10ab-464d-8ac7-e253eac8a64b","Type":"ContainerStarted","Data":"bcb5e32f74667a5a571858851e98cb30aef840aee4b99407b5c83937f0aa59ab"} Oct 30 04:04:18.497133 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.496865 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" podStartSLOduration=2.496851416 podStartE2EDuration="2.496851416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:18.496321664 +0000 UTC m=+4970.610636640" watchObservedRunningTime="2026-10-30 04:04:18.496851416 +0000 UTC m=+4970.611166385" Oct 30 04:04:18.542251 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.542187 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.5421690190000001 podStartE2EDuration="1.542169019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:18.541697905 +0000 UTC m=+4970.656012879" watchObservedRunningTime="2026-10-30 04:04:18.542169019 +0000 UTC m=+4970.656483994" Oct 30 04:04:18.617375 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.617340 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26"] Oct 30 04:04:18.617572 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.617547 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" podUID="0f856657-3fc0-4339-9fb9-02e4e193d103" containerName="ta-container" containerID="cri-o://76e4a369d11386c9322caabf73c60b5e996a0d67fb06a3ec9aca15c7479811be" gracePeriod=30 Oct 30 04:04:18.863233 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.863208 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" Oct 30 04:04:18.959533 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.959499 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f856657-3fc0-4339-9fb9-02e4e193d103-ta-internal\" (UniqueName: \"kubernetes.io/configmap/0f856657-3fc0-4339-9fb9-02e4e193d103-ta-internal\") pod \"0f856657-3fc0-4339-9fb9-02e4e193d103\" (UID: \"0f856657-3fc0-4339-9fb9-02e4e193d103\") " Oct 30 04:04:18.959725 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.959553 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f856657-3fc0-4339-9fb9-02e4e193d103-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/0f856657-3fc0-4339-9fb9-02e4e193d103-data-science-collector-ta-server-cert\") pod \"0f856657-3fc0-4339-9fb9-02e4e193d103\" (UID: \"0f856657-3fc0-4339-9fb9-02e4e193d103\") " Oct 30 04:04:18.959725 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.959645 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f856657-3fc0-4339-9fb9-02e4e193d103-kube-api-access-zv27l\" (UniqueName: \"kubernetes.io/projected/0f856657-3fc0-4339-9fb9-02e4e193d103-kube-api-access-zv27l\") pod \"0f856657-3fc0-4339-9fb9-02e4e193d103\" (UID: \"0f856657-3fc0-4339-9fb9-02e4e193d103\") " Oct 30 04:04:18.959920 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.959852 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f856657-3fc0-4339-9fb9-02e4e193d103-ta-internal" pod "0f856657-3fc0-4339-9fb9-02e4e193d103" (UID: "0f856657-3fc0-4339-9fb9-02e4e193d103"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:18.961702 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.961666 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f856657-3fc0-4339-9fb9-02e4e193d103-kube-api-access-zv27l" pod "0f856657-3fc0-4339-9fb9-02e4e193d103" (UID: "0f856657-3fc0-4339-9fb9-02e4e193d103"). InnerVolumeSpecName "kube-api-access-zv27l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:18.961702 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:18.961686 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f856657-3fc0-4339-9fb9-02e4e193d103-data-science-collector-ta-server-cert" pod "0f856657-3fc0-4339-9fb9-02e4e193d103" (UID: "0f856657-3fc0-4339-9fb9-02e4e193d103"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:19.060592 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:19.060493 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/0f856657-3fc0-4339-9fb9-02e4e193d103-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:19.060592 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:19.060530 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv27l\" (UniqueName: \"kubernetes.io/projected/0f856657-3fc0-4339-9fb9-02e4e193d103-kube-api-access-zv27l\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:19.060592 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:19.060546 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/0f856657-3fc0-4339-9fb9-02e4e193d103-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:19.461506 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:19.461425 5882 generic.go:356] "Generic (PLEG): container finished" podID="0f856657-3fc0-4339-9fb9-02e4e193d103" containerID="76e4a369d11386c9322caabf73c60b5e996a0d67fb06a3ec9aca15c7479811be" exitCode=0 Oct 30 04:04:19.461506 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:19.461470 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" Oct 30 04:04:19.461734 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:19.461510 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" event={"ID":"0f856657-3fc0-4339-9fb9-02e4e193d103","Type":"ContainerDied","Data":"76e4a369d11386c9322caabf73c60b5e996a0d67fb06a3ec9aca15c7479811be"} Oct 30 04:04:19.461734 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:19.461548 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26" event={"ID":"0f856657-3fc0-4339-9fb9-02e4e193d103","Type":"ContainerDied","Data":"45f665c69df02f78523b63db4e86d25822c87fa3930a7da76106156fce7790bb"} Oct 30 04:04:19.461734 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:19.461560 5882 scope.go:122] "RemoveContainer" containerID="76e4a369d11386c9322caabf73c60b5e996a0d67fb06a3ec9aca15c7479811be" Oct 30 04:04:19.471205 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:19.471187 5882 scope.go:122] "RemoveContainer" containerID="76e4a369d11386c9322caabf73c60b5e996a0d67fb06a3ec9aca15c7479811be" Oct 30 04:04:19.471471 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:04:19.471448 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e4a369d11386c9322caabf73c60b5e996a0d67fb06a3ec9aca15c7479811be\": container with ID starting with 76e4a369d11386c9322caabf73c60b5e996a0d67fb06a3ec9aca15c7479811be not found: ID does not exist" containerID="76e4a369d11386c9322caabf73c60b5e996a0d67fb06a3ec9aca15c7479811be" Oct 30 04:04:19.471545 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:19.471478 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e4a369d11386c9322caabf73c60b5e996a0d67fb06a3ec9aca15c7479811be"} err="failed to get container status \"76e4a369d11386c9322caabf73c60b5e996a0d67fb06a3ec9aca15c7479811be\": rpc error: code = NotFound desc = could not find container \"76e4a369d11386c9322caabf73c60b5e996a0d67fb06a3ec9aca15c7479811be\": container with ID starting with 76e4a369d11386c9322caabf73c60b5e996a0d67fb06a3ec9aca15c7479811be not found: ID does not exist" Oct 30 04:04:19.516604 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:19.516569 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26"] Oct 30 04:04:19.530355 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:19.530332 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-ff587cc8b-v9m26"] Oct 30 04:04:20.009689 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:20.009657 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f856657-3fc0-4339-9fb9-02e4e193d103" path="/var/lib/kubelet/pods/0f856657-3fc0-4339-9fb9-02e4e193d103/volumes" Oct 30 04:04:23.942336 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:04:23.942298 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:04:25.813307 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:25.813266 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:25.813760 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:25.813498 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="d8a6ce70-10ab-464d-8ac7-e253eac8a64b" containerName="otc-container" containerID="cri-o://4a92021207e7bfc8978fad23a4299a16381387fcb2f85c57161fb654695b735f" gracePeriod=30 Oct 30 04:04:26.040037 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:26.039995 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55"] Oct 30 04:04:26.040446 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:26.040427 5882 cpu_manager.go:414] "containerMap: removing container" podUID="0f856657-3fc0-4339-9fb9-02e4e193d103" containerName="ta-container" Oct 30 04:04:26.040536 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:26.040448 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f856657-3fc0-4339-9fb9-02e4e193d103" containerName="ta-container" Oct 30 04:04:26.040590 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:26.040572 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f856657-3fc0-4339-9fb9-02e4e193d103" containerName="ta-container" Oct 30 04:04:26.043560 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:26.043539 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" Oct 30 04:04:26.065443 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:26.065384 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55"] Oct 30 04:04:26.137655 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:26.137599 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99hj6\" (UniqueName: \"kubernetes.io/projected/631dffab-7a9c-403e-8d9c-c741e1af0055-kube-api-access-99hj6\") pod \"data-science-collector-targetallocator-59cc55784d-47d55\" (UID: \"631dffab-7a9c-403e-8d9c-c741e1af0055\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" Oct 30 04:04:26.137813 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:26.137736 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/631dffab-7a9c-403e-8d9c-c741e1af0055-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-59cc55784d-47d55\" (UID: \"631dffab-7a9c-403e-8d9c-c741e1af0055\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" Oct 30 04:04:26.137813 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:26.137798 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/631dffab-7a9c-403e-8d9c-c741e1af0055-ta-internal\") pod \"data-science-collector-targetallocator-59cc55784d-47d55\" (UID: \"631dffab-7a9c-403e-8d9c-c741e1af0055\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" Oct 30 04:04:26.238704 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:26.238664 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/631dffab-7a9c-403e-8d9c-c741e1af0055-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-59cc55784d-47d55\" (UID: \"631dffab-7a9c-403e-8d9c-c741e1af0055\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" Oct 30 04:04:26.238870 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:26.238816 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/631dffab-7a9c-403e-8d9c-c741e1af0055-ta-internal\") pod \"data-science-collector-targetallocator-59cc55784d-47d55\" (UID: \"631dffab-7a9c-403e-8d9c-c741e1af0055\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" Oct 30 04:04:26.238929 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:26.238871 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99hj6\" (UniqueName: \"kubernetes.io/projected/631dffab-7a9c-403e-8d9c-c741e1af0055-kube-api-access-99hj6\") pod \"data-science-collector-targetallocator-59cc55784d-47d55\" (UID: \"631dffab-7a9c-403e-8d9c-c741e1af0055\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" Oct 30 04:04:26.239506 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:26.239482 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/631dffab-7a9c-403e-8d9c-c741e1af0055-ta-internal\") pod \"data-science-collector-targetallocator-59cc55784d-47d55\" (UID: \"631dffab-7a9c-403e-8d9c-c741e1af0055\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" Oct 30 04:04:26.241136 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:26.241115 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/631dffab-7a9c-403e-8d9c-c741e1af0055-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-59cc55784d-47d55\" (UID: \"631dffab-7a9c-403e-8d9c-c741e1af0055\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" Oct 30 04:04:26.264017 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:26.263987 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99hj6\" (UniqueName: \"kubernetes.io/projected/631dffab-7a9c-403e-8d9c-c741e1af0055-kube-api-access-99hj6\") pod \"data-science-collector-targetallocator-59cc55784d-47d55\" (UID: \"631dffab-7a9c-403e-8d9c-c741e1af0055\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" Oct 30 04:04:26.353962 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:26.353883 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" Oct 30 04:04:26.647633 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:26.647581 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55"] Oct 30 04:04:26.648114 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:04:26.648085 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod631dffab_7a9c_403e_8d9c_c741e1af0055.slice/crio-337d9fbdc802958d60685d193ec54ad106db2dc86f22a89765dd11141c962d1b WatchSource:0}: Error finding container 337d9fbdc802958d60685d193ec54ad106db2dc86f22a89765dd11141c962d1b: Status 404 returned error can't find the container with id 337d9fbdc802958d60685d193ec54ad106db2dc86f22a89765dd11141c962d1b Oct 30 04:04:26.650213 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:26.650180 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:26.650275 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:26.650252 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:27.508384 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:27.508349 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" event={"ID":"631dffab-7a9c-403e-8d9c-c741e1af0055","Type":"ContainerStarted","Data":"7521f96d9ed33af040ffcc753b9d025a3c88f3497996f55eefed7a7445c93c29"} Oct 30 04:04:27.508811 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:27.508396 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" event={"ID":"631dffab-7a9c-403e-8d9c-c741e1af0055","Type":"ContainerStarted","Data":"337d9fbdc802958d60685d193ec54ad106db2dc86f22a89765dd11141c962d1b"} Oct 30 04:04:27.508811 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:27.508540 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" Oct 30 04:04:27.510311 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:27.510290 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" Oct 30 04:04:27.553499 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:27.553452 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" podStartSLOduration=1.5534400000000002 podStartE2EDuration="1.55344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:27.553158771 +0000 UTC m=+4979.667473743" watchObservedRunningTime="2026-10-30 04:04:27.55344 +0000 UTC m=+4979.667754953" Oct 30 04:04:27.626559 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:27.626528 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k"] Oct 30 04:04:27.626747 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:27.626723 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" podUID="b1826f77-7ce4-4e29-bd49-23a730e22587" containerName="ta-container" containerID="cri-o://44910d2cfdf061d7db3c7f3ada2a18328840edb14e1ce4dbd0e4f8b5f3bda998" gracePeriod=30 Oct 30 04:04:27.864594 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:27.864569 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" Oct 30 04:04:27.956181 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:27.956150 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1826f77-7ce4-4e29-bd49-23a730e22587-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b1826f77-7ce4-4e29-bd49-23a730e22587-data-science-collector-ta-server-cert\") pod \"b1826f77-7ce4-4e29-bd49-23a730e22587\" (UID: \"b1826f77-7ce4-4e29-bd49-23a730e22587\") " Oct 30 04:04:27.956331 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:27.956212 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1826f77-7ce4-4e29-bd49-23a730e22587-ta-internal\" (UniqueName: \"kubernetes.io/configmap/b1826f77-7ce4-4e29-bd49-23a730e22587-ta-internal\") pod \"b1826f77-7ce4-4e29-bd49-23a730e22587\" (UID: \"b1826f77-7ce4-4e29-bd49-23a730e22587\") " Oct 30 04:04:27.956331 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:27.956311 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1826f77-7ce4-4e29-bd49-23a730e22587-kube-api-access-gnxkx\" (UniqueName: \"kubernetes.io/projected/b1826f77-7ce4-4e29-bd49-23a730e22587-kube-api-access-gnxkx\") pod \"b1826f77-7ce4-4e29-bd49-23a730e22587\" (UID: \"b1826f77-7ce4-4e29-bd49-23a730e22587\") " Oct 30 04:04:27.956569 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:27.956547 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1826f77-7ce4-4e29-bd49-23a730e22587-ta-internal" pod "b1826f77-7ce4-4e29-bd49-23a730e22587" (UID: "b1826f77-7ce4-4e29-bd49-23a730e22587"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:27.956674 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:27.956658 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b1826f77-7ce4-4e29-bd49-23a730e22587-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:27.958232 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:27.958207 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1826f77-7ce4-4e29-bd49-23a730e22587-data-science-collector-ta-server-cert" pod "b1826f77-7ce4-4e29-bd49-23a730e22587" (UID: "b1826f77-7ce4-4e29-bd49-23a730e22587"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:27.958317 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:27.958229 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1826f77-7ce4-4e29-bd49-23a730e22587-kube-api-access-gnxkx" pod "b1826f77-7ce4-4e29-bd49-23a730e22587" (UID: "b1826f77-7ce4-4e29-bd49-23a730e22587"). InnerVolumeSpecName "kube-api-access-gnxkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:28.057803 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.057721 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnxkx\" (UniqueName: \"kubernetes.io/projected/b1826f77-7ce4-4e29-bd49-23a730e22587-kube-api-access-gnxkx\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:28.057803 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.057752 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b1826f77-7ce4-4e29-bd49-23a730e22587-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:28.157998 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.157973 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:28.258656 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.258612 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-tls-certs\") pod \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\" (UID: \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\") " Oct 30 04:04:28.258795 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.258675 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-data-science-collector-ta-client-cert\") pod \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\" (UID: \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\") " Oct 30 04:04:28.258839 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.258820 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-kube-api-access-29fsx\" (UniqueName: \"kubernetes.io/projected/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-kube-api-access-29fsx\") pod \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\" (UID: \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\") " Oct 30 04:04:28.258880 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.258856 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-monitoring-tls-certs\") pod \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\" (UID: \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\") " Oct 30 04:04:28.258942 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.258887 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-otc-internal\") pod \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\" (UID: \"d8a6ce70-10ab-464d-8ac7-e253eac8a64b\") " Oct 30 04:04:28.259323 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.259291 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-otc-internal" pod "d8a6ce70-10ab-464d-8ac7-e253eac8a64b" (UID: "d8a6ce70-10ab-464d-8ac7-e253eac8a64b"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:28.260830 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.260803 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-kube-api-access-29fsx" pod "d8a6ce70-10ab-464d-8ac7-e253eac8a64b" (UID: "d8a6ce70-10ab-464d-8ac7-e253eac8a64b"). InnerVolumeSpecName "kube-api-access-29fsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:28.260934 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.260871 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-data-science-collector-ta-client-cert" pod "d8a6ce70-10ab-464d-8ac7-e253eac8a64b" (UID: "d8a6ce70-10ab-464d-8ac7-e253eac8a64b"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:28.261125 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.261107 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-tls-certs" pod "d8a6ce70-10ab-464d-8ac7-e253eac8a64b" (UID: "d8a6ce70-10ab-464d-8ac7-e253eac8a64b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:28.261179 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.261160 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-monitoring-tls-certs" pod "d8a6ce70-10ab-464d-8ac7-e253eac8a64b" (UID: "d8a6ce70-10ab-464d-8ac7-e253eac8a64b"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:28.359868 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.359788 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29fsx\" (UniqueName: \"kubernetes.io/projected/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-kube-api-access-29fsx\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:28.359868 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.359820 5882 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-monitoring-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:28.359868 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.359831 5882 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-otc-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:28.359868 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.359840 5882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:28.359868 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.359851 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d8a6ce70-10ab-464d-8ac7-e253eac8a64b-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:28.514111 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.514077 5882 generic.go:356] "Generic (PLEG): container finished" podID="b1826f77-7ce4-4e29-bd49-23a730e22587" containerID="44910d2cfdf061d7db3c7f3ada2a18328840edb14e1ce4dbd0e4f8b5f3bda998" exitCode=0 Oct 30 04:04:28.514523 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.514124 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" Oct 30 04:04:28.514523 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.514160 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" event={"ID":"b1826f77-7ce4-4e29-bd49-23a730e22587","Type":"ContainerDied","Data":"44910d2cfdf061d7db3c7f3ada2a18328840edb14e1ce4dbd0e4f8b5f3bda998"} Oct 30 04:04:28.514523 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.514206 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k" event={"ID":"b1826f77-7ce4-4e29-bd49-23a730e22587","Type":"ContainerDied","Data":"1f2c934540daf61712824e48e290c0e4cbafff0fa64626c4755cd7e946d6eb5f"} Oct 30 04:04:28.514523 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.514232 5882 scope.go:122] "RemoveContainer" containerID="44910d2cfdf061d7db3c7f3ada2a18328840edb14e1ce4dbd0e4f8b5f3bda998" Oct 30 04:04:28.515507 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.515481 5882 generic.go:356] "Generic (PLEG): container finished" podID="d8a6ce70-10ab-464d-8ac7-e253eac8a64b" containerID="4a92021207e7bfc8978fad23a4299a16381387fcb2f85c57161fb654695b735f" exitCode=0 Oct 30 04:04:28.515637 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.515517 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d8a6ce70-10ab-464d-8ac7-e253eac8a64b","Type":"ContainerDied","Data":"4a92021207e7bfc8978fad23a4299a16381387fcb2f85c57161fb654695b735f"} Oct 30 04:04:28.515637 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.515532 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:28.515637 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.515583 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d8a6ce70-10ab-464d-8ac7-e253eac8a64b","Type":"ContainerDied","Data":"bcb5e32f74667a5a571858851e98cb30aef840aee4b99407b5c83937f0aa59ab"} Oct 30 04:04:28.524377 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.524352 5882 scope.go:122] "RemoveContainer" containerID="44910d2cfdf061d7db3c7f3ada2a18328840edb14e1ce4dbd0e4f8b5f3bda998" Oct 30 04:04:28.524680 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:04:28.524661 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44910d2cfdf061d7db3c7f3ada2a18328840edb14e1ce4dbd0e4f8b5f3bda998\": container with ID starting with 44910d2cfdf061d7db3c7f3ada2a18328840edb14e1ce4dbd0e4f8b5f3bda998 not found: ID does not exist" containerID="44910d2cfdf061d7db3c7f3ada2a18328840edb14e1ce4dbd0e4f8b5f3bda998" Oct 30 04:04:28.524775 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.524689 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44910d2cfdf061d7db3c7f3ada2a18328840edb14e1ce4dbd0e4f8b5f3bda998"} err="failed to get container status \"44910d2cfdf061d7db3c7f3ada2a18328840edb14e1ce4dbd0e4f8b5f3bda998\": rpc error: code = NotFound desc = could not find container \"44910d2cfdf061d7db3c7f3ada2a18328840edb14e1ce4dbd0e4f8b5f3bda998\": container with ID starting with 44910d2cfdf061d7db3c7f3ada2a18328840edb14e1ce4dbd0e4f8b5f3bda998 not found: ID does not exist" Oct 30 04:04:28.524775 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.524706 5882 scope.go:122] "RemoveContainer" containerID="4a92021207e7bfc8978fad23a4299a16381387fcb2f85c57161fb654695b735f" Oct 30 04:04:28.532711 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.532689 5882 scope.go:122] "RemoveContainer" containerID="4a92021207e7bfc8978fad23a4299a16381387fcb2f85c57161fb654695b735f" Oct 30 04:04:28.532985 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:04:28.532967 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a92021207e7bfc8978fad23a4299a16381387fcb2f85c57161fb654695b735f\": container with ID starting with 4a92021207e7bfc8978fad23a4299a16381387fcb2f85c57161fb654695b735f not found: ID does not exist" containerID="4a92021207e7bfc8978fad23a4299a16381387fcb2f85c57161fb654695b735f" Oct 30 04:04:28.533038 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.532994 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a92021207e7bfc8978fad23a4299a16381387fcb2f85c57161fb654695b735f"} err="failed to get container status \"4a92021207e7bfc8978fad23a4299a16381387fcb2f85c57161fb654695b735f\": rpc error: code = NotFound desc = could not find container \"4a92021207e7bfc8978fad23a4299a16381387fcb2f85c57161fb654695b735f\": container with ID starting with 4a92021207e7bfc8978fad23a4299a16381387fcb2f85c57161fb654695b735f not found: ID does not exist" Oct 30 04:04:28.569728 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.569690 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:28.583541 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.583512 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:28.610912 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.610851 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:28.611231 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.611218 5882 cpu_manager.go:414] "containerMap: removing container" podUID="b1826f77-7ce4-4e29-bd49-23a730e22587" containerName="ta-container" Oct 30 04:04:28.611268 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.611233 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1826f77-7ce4-4e29-bd49-23a730e22587" containerName="ta-container" Oct 30 04:04:28.611268 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.611242 5882 cpu_manager.go:414] "containerMap: removing container" podUID="d8a6ce70-10ab-464d-8ac7-e253eac8a64b" containerName="otc-container" Oct 30 04:04:28.611268 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.611247 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a6ce70-10ab-464d-8ac7-e253eac8a64b" containerName="otc-container" Oct 30 04:04:28.611365 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.611300 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1826f77-7ce4-4e29-bd49-23a730e22587" containerName="ta-container" Oct 30 04:04:28.611365 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.611307 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a6ce70-10ab-464d-8ac7-e253eac8a64b" containerName="otc-container" Oct 30 04:04:28.614547 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.614517 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k"] Oct 30 04:04:28.614739 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.614694 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:28.616451 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.616430 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6967c5f799-4d55k"] Oct 30 04:04:28.634438 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.634410 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:04:28.634562 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.634502 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:04:28.634798 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.634627 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-45ch8\"" Oct 30 04:04:28.634798 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.634772 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:04:28.635086 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.635067 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-72c79cac\"" Oct 30 04:04:28.635521 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.635485 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:28.764091 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.764055 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4zth\" (UniqueName: \"kubernetes.io/projected/3386baa6-cbf0-48f2-afa3-4e560b800718-kube-api-access-h4zth\") pod \"data-science-collector-collector-1\" (UID: \"3386baa6-cbf0-48f2-afa3-4e560b800718\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:28.764260 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.764112 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"3386baa6-cbf0-48f2-afa3-4e560b800718\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:28.764260 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.764197 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3386baa6-cbf0-48f2-afa3-4e560b800718\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:28.764338 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.764266 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3386baa6-cbf0-48f2-afa3-4e560b800718\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:28.764338 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.764296 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3386baa6-cbf0-48f2-afa3-4e560b800718-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"3386baa6-cbf0-48f2-afa3-4e560b800718\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:28.865706 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.865632 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3386baa6-cbf0-48f2-afa3-4e560b800718-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"3386baa6-cbf0-48f2-afa3-4e560b800718\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:28.865706 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.865698 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4zth\" (UniqueName: \"kubernetes.io/projected/3386baa6-cbf0-48f2-afa3-4e560b800718-kube-api-access-h4zth\") pod \"data-science-collector-collector-1\" (UID: \"3386baa6-cbf0-48f2-afa3-4e560b800718\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:28.865895 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.865739 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"3386baa6-cbf0-48f2-afa3-4e560b800718\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:28.865895 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.865774 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3386baa6-cbf0-48f2-afa3-4e560b800718\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:28.865895 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.865810 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3386baa6-cbf0-48f2-afa3-4e560b800718\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:28.866430 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.866403 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3386baa6-cbf0-48f2-afa3-4e560b800718-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"3386baa6-cbf0-48f2-afa3-4e560b800718\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:28.868362 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.868338 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3386baa6-cbf0-48f2-afa3-4e560b800718\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:28.868414 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.868380 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"3386baa6-cbf0-48f2-afa3-4e560b800718\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:28.868453 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.868436 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3386baa6-cbf0-48f2-afa3-4e560b800718\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:28.890902 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.890872 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4zth\" (UniqueName: \"kubernetes.io/projected/3386baa6-cbf0-48f2-afa3-4e560b800718-kube-api-access-h4zth\") pod \"data-science-collector-collector-1\" (UID: \"3386baa6-cbf0-48f2-afa3-4e560b800718\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:28.926739 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:28.926711 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:29.217353 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:29.217324 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:29.217463 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:04:29.217406 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3386baa6_cbf0_48f2_afa3_4e560b800718.slice/crio-035ee9bb087d25612b95cbd9403b4d350bf53d39aee990d5523421c173c312d9 WatchSource:0}: Error finding container 035ee9bb087d25612b95cbd9403b4d350bf53d39aee990d5523421c173c312d9: Status 404 returned error can't find the container with id 035ee9bb087d25612b95cbd9403b4d350bf53d39aee990d5523421c173c312d9 Oct 30 04:04:29.219571 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:29.219543 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:29.219664 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:29.219599 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:29.524600 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:29.524560 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3386baa6-cbf0-48f2-afa3-4e560b800718","Type":"ContainerStarted","Data":"d88dc536a93dcf4da735530056fa166c29e372ee24af7873d321cfffd063a965"} Oct 30 04:04:29.525110 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:29.524667 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3386baa6-cbf0-48f2-afa3-4e560b800718","Type":"ContainerStarted","Data":"035ee9bb087d25612b95cbd9403b4d350bf53d39aee990d5523421c173c312d9"} Oct 30 04:04:30.009631 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:30.009579 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1826f77-7ce4-4e29-bd49-23a730e22587" path="/var/lib/kubelet/pods/b1826f77-7ce4-4e29-bd49-23a730e22587/volumes" Oct 30 04:04:30.010087 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:30.010063 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8a6ce70-10ab-464d-8ac7-e253eac8a64b" path="/var/lib/kubelet/pods/d8a6ce70-10ab-464d-8ac7-e253eac8a64b/volumes" Oct 30 04:04:34.816077 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:34.816022 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=6.816006644 podStartE2EDuration="6.816006644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:29.570389076 +0000 UTC m=+4981.684704052" watchObservedRunningTime="2026-10-30 04:04:34.816006644 +0000 UTC m=+4986.930321619" Oct 30 04:04:34.816490 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:34.816461 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:34.816706 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:34.816680 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="3386baa6-cbf0-48f2-afa3-4e560b800718" containerName="otc-container" containerID="cri-o://d88dc536a93dcf4da735530056fa166c29e372ee24af7873d321cfffd063a965" gracePeriod=30 Oct 30 04:04:35.039686 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:35.039645 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd"] Oct 30 04:04:35.042666 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:35.042642 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" Oct 30 04:04:35.064672 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:35.064643 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd"] Oct 30 04:04:35.122233 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:35.122148 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/21def72f-e0cb-4438-8a67-eb38e5341d36-ta-internal\") pod \"data-science-collector-targetallocator-67cc54bdf-6stfd\" (UID: \"21def72f-e0cb-4438-8a67-eb38e5341d36\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" Oct 30 04:04:35.122233 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:35.122191 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d7ts\" (UniqueName: \"kubernetes.io/projected/21def72f-e0cb-4438-8a67-eb38e5341d36-kube-api-access-5d7ts\") pod \"data-science-collector-targetallocator-67cc54bdf-6stfd\" (UID: \"21def72f-e0cb-4438-8a67-eb38e5341d36\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" Oct 30 04:04:35.122420 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:35.122261 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/21def72f-e0cb-4438-8a67-eb38e5341d36-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-67cc54bdf-6stfd\" (UID: \"21def72f-e0cb-4438-8a67-eb38e5341d36\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" Oct 30 04:04:35.223410 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:35.223373 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/21def72f-e0cb-4438-8a67-eb38e5341d36-ta-internal\") pod \"data-science-collector-targetallocator-67cc54bdf-6stfd\" (UID: \"21def72f-e0cb-4438-8a67-eb38e5341d36\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" Oct 30 04:04:35.223410 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:35.223416 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d7ts\" (UniqueName: \"kubernetes.io/projected/21def72f-e0cb-4438-8a67-eb38e5341d36-kube-api-access-5d7ts\") pod \"data-science-collector-targetallocator-67cc54bdf-6stfd\" (UID: \"21def72f-e0cb-4438-8a67-eb38e5341d36\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" Oct 30 04:04:35.223609 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:35.223443 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/21def72f-e0cb-4438-8a67-eb38e5341d36-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-67cc54bdf-6stfd\" (UID: \"21def72f-e0cb-4438-8a67-eb38e5341d36\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" Oct 30 04:04:35.224122 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:35.224099 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/21def72f-e0cb-4438-8a67-eb38e5341d36-ta-internal\") pod \"data-science-collector-targetallocator-67cc54bdf-6stfd\" (UID: \"21def72f-e0cb-4438-8a67-eb38e5341d36\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" Oct 30 04:04:35.225963 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:35.225939 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/21def72f-e0cb-4438-8a67-eb38e5341d36-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-67cc54bdf-6stfd\" (UID: \"21def72f-e0cb-4438-8a67-eb38e5341d36\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" Oct 30 04:04:35.248380 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:35.248356 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d7ts\" (UniqueName: \"kubernetes.io/projected/21def72f-e0cb-4438-8a67-eb38e5341d36-kube-api-access-5d7ts\") pod \"data-science-collector-targetallocator-67cc54bdf-6stfd\" (UID: \"21def72f-e0cb-4438-8a67-eb38e5341d36\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" Oct 30 04:04:35.353383 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:35.353355 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" Oct 30 04:04:35.645025 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:35.644987 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd"] Oct 30 04:04:35.645784 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:04:35.645749 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21def72f_e0cb_4438_8a67_eb38e5341d36.slice/crio-2f4ad7b1e69dbd8c295d3cb39b0a7d18591cfcc0b721cd6a270552dd26cdc09a WatchSource:0}: Error finding container 2f4ad7b1e69dbd8c295d3cb39b0a7d18591cfcc0b721cd6a270552dd26cdc09a: Status 404 returned error can't find the container with id 2f4ad7b1e69dbd8c295d3cb39b0a7d18591cfcc0b721cd6a270552dd26cdc09a Oct 30 04:04:35.647940 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:35.647907 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:35.647990 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:35.647967 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:36.041822 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.041796 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:36.129652 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.129607 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3386baa6-cbf0-48f2-afa3-4e560b800718-otc-internal\" (UniqueName: \"kubernetes.io/configmap/3386baa6-cbf0-48f2-afa3-4e560b800718-otc-internal\") pod \"3386baa6-cbf0-48f2-afa3-4e560b800718\" (UID: \"3386baa6-cbf0-48f2-afa3-4e560b800718\") " Oct 30 04:04:36.129915 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.129666 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-tls-certs\" (UniqueName: \"kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-tls-certs\") pod \"3386baa6-cbf0-48f2-afa3-4e560b800718\" (UID: \"3386baa6-cbf0-48f2-afa3-4e560b800718\") " Oct 30 04:04:36.129915 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.129749 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3386baa6-cbf0-48f2-afa3-4e560b800718-kube-api-access-h4zth\" (UniqueName: \"kubernetes.io/projected/3386baa6-cbf0-48f2-afa3-4e560b800718-kube-api-access-h4zth\") pod \"3386baa6-cbf0-48f2-afa3-4e560b800718\" (UID: \"3386baa6-cbf0-48f2-afa3-4e560b800718\") " Oct 30 04:04:36.129915 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.129785 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-data-science-collector-ta-client-cert\") pod \"3386baa6-cbf0-48f2-afa3-4e560b800718\" (UID: \"3386baa6-cbf0-48f2-afa3-4e560b800718\") " Oct 30 04:04:36.129915 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.129857 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-monitoring-tls-certs\") pod \"3386baa6-cbf0-48f2-afa3-4e560b800718\" (UID: \"3386baa6-cbf0-48f2-afa3-4e560b800718\") " Oct 30 04:04:36.130149 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.130110 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3386baa6-cbf0-48f2-afa3-4e560b800718-otc-internal" pod "3386baa6-cbf0-48f2-afa3-4e560b800718" (UID: "3386baa6-cbf0-48f2-afa3-4e560b800718"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:36.131857 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.131831 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-monitoring-tls-certs" pod "3386baa6-cbf0-48f2-afa3-4e560b800718" (UID: "3386baa6-cbf0-48f2-afa3-4e560b800718"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:36.132071 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.132039 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-tls-certs" pod "3386baa6-cbf0-48f2-afa3-4e560b800718" (UID: "3386baa6-cbf0-48f2-afa3-4e560b800718"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:36.132071 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.132052 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3386baa6-cbf0-48f2-afa3-4e560b800718-kube-api-access-h4zth" pod "3386baa6-cbf0-48f2-afa3-4e560b800718" (UID: "3386baa6-cbf0-48f2-afa3-4e560b800718"). InnerVolumeSpecName "kube-api-access-h4zth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:36.132071 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.132064 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-data-science-collector-ta-client-cert" pod "3386baa6-cbf0-48f2-afa3-4e560b800718" (UID: "3386baa6-cbf0-48f2-afa3-4e560b800718"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:36.230742 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.230661 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4zth\" (UniqueName: \"kubernetes.io/projected/3386baa6-cbf0-48f2-afa3-4e560b800718-kube-api-access-h4zth\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:36.230742 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.230701 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:36.230742 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.230720 5882 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-monitoring-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:36.230742 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.230736 5882 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3386baa6-cbf0-48f2-afa3-4e560b800718-otc-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:36.230975 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.230752 5882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3386baa6-cbf0-48f2-afa3-4e560b800718-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:36.559820 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.559780 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" event={"ID":"21def72f-e0cb-4438-8a67-eb38e5341d36","Type":"ContainerStarted","Data":"c199d972c59045cbdba20403fd15c14ff4d29f9f5c1f8f0e9c2ba16c1bcff9b0"} Oct 30 04:04:36.559982 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.559839 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" Oct 30 04:04:36.559982 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.559856 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" event={"ID":"21def72f-e0cb-4438-8a67-eb38e5341d36","Type":"ContainerStarted","Data":"2f4ad7b1e69dbd8c295d3cb39b0a7d18591cfcc0b721cd6a270552dd26cdc09a"} Oct 30 04:04:36.561415 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.561380 5882 generic.go:356] "Generic (PLEG): container finished" podID="3386baa6-cbf0-48f2-afa3-4e560b800718" containerID="d88dc536a93dcf4da735530056fa166c29e372ee24af7873d321cfffd063a965" exitCode=0 Oct 30 04:04:36.561545 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.561426 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:36.561545 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.561454 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3386baa6-cbf0-48f2-afa3-4e560b800718","Type":"ContainerDied","Data":"d88dc536a93dcf4da735530056fa166c29e372ee24af7873d321cfffd063a965"} Oct 30 04:04:36.561545 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.561493 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3386baa6-cbf0-48f2-afa3-4e560b800718","Type":"ContainerDied","Data":"035ee9bb087d25612b95cbd9403b4d350bf53d39aee990d5523421c173c312d9"} Oct 30 04:04:36.561545 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.561524 5882 scope.go:122] "RemoveContainer" containerID="d88dc536a93dcf4da735530056fa166c29e372ee24af7873d321cfffd063a965" Oct 30 04:04:36.561871 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.561855 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" Oct 30 04:04:36.570371 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.570353 5882 scope.go:122] "RemoveContainer" containerID="d88dc536a93dcf4da735530056fa166c29e372ee24af7873d321cfffd063a965" Oct 30 04:04:36.570636 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:04:36.570602 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d88dc536a93dcf4da735530056fa166c29e372ee24af7873d321cfffd063a965\": container with ID starting with d88dc536a93dcf4da735530056fa166c29e372ee24af7873d321cfffd063a965 not found: ID does not exist" containerID="d88dc536a93dcf4da735530056fa166c29e372ee24af7873d321cfffd063a965" Oct 30 04:04:36.570692 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.570642 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d88dc536a93dcf4da735530056fa166c29e372ee24af7873d321cfffd063a965"} err="failed to get container status \"d88dc536a93dcf4da735530056fa166c29e372ee24af7873d321cfffd063a965\": rpc error: code = NotFound desc = could not find container \"d88dc536a93dcf4da735530056fa166c29e372ee24af7873d321cfffd063a965\": container with ID starting with d88dc536a93dcf4da735530056fa166c29e372ee24af7873d321cfffd063a965 not found: ID does not exist" Oct 30 04:04:36.604430 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.604380 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" podStartSLOduration=1.604368124 podStartE2EDuration="1.604368124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:36.60351389 +0000 UTC m=+4988.717828868" watchObservedRunningTime="2026-10-30 04:04:36.604368124 +0000 UTC m=+4988.718683098" Oct 30 04:04:36.676470 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.676438 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55"] Oct 30 04:04:36.676639 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.676601 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" podUID="631dffab-7a9c-403e-8d9c-c741e1af0055" containerName="ta-container" containerID="cri-o://7521f96d9ed33af040ffcc753b9d025a3c88f3497996f55eefed7a7445c93c29" gracePeriod=30 Oct 30 04:04:36.693986 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.693960 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:36.701464 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.701439 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:36.738445 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.738415 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:36.738808 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.738790 5882 cpu_manager.go:414] "containerMap: removing container" podUID="3386baa6-cbf0-48f2-afa3-4e560b800718" containerName="otc-container" Oct 30 04:04:36.738808 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.738806 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3386baa6-cbf0-48f2-afa3-4e560b800718" containerName="otc-container" Oct 30 04:04:36.738927 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.738894 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3386baa6-cbf0-48f2-afa3-4e560b800718" containerName="otc-container" Oct 30 04:04:36.745083 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.745059 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:36.762981 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.762946 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:04:36.763144 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.763013 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:04:36.763144 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.763088 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-45ch8\"" Oct 30 04:04:36.763275 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.763245 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:04:36.763339 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.763317 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:04:36.764197 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.764174 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:36.836492 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.836411 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:36.836628 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.836529 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:36.836628 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.836567 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:36.836707 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.836664 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh9pb\" (UniqueName: \"kubernetes.io/projected/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-kube-api-access-hh9pb\") pod \"data-science-collector-collector-1\" (UID: \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:36.836707 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.836685 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:36.917013 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.916990 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" Oct 30 04:04:36.937875 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.937853 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/631dffab-7a9c-403e-8d9c-c741e1af0055-kube-api-access-99hj6\" (UniqueName: \"kubernetes.io/projected/631dffab-7a9c-403e-8d9c-c741e1af0055-kube-api-access-99hj6\") pod \"631dffab-7a9c-403e-8d9c-c741e1af0055\" (UID: \"631dffab-7a9c-403e-8d9c-c741e1af0055\") " Oct 30 04:04:36.937981 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.937893 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/631dffab-7a9c-403e-8d9c-c741e1af0055-ta-internal\" (UniqueName: \"kubernetes.io/configmap/631dffab-7a9c-403e-8d9c-c741e1af0055-ta-internal\") pod \"631dffab-7a9c-403e-8d9c-c741e1af0055\" (UID: \"631dffab-7a9c-403e-8d9c-c741e1af0055\") " Oct 30 04:04:36.937981 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.937940 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/631dffab-7a9c-403e-8d9c-c741e1af0055-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/631dffab-7a9c-403e-8d9c-c741e1af0055-data-science-collector-ta-server-cert\") pod \"631dffab-7a9c-403e-8d9c-c741e1af0055\" (UID: \"631dffab-7a9c-403e-8d9c-c741e1af0055\") " Oct 30 04:04:36.938064 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.938056 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:36.938131 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.938119 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:36.938257 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.938231 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:36.938333 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.938287 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631dffab-7a9c-403e-8d9c-c741e1af0055-ta-internal" pod "631dffab-7a9c-403e-8d9c-c741e1af0055" (UID: "631dffab-7a9c-403e-8d9c-c741e1af0055"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:36.938333 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.938293 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh9pb\" (UniqueName: \"kubernetes.io/projected/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-kube-api-access-hh9pb\") pod \"data-science-collector-collector-1\" (UID: \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:36.938333 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.938321 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:36.938499 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.938395 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/631dffab-7a9c-403e-8d9c-c741e1af0055-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:36.938873 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.938848 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:36.940196 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.940166 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/631dffab-7a9c-403e-8d9c-c741e1af0055-data-science-collector-ta-server-cert" pod "631dffab-7a9c-403e-8d9c-c741e1af0055" (UID: "631dffab-7a9c-403e-8d9c-c741e1af0055"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:36.940284 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.940203 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631dffab-7a9c-403e-8d9c-c741e1af0055-kube-api-access-99hj6" pod "631dffab-7a9c-403e-8d9c-c741e1af0055" (UID: "631dffab-7a9c-403e-8d9c-c741e1af0055"). InnerVolumeSpecName "kube-api-access-99hj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:36.940815 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.940791 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:36.940958 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.940939 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:36.941017 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.940974 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:36.963954 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:36.963929 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh9pb\" (UniqueName: \"kubernetes.io/projected/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-kube-api-access-hh9pb\") pod \"data-science-collector-collector-1\" (UID: \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:37.039637 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:37.039590 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99hj6\" (UniqueName: \"kubernetes.io/projected/631dffab-7a9c-403e-8d9c-c741e1af0055-kube-api-access-99hj6\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:37.039637 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:37.039640 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/631dffab-7a9c-403e-8d9c-c741e1af0055-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:37.057547 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:37.057519 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:37.349975 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:37.349946 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:37.350081 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:04:37.349978 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04423bfc_5f36_44c9_89e8_0f0b1aeb914f.slice/crio-13d59818e4af01ffad01aeaed10a0d6347be5c1c517c994646121e0e0fc74749 WatchSource:0}: Error finding container 13d59818e4af01ffad01aeaed10a0d6347be5c1c517c994646121e0e0fc74749: Status 404 returned error can't find the container with id 13d59818e4af01ffad01aeaed10a0d6347be5c1c517c994646121e0e0fc74749 Oct 30 04:04:37.352365 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:37.352330 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:37.352440 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:37.352406 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:37.568562 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:37.568531 5882 generic.go:356] "Generic (PLEG): container finished" podID="631dffab-7a9c-403e-8d9c-c741e1af0055" containerID="7521f96d9ed33af040ffcc753b9d025a3c88f3497996f55eefed7a7445c93c29" exitCode=0 Oct 30 04:04:37.568750 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:37.568574 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" Oct 30 04:04:37.568750 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:37.568638 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" event={"ID":"631dffab-7a9c-403e-8d9c-c741e1af0055","Type":"ContainerDied","Data":"7521f96d9ed33af040ffcc753b9d025a3c88f3497996f55eefed7a7445c93c29"} Oct 30 04:04:37.568750 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:37.568678 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55" event={"ID":"631dffab-7a9c-403e-8d9c-c741e1af0055","Type":"ContainerDied","Data":"337d9fbdc802958d60685d193ec54ad106db2dc86f22a89765dd11141c962d1b"} Oct 30 04:04:37.568750 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:37.568696 5882 scope.go:122] "RemoveContainer" containerID="7521f96d9ed33af040ffcc753b9d025a3c88f3497996f55eefed7a7445c93c29" Oct 30 04:04:37.570467 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:37.570447 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"04423bfc-5f36-44c9-89e8-0f0b1aeb914f","Type":"ContainerStarted","Data":"97c0c437a2a33a8db13abc97677cefaa15231914b94f8fc844e4c908ce1747c5"} Oct 30 04:04:37.570591 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:37.570491 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"04423bfc-5f36-44c9-89e8-0f0b1aeb914f","Type":"ContainerStarted","Data":"13d59818e4af01ffad01aeaed10a0d6347be5c1c517c994646121e0e0fc74749"} Oct 30 04:04:37.577302 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:37.577285 5882 scope.go:122] "RemoveContainer" containerID="7521f96d9ed33af040ffcc753b9d025a3c88f3497996f55eefed7a7445c93c29" Oct 30 04:04:37.577552 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:04:37.577535 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7521f96d9ed33af040ffcc753b9d025a3c88f3497996f55eefed7a7445c93c29\": container with ID starting with 7521f96d9ed33af040ffcc753b9d025a3c88f3497996f55eefed7a7445c93c29 not found: ID does not exist" containerID="7521f96d9ed33af040ffcc753b9d025a3c88f3497996f55eefed7a7445c93c29" Oct 30 04:04:37.577594 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:37.577560 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7521f96d9ed33af040ffcc753b9d025a3c88f3497996f55eefed7a7445c93c29"} err="failed to get container status \"7521f96d9ed33af040ffcc753b9d025a3c88f3497996f55eefed7a7445c93c29\": rpc error: code = NotFound desc = could not find container \"7521f96d9ed33af040ffcc753b9d025a3c88f3497996f55eefed7a7445c93c29\": container with ID starting with 7521f96d9ed33af040ffcc753b9d025a3c88f3497996f55eefed7a7445c93c29 not found: ID does not exist" Oct 30 04:04:37.631395 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:37.631306 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.631290127 podStartE2EDuration="1.631290127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:37.630825378 +0000 UTC m=+4989.745140353" watchObservedRunningTime="2026-10-30 04:04:37.631290127 +0000 UTC m=+4989.745605101" Oct 30 04:04:37.673380 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:37.673349 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55"] Oct 30 04:04:37.680842 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:37.680812 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-59cc55784d-47d55"] Oct 30 04:04:38.011750 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:38.011714 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3386baa6-cbf0-48f2-afa3-4e560b800718" path="/var/lib/kubelet/pods/3386baa6-cbf0-48f2-afa3-4e560b800718/volumes" Oct 30 04:04:38.012113 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:38.012100 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="631dffab-7a9c-403e-8d9c-c741e1af0055" path="/var/lib/kubelet/pods/631dffab-7a9c-403e-8d9c-c741e1af0055/volumes" Oct 30 04:04:40.522749 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:40.522711 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7"] Oct 30 04:04:40.523158 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:40.522938 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7" podUID="f4dc26eb-1fe7-451e-aa4b-5380b08875ee" containerName="thanos-querier" containerID="cri-o://d47c9da69add40f72bd5bd52581bc941486d1b9f64e0fa2d4b6716a564ec4e12" gracePeriod=30 Oct 30 04:04:40.674784 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:40.674746 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:04:40.675489 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:40.674998 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="05dd6cb8-8908-4609-937d-80bb026a7937" containerName="alertmanager" containerID="cri-o://6f4d2d5a34244e9647294b45608893bda58876c135e5545e3a851212e49b3f2a" gracePeriod=120 Oct 30 04:04:40.675489 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:40.675364 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="05dd6cb8-8908-4609-937d-80bb026a7937" containerName="config-reloader" containerID="cri-o://1caf7eb6375cba7f47288ef8c74cd0ac29431c00f325e7018ba367429fd4735a" gracePeriod=120 Oct 30 04:04:40.798761 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:40.798737 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7" Oct 30 04:04:40.868659 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:40.868611 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4dc26eb-1fe7-451e-aa4b-5380b08875ee-kube-api-access-n965d\" (UniqueName: \"kubernetes.io/projected/f4dc26eb-1fe7-451e-aa4b-5380b08875ee-kube-api-access-n965d\") pod \"f4dc26eb-1fe7-451e-aa4b-5380b08875ee\" (UID: \"f4dc26eb-1fe7-451e-aa4b-5380b08875ee\") " Oct 30 04:04:40.870590 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:40.870569 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dc26eb-1fe7-451e-aa4b-5380b08875ee-kube-api-access-n965d" pod "f4dc26eb-1fe7-451e-aa4b-5380b08875ee" (UID: "f4dc26eb-1fe7-451e-aa4b-5380b08875ee"). InnerVolumeSpecName "kube-api-access-n965d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:40.969470 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:40.969435 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n965d\" (UniqueName: \"kubernetes.io/projected/f4dc26eb-1fe7-451e-aa4b-5380b08875ee-kube-api-access-n965d\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.357275 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.357244 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:41.357506 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.357459 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="04423bfc-5f36-44c9-89e8-0f0b1aeb914f" containerName="otc-container" containerID="cri-o://97c0c437a2a33a8db13abc97677cefaa15231914b94f8fc844e4c908ce1747c5" gracePeriod=30 Oct 30 04:04:41.403470 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.403435 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd"] Oct 30 04:04:41.403699 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.403666 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" podUID="21def72f-e0cb-4438-8a67-eb38e5341d36" containerName="ta-container" containerID="cri-o://c199d972c59045cbdba20403fd15c14ff4d29f9f5c1f8f0e9c2ba16c1bcff9b0" gracePeriod=30 Oct 30 04:04:41.523920 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.523887 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:04:41.524322 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.524110 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="555eacfa-6f9e-4efd-8fec-8b38c8c50283" containerName="perses" containerID="cri-o://bf8d2de4eda133b7faf8f8777b9261c576a56de2b72aa89ffcc3dee40e754ada" gracePeriod=30 Oct 30 04:04:41.594005 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.593972 5882 generic.go:356] "Generic (PLEG): container finished" podID="04423bfc-5f36-44c9-89e8-0f0b1aeb914f" containerID="97c0c437a2a33a8db13abc97677cefaa15231914b94f8fc844e4c908ce1747c5" exitCode=0 Oct 30 04:04:41.594151 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.594047 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"04423bfc-5f36-44c9-89e8-0f0b1aeb914f","Type":"ContainerDied","Data":"97c0c437a2a33a8db13abc97677cefaa15231914b94f8fc844e4c908ce1747c5"} Oct 30 04:04:41.596365 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.596337 5882 generic.go:356] "Generic (PLEG): container finished" podID="05dd6cb8-8908-4609-937d-80bb026a7937" containerID="1caf7eb6375cba7f47288ef8c74cd0ac29431c00f325e7018ba367429fd4735a" exitCode=0 Oct 30 04:04:41.596435 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.596368 5882 generic.go:356] "Generic (PLEG): container finished" podID="05dd6cb8-8908-4609-937d-80bb026a7937" containerID="6f4d2d5a34244e9647294b45608893bda58876c135e5545e3a851212e49b3f2a" exitCode=0 Oct 30 04:04:41.596435 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.596399 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"05dd6cb8-8908-4609-937d-80bb026a7937","Type":"ContainerDied","Data":"1caf7eb6375cba7f47288ef8c74cd0ac29431c00f325e7018ba367429fd4735a"} Oct 30 04:04:41.596520 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.596445 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"05dd6cb8-8908-4609-937d-80bb026a7937","Type":"ContainerDied","Data":"6f4d2d5a34244e9647294b45608893bda58876c135e5545e3a851212e49b3f2a"} Oct 30 04:04:41.597721 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.597680 5882 generic.go:356] "Generic (PLEG): container finished" podID="f4dc26eb-1fe7-451e-aa4b-5380b08875ee" containerID="d47c9da69add40f72bd5bd52581bc941486d1b9f64e0fa2d4b6716a564ec4e12" exitCode=0 Oct 30 04:04:41.597859 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.597766 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7" event={"ID":"f4dc26eb-1fe7-451e-aa4b-5380b08875ee","Type":"ContainerDied","Data":"d47c9da69add40f72bd5bd52581bc941486d1b9f64e0fa2d4b6716a564ec4e12"} Oct 30 04:04:41.597859 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.597795 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7" event={"ID":"f4dc26eb-1fe7-451e-aa4b-5380b08875ee","Type":"ContainerDied","Data":"5df0676ea2cdfd6c45b8392b23a73857c5170651d9de3d97faf39f3de4460f94"} Oct 30 04:04:41.597859 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.597817 5882 scope.go:122] "RemoveContainer" containerID="d47c9da69add40f72bd5bd52581bc941486d1b9f64e0fa2d4b6716a564ec4e12" Oct 30 04:04:41.597980 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.597963 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7" Oct 30 04:04:41.605479 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.605457 5882 generic.go:356] "Generic (PLEG): container finished" podID="21def72f-e0cb-4438-8a67-eb38e5341d36" containerID="c199d972c59045cbdba20403fd15c14ff4d29f9f5c1f8f0e9c2ba16c1bcff9b0" exitCode=0 Oct 30 04:04:41.605556 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.605522 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" event={"ID":"21def72f-e0cb-4438-8a67-eb38e5341d36","Type":"ContainerDied","Data":"c199d972c59045cbdba20403fd15c14ff4d29f9f5c1f8f0e9c2ba16c1bcff9b0"} Oct 30 04:04:41.658461 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.658438 5882 scope.go:122] "RemoveContainer" containerID="d47c9da69add40f72bd5bd52581bc941486d1b9f64e0fa2d4b6716a564ec4e12" Oct 30 04:04:41.658827 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:04:41.658805 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47c9da69add40f72bd5bd52581bc941486d1b9f64e0fa2d4b6716a564ec4e12\": container with ID starting with d47c9da69add40f72bd5bd52581bc941486d1b9f64e0fa2d4b6716a564ec4e12 not found: ID does not exist" containerID="d47c9da69add40f72bd5bd52581bc941486d1b9f64e0fa2d4b6716a564ec4e12" Oct 30 04:04:41.658935 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.658837 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47c9da69add40f72bd5bd52581bc941486d1b9f64e0fa2d4b6716a564ec4e12"} err="failed to get container status \"d47c9da69add40f72bd5bd52581bc941486d1b9f64e0fa2d4b6716a564ec4e12\": rpc error: code = NotFound desc = could not find container \"d47c9da69add40f72bd5bd52581bc941486d1b9f64e0fa2d4b6716a564ec4e12\": container with ID starting with d47c9da69add40f72bd5bd52581bc941486d1b9f64e0fa2d4b6716a564ec4e12 not found: ID does not exist" Oct 30 04:04:41.662960 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.662930 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:41.674629 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.674588 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-cluster-tls-config\") pod \"05dd6cb8-8908-4609-937d-80bb026a7937\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " Oct 30 04:04:41.674730 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.674661 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-config-volume\" (UniqueName: \"kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-config-volume\") pod \"05dd6cb8-8908-4609-937d-80bb026a7937\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " Oct 30 04:04:41.674730 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.674691 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05dd6cb8-8908-4609-937d-80bb026a7937-tls-assets\" (UniqueName: \"kubernetes.io/projected/05dd6cb8-8908-4609-937d-80bb026a7937-tls-assets\") pod \"05dd6cb8-8908-4609-937d-80bb026a7937\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " Oct 30 04:04:41.674730 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.674716 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-web-config\" (UniqueName: \"kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-web-config\") pod \"05dd6cb8-8908-4609-937d-80bb026a7937\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " Oct 30 04:04:41.674861 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.674772 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05dd6cb8-8908-4609-937d-80bb026a7937-config-out\" (UniqueName: \"kubernetes.io/empty-dir/05dd6cb8-8908-4609-937d-80bb026a7937-config-out\") pod \"05dd6cb8-8908-4609-937d-80bb026a7937\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " Oct 30 04:04:41.674861 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.674837 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05dd6cb8-8908-4609-937d-80bb026a7937-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/05dd6cb8-8908-4609-937d-80bb026a7937-alertmanager-data-science-monitoringstack-db\") pod \"05dd6cb8-8908-4609-937d-80bb026a7937\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " Oct 30 04:04:41.674959 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.674872 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05dd6cb8-8908-4609-937d-80bb026a7937-kube-api-access-zvk8s\" (UniqueName: \"kubernetes.io/projected/05dd6cb8-8908-4609-937d-80bb026a7937-kube-api-access-zvk8s\") pod \"05dd6cb8-8908-4609-937d-80bb026a7937\" (UID: \"05dd6cb8-8908-4609-937d-80bb026a7937\") " Oct 30 04:04:41.675439 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.675414 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05dd6cb8-8908-4609-937d-80bb026a7937-alertmanager-data-science-monitoringstack-db" pod "05dd6cb8-8908-4609-937d-80bb026a7937" (UID: "05dd6cb8-8908-4609-937d-80bb026a7937"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:04:41.677519 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.677492 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05dd6cb8-8908-4609-937d-80bb026a7937-kube-api-access-zvk8s" pod "05dd6cb8-8908-4609-937d-80bb026a7937" (UID: "05dd6cb8-8908-4609-937d-80bb026a7937"). InnerVolumeSpecName "kube-api-access-zvk8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:41.677653 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.677565 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05dd6cb8-8908-4609-937d-80bb026a7937-tls-assets" pod "05dd6cb8-8908-4609-937d-80bb026a7937" (UID: "05dd6cb8-8908-4609-937d-80bb026a7937"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:41.677716 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.677676 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-config-volume" pod "05dd6cb8-8908-4609-937d-80bb026a7937" (UID: "05dd6cb8-8908-4609-937d-80bb026a7937"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:41.679860 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.679321 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05dd6cb8-8908-4609-937d-80bb026a7937-config-out" pod "05dd6cb8-8908-4609-937d-80bb026a7937" (UID: "05dd6cb8-8908-4609-937d-80bb026a7937"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:04:41.683414 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.683390 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-cluster-tls-config" pod "05dd6cb8-8908-4609-937d-80bb026a7937" (UID: "05dd6cb8-8908-4609-937d-80bb026a7937"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:41.691957 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.691885 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-web-config" pod "05dd6cb8-8908-4609-937d-80bb026a7937" (UID: "05dd6cb8-8908-4609-937d-80bb026a7937"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:41.701405 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.701378 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7"] Oct 30 04:04:41.704989 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.704963 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-zvcm7"] Oct 30 04:04:41.709906 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.706510 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:41.709906 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.709379 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" Oct 30 04:04:41.771453 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.771429 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:41.775698 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.775671 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21def72f-e0cb-4438-8a67-eb38e5341d36-ta-internal\" (UniqueName: \"kubernetes.io/configmap/21def72f-e0cb-4438-8a67-eb38e5341d36-ta-internal\") pod \"21def72f-e0cb-4438-8a67-eb38e5341d36\" (UID: \"21def72f-e0cb-4438-8a67-eb38e5341d36\") " Oct 30 04:04:41.775827 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.775708 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21def72f-e0cb-4438-8a67-eb38e5341d36-kube-api-access-5d7ts\" (UniqueName: \"kubernetes.io/projected/21def72f-e0cb-4438-8a67-eb38e5341d36-kube-api-access-5d7ts\") pod \"21def72f-e0cb-4438-8a67-eb38e5341d36\" (UID: \"21def72f-e0cb-4438-8a67-eb38e5341d36\") " Oct 30 04:04:41.775827 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.775727 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-data-science-collector-ta-client-cert\") pod \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\" (UID: \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\") " Oct 30 04:04:41.775827 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.775752 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-tls-certs\" (UniqueName: \"kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-tls-certs\") pod \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\" (UID: \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\") " Oct 30 04:04:41.775827 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.775782 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/555eacfa-6f9e-4efd-8fec-8b38c8c50283-config\" (UniqueName: \"kubernetes.io/configmap/555eacfa-6f9e-4efd-8fec-8b38c8c50283-config\") pod \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\" (UID: \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\") " Oct 30 04:04:41.775827 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.775800 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-otc-internal\" (UniqueName: \"kubernetes.io/configmap/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-otc-internal\") pod \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\" (UID: \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\") " Oct 30 04:04:41.775827 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.775823 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-kube-api-access-hh9pb\" (UniqueName: \"kubernetes.io/projected/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-kube-api-access-hh9pb\") pod \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\" (UID: \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\") " Oct 30 04:04:41.776208 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.775875 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/555eacfa-6f9e-4efd-8fec-8b38c8c50283-plugins\" (UniqueName: \"kubernetes.io/empty-dir/555eacfa-6f9e-4efd-8fec-8b38c8c50283-plugins\") pod \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\" (UID: \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\") " Oct 30 04:04:41.776208 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.775966 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\" (UID: \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\") " Oct 30 04:04:41.776208 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.776010 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-monitoring-tls-certs\") pod \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\" (UID: \"04423bfc-5f36-44c9-89e8-0f0b1aeb914f\") " Oct 30 04:04:41.776208 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.776056 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21def72f-e0cb-4438-8a67-eb38e5341d36-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/21def72f-e0cb-4438-8a67-eb38e5341d36-data-science-collector-ta-server-cert\") pod \"21def72f-e0cb-4438-8a67-eb38e5341d36\" (UID: \"21def72f-e0cb-4438-8a67-eb38e5341d36\") " Oct 30 04:04:41.776208 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.776078 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21def72f-e0cb-4438-8a67-eb38e5341d36-ta-internal" pod "21def72f-e0cb-4438-8a67-eb38e5341d36" (UID: "21def72f-e0cb-4438-8a67-eb38e5341d36"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:41.776208 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.776117 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555eacfa-6f9e-4efd-8fec-8b38c8c50283-kube-api-access-rcb42\" (UniqueName: \"kubernetes.io/projected/555eacfa-6f9e-4efd-8fec-8b38c8c50283-kube-api-access-rcb42\") pod \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\" (UID: \"555eacfa-6f9e-4efd-8fec-8b38c8c50283\") " Oct 30 04:04:41.776208 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.776143 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555eacfa-6f9e-4efd-8fec-8b38c8c50283-config" pod "555eacfa-6f9e-4efd-8fec-8b38c8c50283" (UID: "555eacfa-6f9e-4efd-8fec-8b38c8c50283"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:41.776547 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.776355 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/555eacfa-6f9e-4efd-8fec-8b38c8c50283-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.776547 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.776375 5882 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/05dd6cb8-8908-4609-937d-80bb026a7937-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.776547 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.776391 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvk8s\" (UniqueName: \"kubernetes.io/projected/05dd6cb8-8908-4609-937d-80bb026a7937-kube-api-access-zvk8s\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.776547 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.776406 5882 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-cluster-tls-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.776547 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.776420 5882 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-config-volume\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.776547 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.776437 5882 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05dd6cb8-8908-4609-937d-80bb026a7937-tls-assets\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.776547 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.776450 5882 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05dd6cb8-8908-4609-937d-80bb026a7937-web-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.776547 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.776461 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/21def72f-e0cb-4438-8a67-eb38e5341d36-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.776547 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.776472 5882 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05dd6cb8-8908-4609-937d-80bb026a7937-config-out\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.777187 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.777109 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-otc-internal" pod "04423bfc-5f36-44c9-89e8-0f0b1aeb914f" (UID: "04423bfc-5f36-44c9-89e8-0f0b1aeb914f"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:41.780541 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.780519 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21def72f-e0cb-4438-8a67-eb38e5341d36-kube-api-access-5d7ts" pod "21def72f-e0cb-4438-8a67-eb38e5341d36" (UID: "21def72f-e0cb-4438-8a67-eb38e5341d36"). InnerVolumeSpecName "kube-api-access-5d7ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:41.780941 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.780920 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555eacfa-6f9e-4efd-8fec-8b38c8c50283-kube-api-access-rcb42" pod "555eacfa-6f9e-4efd-8fec-8b38c8c50283" (UID: "555eacfa-6f9e-4efd-8fec-8b38c8c50283"). InnerVolumeSpecName "kube-api-access-rcb42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:41.781173 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.781153 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e" pod "555eacfa-6f9e-4efd-8fec-8b38c8c50283" (UID: "555eacfa-6f9e-4efd-8fec-8b38c8c50283"). InnerVolumeSpecName "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:04:41.781253 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.781214 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-kube-api-access-hh9pb" pod "04423bfc-5f36-44c9-89e8-0f0b1aeb914f" (UID: "04423bfc-5f36-44c9-89e8-0f0b1aeb914f"). InnerVolumeSpecName "kube-api-access-hh9pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:41.781308 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.781254 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-data-science-collector-ta-client-cert" pod "04423bfc-5f36-44c9-89e8-0f0b1aeb914f" (UID: "04423bfc-5f36-44c9-89e8-0f0b1aeb914f"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:41.782089 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.782068 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21def72f-e0cb-4438-8a67-eb38e5341d36-data-science-collector-ta-server-cert" pod "21def72f-e0cb-4438-8a67-eb38e5341d36" (UID: "21def72f-e0cb-4438-8a67-eb38e5341d36"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:41.782218 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.782200 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-tls-certs" pod "04423bfc-5f36-44c9-89e8-0f0b1aeb914f" (UID: "04423bfc-5f36-44c9-89e8-0f0b1aeb914f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:41.782253 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.782226 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-monitoring-tls-certs" pod "04423bfc-5f36-44c9-89e8-0f0b1aeb914f" (UID: "04423bfc-5f36-44c9-89e8-0f0b1aeb914f"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:41.877099 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.877016 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcb42\" (UniqueName: \"kubernetes.io/projected/555eacfa-6f9e-4efd-8fec-8b38c8c50283-kube-api-access-rcb42\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.877099 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.877061 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d7ts\" (UniqueName: \"kubernetes.io/projected/21def72f-e0cb-4438-8a67-eb38e5341d36-kube-api-access-5d7ts\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.877099 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.877077 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.877099 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.877092 5882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.877412 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.877106 5882 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-otc-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.877412 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.877118 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh9pb\" (UniqueName: \"kubernetes.io/projected/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-kube-api-access-hh9pb\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.877412 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.877153 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:04:41.877412 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.877168 5882 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/04423bfc-5f36-44c9-89e8-0f0b1aeb914f-monitoring-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.877412 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.877180 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/21def72f-e0cb-4438-8a67-eb38e5341d36-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.890268 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.890248 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:04:41.929823 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.929785 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/555eacfa-6f9e-4efd-8fec-8b38c8c50283-plugins" pod "555eacfa-6f9e-4efd-8fec-8b38c8c50283" (UID: "555eacfa-6f9e-4efd-8fec-8b38c8c50283"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:04:41.978224 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.978189 5882 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/555eacfa-6f9e-4efd-8fec-8b38c8c50283-plugins\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.978224 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:41.978219 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" Oct 30 04:04:42.009273 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.009233 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4dc26eb-1fe7-451e-aa4b-5380b08875ee" path="/var/lib/kubelet/pods/f4dc26eb-1fe7-451e-aa4b-5380b08875ee/volumes" Oct 30 04:04:42.611976 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.611944 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:42.612469 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.611947 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"04423bfc-5f36-44c9-89e8-0f0b1aeb914f","Type":"ContainerDied","Data":"13d59818e4af01ffad01aeaed10a0d6347be5c1c517c994646121e0e0fc74749"} Oct 30 04:04:42.612469 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.612084 5882 scope.go:122] "RemoveContainer" containerID="97c0c437a2a33a8db13abc97677cefaa15231914b94f8fc844e4c908ce1747c5" Oct 30 04:04:42.614396 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.614328 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:42.614512 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.614334 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"05dd6cb8-8908-4609-937d-80bb026a7937","Type":"ContainerDied","Data":"4fb7a461e30c88137dfca19583a7533570c1e976d15749eec09d8eedae107656"} Oct 30 04:04:42.616001 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.615974 5882 generic.go:356] "Generic (PLEG): container finished" podID="555eacfa-6f9e-4efd-8fec-8b38c8c50283" containerID="bf8d2de4eda133b7faf8f8777b9261c576a56de2b72aa89ffcc3dee40e754ada" exitCode=0 Oct 30 04:04:42.616125 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.616014 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:42.616125 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.616066 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"555eacfa-6f9e-4efd-8fec-8b38c8c50283","Type":"ContainerDied","Data":"bf8d2de4eda133b7faf8f8777b9261c576a56de2b72aa89ffcc3dee40e754ada"} Oct 30 04:04:42.616125 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.616102 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"555eacfa-6f9e-4efd-8fec-8b38c8c50283","Type":"ContainerDied","Data":"743a0746af7dc19b1045ff8d59a727f4912962c1349296cd80796d8dc7814c8c"} Oct 30 04:04:42.618518 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.618497 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" Oct 30 04:04:42.618645 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.618495 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd" event={"ID":"21def72f-e0cb-4438-8a67-eb38e5341d36","Type":"ContainerDied","Data":"2f4ad7b1e69dbd8c295d3cb39b0a7d18591cfcc0b721cd6a270552dd26cdc09a"} Oct 30 04:04:42.624901 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.624878 5882 scope.go:122] "RemoveContainer" containerID="1caf7eb6375cba7f47288ef8c74cd0ac29431c00f325e7018ba367429fd4735a" Oct 30 04:04:42.633608 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.633583 5882 scope.go:122] "RemoveContainer" containerID="6f4d2d5a34244e9647294b45608893bda58876c135e5545e3a851212e49b3f2a" Oct 30 04:04:42.641596 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.641577 5882 scope.go:122] "RemoveContainer" containerID="1db668d90a12f794e32357fb5a0e1402633a4f13679b7e6132d5e60a9c2df932" Oct 30 04:04:42.648623 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.648595 5882 scope.go:122] "RemoveContainer" containerID="bf8d2de4eda133b7faf8f8777b9261c576a56de2b72aa89ffcc3dee40e754ada" Oct 30 04:04:42.655891 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.655872 5882 scope.go:122] "RemoveContainer" containerID="bf8d2de4eda133b7faf8f8777b9261c576a56de2b72aa89ffcc3dee40e754ada" Oct 30 04:04:42.656140 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:04:42.656123 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf8d2de4eda133b7faf8f8777b9261c576a56de2b72aa89ffcc3dee40e754ada\": container with ID starting with bf8d2de4eda133b7faf8f8777b9261c576a56de2b72aa89ffcc3dee40e754ada not found: ID does not exist" containerID="bf8d2de4eda133b7faf8f8777b9261c576a56de2b72aa89ffcc3dee40e754ada" Oct 30 04:04:42.656201 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.656147 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf8d2de4eda133b7faf8f8777b9261c576a56de2b72aa89ffcc3dee40e754ada"} err="failed to get container status \"bf8d2de4eda133b7faf8f8777b9261c576a56de2b72aa89ffcc3dee40e754ada\": rpc error: code = NotFound desc = could not find container \"bf8d2de4eda133b7faf8f8777b9261c576a56de2b72aa89ffcc3dee40e754ada\": container with ID starting with bf8d2de4eda133b7faf8f8777b9261c576a56de2b72aa89ffcc3dee40e754ada not found: ID does not exist" Oct 30 04:04:42.656201 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.656164 5882 scope.go:122] "RemoveContainer" containerID="c199d972c59045cbdba20403fd15c14ff4d29f9f5c1f8f0e9c2ba16c1bcff9b0" Oct 30 04:04:42.676732 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.676711 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd"] Oct 30 04:04:42.680794 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.680768 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-67cc54bdf-6stfd"] Oct 30 04:04:42.719224 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.719197 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:42.726684 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.726656 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:42.765195 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.765170 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:04:42.769355 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.769332 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:04:42.807114 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.807048 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:04:42.810940 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:42.810910 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:04:44.010443 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:44.010406 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04423bfc-5f36-44c9-89e8-0f0b1aeb914f" path="/var/lib/kubelet/pods/04423bfc-5f36-44c9-89e8-0f0b1aeb914f/volumes" Oct 30 04:04:44.010978 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:44.010959 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05dd6cb8-8908-4609-937d-80bb026a7937" path="/var/lib/kubelet/pods/05dd6cb8-8908-4609-937d-80bb026a7937/volumes" Oct 30 04:04:44.011433 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:44.011418 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21def72f-e0cb-4438-8a67-eb38e5341d36" path="/var/lib/kubelet/pods/21def72f-e0cb-4438-8a67-eb38e5341d36/volumes" Oct 30 04:04:44.012090 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:44.012075 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="555eacfa-6f9e-4efd-8fec-8b38c8c50283" path="/var/lib/kubelet/pods/555eacfa-6f9e-4efd-8fec-8b38c8c50283/volumes" Oct 30 04:04:51.668384 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.668346 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk"] Oct 30 04:04:51.668885 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.668852 5882 cpu_manager.go:414] "containerMap: removing container" podUID="631dffab-7a9c-403e-8d9c-c741e1af0055" containerName="ta-container" Oct 30 04:04:51.668885 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.668868 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631dffab-7a9c-403e-8d9c-c741e1af0055" containerName="ta-container" Oct 30 04:04:51.668965 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.668903 5882 cpu_manager.go:414] "containerMap: removing container" podUID="555eacfa-6f9e-4efd-8fec-8b38c8c50283" containerName="perses" Oct 30 04:04:51.668965 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.668911 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555eacfa-6f9e-4efd-8fec-8b38c8c50283" containerName="perses" Oct 30 04:04:51.668965 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.668928 5882 cpu_manager.go:414] "containerMap: removing container" podUID="05dd6cb8-8908-4609-937d-80bb026a7937" containerName="config-reloader" Oct 30 04:04:51.668965 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.668936 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05dd6cb8-8908-4609-937d-80bb026a7937" containerName="config-reloader" Oct 30 04:04:51.668965 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.668955 5882 cpu_manager.go:414] "containerMap: removing container" podUID="f4dc26eb-1fe7-451e-aa4b-5380b08875ee" containerName="thanos-querier" Oct 30 04:04:51.668965 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.668962 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4dc26eb-1fe7-451e-aa4b-5380b08875ee" containerName="thanos-querier" Oct 30 04:04:51.669159 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.668972 5882 cpu_manager.go:414] "containerMap: removing container" podUID="05dd6cb8-8908-4609-937d-80bb026a7937" containerName="init-config-reloader" Oct 30 04:04:51.669159 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.668980 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05dd6cb8-8908-4609-937d-80bb026a7937" containerName="init-config-reloader" Oct 30 04:04:51.669159 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.668991 5882 cpu_manager.go:414] "containerMap: removing container" podUID="05dd6cb8-8908-4609-937d-80bb026a7937" containerName="alertmanager" Oct 30 04:04:51.669159 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.668998 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05dd6cb8-8908-4609-937d-80bb026a7937" containerName="alertmanager" Oct 30 04:04:51.669159 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.669010 5882 cpu_manager.go:414] "containerMap: removing container" podUID="04423bfc-5f36-44c9-89e8-0f0b1aeb914f" containerName="otc-container" Oct 30 04:04:51.669159 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.669017 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04423bfc-5f36-44c9-89e8-0f0b1aeb914f" containerName="otc-container" Oct 30 04:04:51.669159 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.669029 5882 cpu_manager.go:414] "containerMap: removing container" podUID="21def72f-e0cb-4438-8a67-eb38e5341d36" containerName="ta-container" Oct 30 04:04:51.669159 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.669036 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21def72f-e0cb-4438-8a67-eb38e5341d36" containerName="ta-container" Oct 30 04:04:51.669159 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.669118 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="555eacfa-6f9e-4efd-8fec-8b38c8c50283" containerName="perses" Oct 30 04:04:51.669159 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.669131 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="05dd6cb8-8908-4609-937d-80bb026a7937" containerName="config-reloader" Oct 30 04:04:51.669159 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.669142 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4dc26eb-1fe7-451e-aa4b-5380b08875ee" containerName="thanos-querier" Oct 30 04:04:51.669159 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.669150 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="04423bfc-5f36-44c9-89e8-0f0b1aeb914f" containerName="otc-container" Oct 30 04:04:51.669159 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.669159 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="05dd6cb8-8908-4609-937d-80bb026a7937" containerName="alertmanager" Oct 30 04:04:51.669580 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.669168 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="21def72f-e0cb-4438-8a67-eb38e5341d36" containerName="ta-container" Oct 30 04:04:51.669580 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.669179 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="631dffab-7a9c-403e-8d9c-c741e1af0055" containerName="ta-container" Oct 30 04:04:51.691760 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.691728 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk"] Oct 30 04:04:51.691946 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.691859 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk" Oct 30 04:04:51.709631 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.709600 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 04:04:51.711994 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.711974 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 04:04:51.723865 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.723839 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 04:04:51.761925 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.761893 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldr2m\" (UniqueName: \"kubernetes.io/projected/77273182-1230-48d9-b029-5c6ee5af135f-kube-api-access-ldr2m\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk\" (UID: \"77273182-1230-48d9-b029-5c6ee5af135f\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk" Oct 30 04:04:51.862387 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.862348 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldr2m\" (UniqueName: \"kubernetes.io/projected/77273182-1230-48d9-b029-5c6ee5af135f-kube-api-access-ldr2m\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk\" (UID: \"77273182-1230-48d9-b029-5c6ee5af135f\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk" Oct 30 04:04:51.888336 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:51.888307 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldr2m\" (UniqueName: \"kubernetes.io/projected/77273182-1230-48d9-b029-5c6ee5af135f-kube-api-access-ldr2m\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk\" (UID: \"77273182-1230-48d9-b029-5c6ee5af135f\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk" Oct 30 04:04:52.015775 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.015740 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk" Oct 30 04:04:52.306130 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.306098 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk"] Oct 30 04:04:52.306538 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:04:52.306511 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77273182_1230_48d9_b029_5c6ee5af135f.slice/crio-9444b90f96eda6fec40a3c1cb5983ff5610617e324f73480f2bd3f4dfea85bb1 WatchSource:0}: Error finding container 9444b90f96eda6fec40a3c1cb5983ff5610617e324f73480f2bd3f4dfea85bb1: Status 404 returned error can't find the container with id 9444b90f96eda6fec40a3c1cb5983ff5610617e324f73480f2bd3f4dfea85bb1 Oct 30 04:04:52.668671 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.668568 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk" event={"ID":"77273182-1230-48d9-b029-5c6ee5af135f","Type":"ContainerStarted","Data":"d3fb2e12d877cb860f918d33048328718052070b18f71f348d669616d5ff376c"} Oct 30 04:04:52.668671 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.668628 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk" event={"ID":"77273182-1230-48d9-b029-5c6ee5af135f","Type":"ContainerStarted","Data":"9444b90f96eda6fec40a3c1cb5983ff5610617e324f73480f2bd3f4dfea85bb1"} Oct 30 04:04:52.711460 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.711408 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk" podStartSLOduration=1.711393961 podStartE2EDuration="1.711393961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:52.711297816 +0000 UTC m=+5004.825612796" watchObservedRunningTime="2026-10-30 04:04:52.711393961 +0000 UTC m=+5004.825708927" Oct 30 04:04:52.763500 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.763460 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:52.766883 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.766859 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:52.781996 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.781966 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:04:52.785260 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.785237 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:52.788000 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.787976 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:52.797187 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.797162 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:04:52.804860 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.804827 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:04:52.856702 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.856670 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 30 04:04:52.856948 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.856926 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:04:52.857045 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.856971 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-js864\"" Oct 30 04:04:52.857045 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.857001 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:04:52.857045 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.857033 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:04:52.866783 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.866755 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5"] Oct 30 04:04:52.870167 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.870149 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" Oct 30 04:04:52.891550 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.891520 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5"] Oct 30 04:04:52.956850 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.956765 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 30 04:04:52.957011 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.956866 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 30 04:04:52.957011 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.956901 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-tk52k\"" Oct 30 04:04:52.972588 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.972557 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:52.972588 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.972590 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/86713a18-dde5-45e0-9c9a-0dfabea25f6d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:52.972783 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.972637 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:52.972783 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.972756 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/572e1807-f954-49fd-bb72-d1d14e2a7dd9-plugins\") pod \"data-science-perses-0\" (UID: \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:52.972855 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.972782 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:52.972855 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.972806 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94t6t\" (UniqueName: \"kubernetes.io/projected/572e1807-f954-49fd-bb72-d1d14e2a7dd9-kube-api-access-94t6t\") pod \"data-science-perses-0\" (UID: \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:52.972855 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.972844 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/572e1807-f954-49fd-bb72-d1d14e2a7dd9-config\") pod \"data-science-perses-0\" (UID: \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:52.973006 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:52.972870 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlm9t\" (UniqueName: \"kubernetes.io/projected/86713a18-dde5-45e0-9c9a-0dfabea25f6d-kube-api-access-dlm9t\") pod \"data-science-collector-collector-1\" (UID: \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:53.074208 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.074167 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:53.074208 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.074209 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/86713a18-dde5-45e0-9c9a-0dfabea25f6d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:53.074444 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.074242 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:53.074444 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.074268 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kztv\" (UniqueName: \"kubernetes.io/projected/dfa469c4-de6a-4cef-98e5-83837cc5acbb-kube-api-access-6kztv\") pod \"data-science-collector-targetallocator-7f589477ff-4tdk5\" (UID: \"dfa469c4-de6a-4cef-98e5-83837cc5acbb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" Oct 30 04:04:53.074444 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.074316 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/dfa469c4-de6a-4cef-98e5-83837cc5acbb-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7f589477ff-4tdk5\" (UID: \"dfa469c4-de6a-4cef-98e5-83837cc5acbb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" Oct 30 04:04:53.074543 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.074466 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/dfa469c4-de6a-4cef-98e5-83837cc5acbb-ta-internal\") pod \"data-science-collector-targetallocator-7f589477ff-4tdk5\" (UID: \"dfa469c4-de6a-4cef-98e5-83837cc5acbb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" Oct 30 04:04:53.074543 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.074505 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/572e1807-f954-49fd-bb72-d1d14e2a7dd9-plugins\") pod \"data-science-perses-0\" (UID: \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:53.074543 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.074523 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:53.074812 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.074551 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94t6t\" (UniqueName: \"kubernetes.io/projected/572e1807-f954-49fd-bb72-d1d14e2a7dd9-kube-api-access-94t6t\") pod \"data-science-perses-0\" (UID: \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:53.074812 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.074596 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/572e1807-f954-49fd-bb72-d1d14e2a7dd9-config\") pod \"data-science-perses-0\" (UID: \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:53.074812 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.074659 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlm9t\" (UniqueName: \"kubernetes.io/projected/86713a18-dde5-45e0-9c9a-0dfabea25f6d-kube-api-access-dlm9t\") pod \"data-science-collector-collector-1\" (UID: \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:53.074984 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.074962 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/572e1807-f954-49fd-bb72-d1d14e2a7dd9-plugins\") pod \"data-science-perses-0\" (UID: \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:53.075394 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.075370 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/572e1807-f954-49fd-bb72-d1d14e2a7dd9-config\") pod \"data-science-perses-0\" (UID: \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:53.075478 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.075438 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/86713a18-dde5-45e0-9c9a-0dfabea25f6d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:53.076912 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.076889 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:53.076912 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.076897 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:53.077052 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.077029 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:53.100327 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.100300 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlm9t\" (UniqueName: \"kubernetes.io/projected/86713a18-dde5-45e0-9c9a-0dfabea25f6d-kube-api-access-dlm9t\") pod \"data-science-collector-collector-1\" (UID: \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:53.100458 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.100429 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94t6t\" (UniqueName: \"kubernetes.io/projected/572e1807-f954-49fd-bb72-d1d14e2a7dd9-kube-api-access-94t6t\") pod \"data-science-perses-0\" (UID: \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:53.176221 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.176180 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kztv\" (UniqueName: \"kubernetes.io/projected/dfa469c4-de6a-4cef-98e5-83837cc5acbb-kube-api-access-6kztv\") pod \"data-science-collector-targetallocator-7f589477ff-4tdk5\" (UID: \"dfa469c4-de6a-4cef-98e5-83837cc5acbb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" Oct 30 04:04:53.176418 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.176348 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/dfa469c4-de6a-4cef-98e5-83837cc5acbb-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7f589477ff-4tdk5\" (UID: \"dfa469c4-de6a-4cef-98e5-83837cc5acbb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" Oct 30 04:04:53.176532 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.176503 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/dfa469c4-de6a-4cef-98e5-83837cc5acbb-ta-internal\") pod \"data-science-collector-targetallocator-7f589477ff-4tdk5\" (UID: \"dfa469c4-de6a-4cef-98e5-83837cc5acbb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" Oct 30 04:04:53.177166 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.177139 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/dfa469c4-de6a-4cef-98e5-83837cc5acbb-ta-internal\") pod \"data-science-collector-targetallocator-7f589477ff-4tdk5\" (UID: \"dfa469c4-de6a-4cef-98e5-83837cc5acbb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" Oct 30 04:04:53.178884 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.178860 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/dfa469c4-de6a-4cef-98e5-83837cc5acbb-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7f589477ff-4tdk5\" (UID: \"dfa469c4-de6a-4cef-98e5-83837cc5acbb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" Oct 30 04:04:53.201321 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.201286 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kztv\" (UniqueName: \"kubernetes.io/projected/dfa469c4-de6a-4cef-98e5-83837cc5acbb-kube-api-access-6kztv\") pod \"data-science-collector-targetallocator-7f589477ff-4tdk5\" (UID: \"dfa469c4-de6a-4cef-98e5-83837cc5acbb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" Oct 30 04:04:53.379893 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.379856 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:04:53.482261 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.482220 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" Oct 30 04:04:53.675882 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:04:53.675856 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86713a18_dde5_45e0_9c9a_0dfabea25f6d.slice/crio-d16c4070bb71143e8df9c5726ff09b738725d0be64d155c99db432ec5dcbff57 WatchSource:0}: Error finding container d16c4070bb71143e8df9c5726ff09b738725d0be64d155c99db432ec5dcbff57: Status 404 returned error can't find the container with id d16c4070bb71143e8df9c5726ff09b738725d0be64d155c99db432ec5dcbff57 Oct 30 04:04:53.676189 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.676057 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:04:53.677954 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.677924 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:53.678036 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.677980 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:53.776903 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.776862 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5"] Oct 30 04:04:53.777285 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:04:53.777255 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfa469c4_de6a_4cef_98e5_83837cc5acbb.slice/crio-f0d35e5616f12e2aae0a534c26749ce323a40825c865f133802b9e0a6e34d3c4 WatchSource:0}: Error finding container f0d35e5616f12e2aae0a534c26749ce323a40825c865f133802b9e0a6e34d3c4: Status 404 returned error can't find the container with id f0d35e5616f12e2aae0a534c26749ce323a40825c865f133802b9e0a6e34d3c4 Oct 30 04:04:53.779489 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.779461 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:53.779562 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:53.779516 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:53.943238 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:04:53.943163 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:04:54.680372 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:54.680272 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"86713a18-dde5-45e0-9c9a-0dfabea25f6d","Type":"ContainerStarted","Data":"5ddcfd28eaf8974ecbbfbb2023244a1a8083d47abfa14421f916c294490e286f"} Oct 30 04:04:54.680372 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:54.680344 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"86713a18-dde5-45e0-9c9a-0dfabea25f6d","Type":"ContainerStarted","Data":"d16c4070bb71143e8df9c5726ff09b738725d0be64d155c99db432ec5dcbff57"} Oct 30 04:04:54.681831 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:54.681802 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" event={"ID":"dfa469c4-de6a-4cef-98e5-83837cc5acbb","Type":"ContainerStarted","Data":"aa7792e7b227468e1e10a81f04192448552a5807526fdf2281f2f609e14ce3ed"} Oct 30 04:04:54.681937 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:54.681843 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" event={"ID":"dfa469c4-de6a-4cef-98e5-83837cc5acbb","Type":"ContainerStarted","Data":"f0d35e5616f12e2aae0a534c26749ce323a40825c865f133802b9e0a6e34d3c4"} Oct 30 04:04:54.681994 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:54.681967 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" Oct 30 04:04:54.683694 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:54.683672 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" Oct 30 04:04:54.726800 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:54.726754 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.726743016 podStartE2EDuration="2.726743016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:54.726290572 +0000 UTC m=+5006.840605554" watchObservedRunningTime="2026-10-30 04:04:54.726743016 +0000 UTC m=+5006.841057988" Oct 30 04:04:54.771531 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:54.771480 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" podStartSLOduration=2.77146485 podStartE2EDuration="2.77146485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:54.770241373 +0000 UTC m=+5006.884556350" watchObservedRunningTime="2026-10-30 04:04:54.77146485 +0000 UTC m=+5006.885779816" Oct 30 04:04:55.196109 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:55.196072 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:55.228933 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:55.228901 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:55.297281 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:55.297243 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:55.297447 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:55.297378 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:55.313844 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:55.313819 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\") DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:55.417443 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:55.417413 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2c422b71c1e0b8fdf34659f2cef7846e06b4353ed88027dc99ade5d7fc29e9d6/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:55.448692 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:55.448608 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:55.498880 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:55.498843 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:55.794817 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:55.793658 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:04:55.795527 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:04:55.795496 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod572e1807_f954_49fd_bb72_d1d14e2a7dd9.slice/crio-ea50a9215c9d61294dc788ea12f5ba3e5e1f4eef861f4a4354c7b5270f72eedf WatchSource:0}: Error finding container ea50a9215c9d61294dc788ea12f5ba3e5e1f4eef861f4a4354c7b5270f72eedf: Status 404 returned error can't find the container with id ea50a9215c9d61294dc788ea12f5ba3e5e1f4eef861f4a4354c7b5270f72eedf Oct 30 04:04:56.292764 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.292726 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk"] Oct 30 04:04:56.292969 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.292885 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk" podUID="77273182-1230-48d9-b029-5c6ee5af135f" containerName="thanos-querier" containerID="cri-o://d3fb2e12d877cb860f918d33048328718052070b18f71f348d669616d5ff376c" gracePeriod=30 Oct 30 04:04:56.516681 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.516640 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:04:56.524889 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.524856 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.544605 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.544517 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:04:56.574551 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.574525 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk" Oct 30 04:04:56.596835 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.596806 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 04:04:56.596835 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.596824 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:04:56.597107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.597082 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 04:04:56.597152 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.597114 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 04:04:56.600674 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.600653 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-8ntbt\"" Oct 30 04:04:56.693640 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.693596 5882 generic.go:356] "Generic (PLEG): container finished" podID="77273182-1230-48d9-b029-5c6ee5af135f" containerID="d3fb2e12d877cb860f918d33048328718052070b18f71f348d669616d5ff376c" exitCode=0 Oct 30 04:04:56.693640 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.693637 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk" Oct 30 04:04:56.693923 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.693652 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk" event={"ID":"77273182-1230-48d9-b029-5c6ee5af135f","Type":"ContainerDied","Data":"d3fb2e12d877cb860f918d33048328718052070b18f71f348d669616d5ff376c"} Oct 30 04:04:56.693923 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.693707 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk" event={"ID":"77273182-1230-48d9-b029-5c6ee5af135f","Type":"ContainerDied","Data":"9444b90f96eda6fec40a3c1cb5983ff5610617e324f73480f2bd3f4dfea85bb1"} Oct 30 04:04:56.693923 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.693732 5882 scope.go:122] "RemoveContainer" containerID="d3fb2e12d877cb860f918d33048328718052070b18f71f348d669616d5ff376c" Oct 30 04:04:56.695719 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.695694 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"572e1807-f954-49fd-bb72-d1d14e2a7dd9","Type":"ContainerStarted","Data":"019f55cd9bf488930db48dfa0b30520b1e36251d2ddea470fadbf52251951aa8"} Oct 30 04:04:56.695837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.695731 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"572e1807-f954-49fd-bb72-d1d14e2a7dd9","Type":"ContainerStarted","Data":"ea50a9215c9d61294dc788ea12f5ba3e5e1f4eef861f4a4354c7b5270f72eedf"} Oct 30 04:04:56.704012 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.703990 5882 scope.go:122] "RemoveContainer" containerID="d3fb2e12d877cb860f918d33048328718052070b18f71f348d669616d5ff376c" Oct 30 04:04:56.704285 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:04:56.704268 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3fb2e12d877cb860f918d33048328718052070b18f71f348d669616d5ff376c\": container with ID starting with d3fb2e12d877cb860f918d33048328718052070b18f71f348d669616d5ff376c not found: ID does not exist" containerID="d3fb2e12d877cb860f918d33048328718052070b18f71f348d669616d5ff376c" Oct 30 04:04:56.704358 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.704291 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3fb2e12d877cb860f918d33048328718052070b18f71f348d669616d5ff376c"} err="failed to get container status \"d3fb2e12d877cb860f918d33048328718052070b18f71f348d669616d5ff376c\": rpc error: code = NotFound desc = could not find container \"d3fb2e12d877cb860f918d33048328718052070b18f71f348d669616d5ff376c\": container with ID starting with d3fb2e12d877cb860f918d33048328718052070b18f71f348d669616d5ff376c not found: ID does not exist" Oct 30 04:04:56.710560 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.710536 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77273182-1230-48d9-b029-5c6ee5af135f-kube-api-access-ldr2m\" (UniqueName: \"kubernetes.io/projected/77273182-1230-48d9-b029-5c6ee5af135f-kube-api-access-ldr2m\") pod \"77273182-1230-48d9-b029-5c6ee5af135f\" (UID: \"77273182-1230-48d9-b029-5c6ee5af135f\") " Oct 30 04:04:56.710744 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.710726 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.710794 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.710782 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.710921 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.710892 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.710981 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.710936 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.711034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.711004 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.711070 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.711044 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz5td\" (UniqueName: \"kubernetes.io/projected/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-kube-api-access-fz5td\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.711117 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.711078 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.712594 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.712571 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77273182-1230-48d9-b029-5c6ee5af135f-kube-api-access-ldr2m" pod "77273182-1230-48d9-b029-5c6ee5af135f" (UID: "77273182-1230-48d9-b029-5c6ee5af135f"). InnerVolumeSpecName "kube-api-access-ldr2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:56.756884 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.756825 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=4.756810113 podStartE2EDuration="4.756810113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:56.754408669 +0000 UTC m=+5008.868723645" watchObservedRunningTime="2026-10-30 04:04:56.756810113 +0000 UTC m=+5008.871125089" Oct 30 04:04:56.812680 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.812569 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.813175 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.812811 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz5td\" (UniqueName: \"kubernetes.io/projected/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-kube-api-access-fz5td\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.813175 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.812870 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.813175 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.812947 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.813175 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.813015 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.813175 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.813084 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.813175 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.813119 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.813175 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.813169 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldr2m\" (UniqueName: \"kubernetes.io/projected/77273182-1230-48d9-b029-5c6ee5af135f-kube-api-access-ldr2m\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:04:56.813584 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.813319 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.815252 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.815229 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.815512 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.815491 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.815567 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.815540 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.815929 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.815905 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.816033 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.816012 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.838041 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.838013 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz5td\" (UniqueName: \"kubernetes.io/projected/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-kube-api-access-fz5td\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:56.855499 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:56.855470 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:04:57.084059 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:57.083969 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk"] Oct 30 04:04:57.100732 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:57.100697 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-rvnqk"] Oct 30 04:04:57.175735 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:04:57.175702 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c665b2_5fe7_4d2c_b3f8_136d6929bc07.slice/crio-9ad6a4d64d798de007f3c38c1041bd1a9e2d625a3c01f080fe260093c971f6fc WatchSource:0}: Error finding container 9ad6a4d64d798de007f3c38c1041bd1a9e2d625a3c01f080fe260093c971f6fc: Status 404 returned error can't find the container with id 9ad6a4d64d798de007f3c38c1041bd1a9e2d625a3c01f080fe260093c971f6fc Oct 30 04:04:57.175985 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:57.175961 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:04:57.703947 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:57.703902 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07","Type":"ContainerStarted","Data":"47e9b3e35d81cf832e7bde4d962a3d0934b762883c65bdd8f714c8bee74a53e2"} Oct 30 04:04:57.704122 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:57.703977 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07","Type":"ContainerStarted","Data":"9ad6a4d64d798de007f3c38c1041bd1a9e2d625a3c01f080fe260093c971f6fc"} Oct 30 04:04:57.704197 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:57.704155 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:04:58.009123 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:04:58.009087 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77273182-1230-48d9-b029-5c6ee5af135f" path="/var/lib/kubelet/pods/77273182-1230-48d9-b029-5c6ee5af135f/volumes" Oct 30 04:05:01.731830 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:05:01.731797 5882 generic.go:356] "Generic (PLEG): container finished" podID="a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" containerID="47e9b3e35d81cf832e7bde4d962a3d0934b762883c65bdd8f714c8bee74a53e2" exitCode=0 Oct 30 04:05:01.732219 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:05:01.731851 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07","Type":"ContainerDied","Data":"47e9b3e35d81cf832e7bde4d962a3d0934b762883c65bdd8f714c8bee74a53e2"} Oct 30 04:05:02.738826 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:05:02.738734 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07","Type":"ContainerStarted","Data":"f0e02235537a98961dc90ce8b472097e9d95581974df235e8e93cf6b94142998"} Oct 30 04:05:02.738826 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:05:02.738810 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07","Type":"ContainerStarted","Data":"a725c6d735b2f41a05a7935ad2dfe9e7b0df5b4cd98fc7d669f777a7403aa89c"} Oct 30 04:05:02.739363 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:05:02.739020 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:05:02.819533 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:05:02.819476 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.334935513 podStartE2EDuration="6.819460031s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.484524518s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:05:02.804983057 +0000 UTC m=+5014.919298033" watchObservedRunningTime="2026-10-30 04:05:02.819460031 +0000 UTC m=+5014.933775023" Oct 30 04:05:08.719212 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:05:08.719178 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:05:08.744889 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:05:08.744864 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:05:23.941931 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:05:23.941896 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:05:53.942065 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:05:53.942027 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:06:01.406345 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:01.406311 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:06:01.406925 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:01.406555 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" containerName="alertmanager" containerID="cri-o://a725c6d735b2f41a05a7935ad2dfe9e7b0df5b4cd98fc7d669f777a7403aa89c" gracePeriod=120 Oct 30 04:06:01.406925 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:01.406600 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" containerName="config-reloader" containerID="cri-o://f0e02235537a98961dc90ce8b472097e9d95581974df235e8e93cf6b94142998" gracePeriod=120 Oct 30 04:06:02.012823 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.012793 5882 generic.go:356] "Generic (PLEG): container finished" podID="a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" containerID="f0e02235537a98961dc90ce8b472097e9d95581974df235e8e93cf6b94142998" exitCode=0 Oct 30 04:06:02.012990 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.012844 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07","Type":"ContainerDied","Data":"f0e02235537a98961dc90ce8b472097e9d95581974df235e8e93cf6b94142998"} Oct 30 04:06:02.068947 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.068901 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:06:02.069386 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.069227 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="86713a18-dde5-45e0-9c9a-0dfabea25f6d" containerName="otc-container" containerID="cri-o://5ddcfd28eaf8974ecbbfbb2023244a1a8083d47abfa14421f916c294490e286f" gracePeriod=30 Oct 30 04:06:02.112093 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.112063 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5"] Oct 30 04:06:02.112287 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.112265 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" podUID="dfa469c4-de6a-4cef-98e5-83837cc5acbb" containerName="ta-container" containerID="cri-o://aa7792e7b227468e1e10a81f04192448552a5807526fdf2281f2f609e14ce3ed" gracePeriod=30 Oct 30 04:06:02.244978 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.244943 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:06:02.245157 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.245135 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="572e1807-f954-49fd-bb72-d1d14e2a7dd9" containerName="perses" containerID="cri-o://019f55cd9bf488930db48dfa0b30520b1e36251d2ddea470fadbf52251951aa8" gracePeriod=30 Oct 30 04:06:02.394521 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.394010 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:06:02.398294 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.398271 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" Oct 30 04:06:02.494362 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.494335 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86713a18-dde5-45e0-9c9a-0dfabea25f6d-kube-api-access-dlm9t\" (UniqueName: \"kubernetes.io/projected/86713a18-dde5-45e0-9c9a-0dfabea25f6d-kube-api-access-dlm9t\") pod \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\" (UID: \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\") " Oct 30 04:06:02.494793 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.494502 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-tls-certs\" (UniqueName: \"kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-tls-certs\") pod \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\" (UID: \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\") " Oct 30 04:06:02.494793 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.494562 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-monitoring-tls-certs\") pod \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\" (UID: \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\") " Oct 30 04:06:02.494793 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.494595 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfa469c4-de6a-4cef-98e5-83837cc5acbb-ta-internal\" (UniqueName: \"kubernetes.io/configmap/dfa469c4-de6a-4cef-98e5-83837cc5acbb-ta-internal\") pod \"dfa469c4-de6a-4cef-98e5-83837cc5acbb\" (UID: \"dfa469c4-de6a-4cef-98e5-83837cc5acbb\") " Oct 30 04:06:02.494793 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.494669 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86713a18-dde5-45e0-9c9a-0dfabea25f6d-otc-internal\" (UniqueName: \"kubernetes.io/configmap/86713a18-dde5-45e0-9c9a-0dfabea25f6d-otc-internal\") pod \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\" (UID: \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\") " Oct 30 04:06:02.494793 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.494697 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfa469c4-de6a-4cef-98e5-83837cc5acbb-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/dfa469c4-de6a-4cef-98e5-83837cc5acbb-data-science-collector-ta-server-cert\") pod \"dfa469c4-de6a-4cef-98e5-83837cc5acbb\" (UID: \"dfa469c4-de6a-4cef-98e5-83837cc5acbb\") " Oct 30 04:06:02.494793 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.494751 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-data-science-collector-ta-client-cert\") pod \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\" (UID: \"86713a18-dde5-45e0-9c9a-0dfabea25f6d\") " Oct 30 04:06:02.495671 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.495450 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86713a18-dde5-45e0-9c9a-0dfabea25f6d-otc-internal" pod "86713a18-dde5-45e0-9c9a-0dfabea25f6d" (UID: "86713a18-dde5-45e0-9c9a-0dfabea25f6d"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:02.495671 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.495454 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa469c4-de6a-4cef-98e5-83837cc5acbb-ta-internal" pod "dfa469c4-de6a-4cef-98e5-83837cc5acbb" (UID: "dfa469c4-de6a-4cef-98e5-83837cc5acbb"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:02.495671 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.495492 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:06:02.496726 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.496702 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86713a18-dde5-45e0-9c9a-0dfabea25f6d-kube-api-access-dlm9t" pod "86713a18-dde5-45e0-9c9a-0dfabea25f6d" (UID: "86713a18-dde5-45e0-9c9a-0dfabea25f6d"). InnerVolumeSpecName "kube-api-access-dlm9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:02.496886 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.496811 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-tls-certs" pod "86713a18-dde5-45e0-9c9a-0dfabea25f6d" (UID: "86713a18-dde5-45e0-9c9a-0dfabea25f6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:02.497038 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.497014 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-monitoring-tls-certs" pod "86713a18-dde5-45e0-9c9a-0dfabea25f6d" (UID: "86713a18-dde5-45e0-9c9a-0dfabea25f6d"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:02.497643 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.497083 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-data-science-collector-ta-client-cert" pod "86713a18-dde5-45e0-9c9a-0dfabea25f6d" (UID: "86713a18-dde5-45e0-9c9a-0dfabea25f6d"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:02.497643 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.497554 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfa469c4-de6a-4cef-98e5-83837cc5acbb-data-science-collector-ta-server-cert" pod "dfa469c4-de6a-4cef-98e5-83837cc5acbb" (UID: "dfa469c4-de6a-4cef-98e5-83837cc5acbb"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:02.596180 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.596150 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\" (UID: \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\") " Oct 30 04:06:02.596329 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.596209 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfa469c4-de6a-4cef-98e5-83837cc5acbb-kube-api-access-6kztv\" (UniqueName: \"kubernetes.io/projected/dfa469c4-de6a-4cef-98e5-83837cc5acbb-kube-api-access-6kztv\") pod \"dfa469c4-de6a-4cef-98e5-83837cc5acbb\" (UID: \"dfa469c4-de6a-4cef-98e5-83837cc5acbb\") " Oct 30 04:06:02.596329 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.596249 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/572e1807-f954-49fd-bb72-d1d14e2a7dd9-kube-api-access-94t6t\" (UniqueName: \"kubernetes.io/projected/572e1807-f954-49fd-bb72-d1d14e2a7dd9-kube-api-access-94t6t\") pod \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\" (UID: \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\") " Oct 30 04:06:02.596329 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.596318 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572e1807-f954-49fd-bb72-d1d14e2a7dd9-plugins\" (UniqueName: \"kubernetes.io/empty-dir/572e1807-f954-49fd-bb72-d1d14e2a7dd9-plugins\") pod \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\" (UID: \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\") " Oct 30 04:06:02.596473 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.596336 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/572e1807-f954-49fd-bb72-d1d14e2a7dd9-config\" (UniqueName: \"kubernetes.io/configmap/572e1807-f954-49fd-bb72-d1d14e2a7dd9-config\") pod \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\" (UID: \"572e1807-f954-49fd-bb72-d1d14e2a7dd9\") " Oct 30 04:06:02.596832 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.596800 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.596832 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.596829 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlm9t\" (UniqueName: \"kubernetes.io/projected/86713a18-dde5-45e0-9c9a-0dfabea25f6d-kube-api-access-dlm9t\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.597023 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.596844 5882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.597023 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.596857 5882 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/86713a18-dde5-45e0-9c9a-0dfabea25f6d-monitoring-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.597023 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.596866 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/dfa469c4-de6a-4cef-98e5-83837cc5acbb-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.597023 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.596874 5882 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/86713a18-dde5-45e0-9c9a-0dfabea25f6d-otc-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.597023 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.596882 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/dfa469c4-de6a-4cef-98e5-83837cc5acbb-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.597452 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.597422 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/572e1807-f954-49fd-bb72-d1d14e2a7dd9-config" pod "572e1807-f954-49fd-bb72-d1d14e2a7dd9" (UID: "572e1807-f954-49fd-bb72-d1d14e2a7dd9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:02.598608 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.598579 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa469c4-de6a-4cef-98e5-83837cc5acbb-kube-api-access-6kztv" pod "dfa469c4-de6a-4cef-98e5-83837cc5acbb" (UID: "dfa469c4-de6a-4cef-98e5-83837cc5acbb"). InnerVolumeSpecName "kube-api-access-6kztv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:02.598884 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.598858 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572e1807-f954-49fd-bb72-d1d14e2a7dd9-kube-api-access-94t6t" pod "572e1807-f954-49fd-bb72-d1d14e2a7dd9" (UID: "572e1807-f954-49fd-bb72-d1d14e2a7dd9"). InnerVolumeSpecName "kube-api-access-94t6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:02.599325 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.599308 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e" pod "572e1807-f954-49fd-bb72-d1d14e2a7dd9" (UID: "572e1807-f954-49fd-bb72-d1d14e2a7dd9"). InnerVolumeSpecName "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:06:02.697597 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.697556 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/572e1807-f954-49fd-bb72-d1d14e2a7dd9-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.697767 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.697637 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:06:02.697767 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.697657 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kztv\" (UniqueName: \"kubernetes.io/projected/dfa469c4-de6a-4cef-98e5-83837cc5acbb-kube-api-access-6kztv\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.697767 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.697673 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94t6t\" (UniqueName: \"kubernetes.io/projected/572e1807-f954-49fd-bb72-d1d14e2a7dd9-kube-api-access-94t6t\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.711304 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.711284 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:06:02.744595 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.744557 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572e1807-f954-49fd-bb72-d1d14e2a7dd9-plugins" pod "572e1807-f954-49fd-bb72-d1d14e2a7dd9" (UID: "572e1807-f954-49fd-bb72-d1d14e2a7dd9"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:02.798710 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.798679 5882 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/572e1807-f954-49fd-bb72-d1d14e2a7dd9-plugins\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.798710 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:02.798709 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" Oct 30 04:06:03.018892 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.018862 5882 generic.go:356] "Generic (PLEG): container finished" podID="572e1807-f954-49fd-bb72-d1d14e2a7dd9" containerID="019f55cd9bf488930db48dfa0b30520b1e36251d2ddea470fadbf52251951aa8" exitCode=0 Oct 30 04:06:03.019086 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.018915 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:06:03.019086 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.018954 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"572e1807-f954-49fd-bb72-d1d14e2a7dd9","Type":"ContainerDied","Data":"019f55cd9bf488930db48dfa0b30520b1e36251d2ddea470fadbf52251951aa8"} Oct 30 04:06:03.019086 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.018991 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"572e1807-f954-49fd-bb72-d1d14e2a7dd9","Type":"ContainerDied","Data":"ea50a9215c9d61294dc788ea12f5ba3e5e1f4eef861f4a4354c7b5270f72eedf"} Oct 30 04:06:03.019086 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.019015 5882 scope.go:122] "RemoveContainer" containerID="019f55cd9bf488930db48dfa0b30520b1e36251d2ddea470fadbf52251951aa8" Oct 30 04:06:03.020476 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.020449 5882 generic.go:356] "Generic (PLEG): container finished" podID="86713a18-dde5-45e0-9c9a-0dfabea25f6d" containerID="5ddcfd28eaf8974ecbbfbb2023244a1a8083d47abfa14421f916c294490e286f" exitCode=0 Oct 30 04:06:03.020606 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.020507 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:06:03.020606 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.020544 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"86713a18-dde5-45e0-9c9a-0dfabea25f6d","Type":"ContainerDied","Data":"5ddcfd28eaf8974ecbbfbb2023244a1a8083d47abfa14421f916c294490e286f"} Oct 30 04:06:03.020606 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.020595 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"86713a18-dde5-45e0-9c9a-0dfabea25f6d","Type":"ContainerDied","Data":"d16c4070bb71143e8df9c5726ff09b738725d0be64d155c99db432ec5dcbff57"} Oct 30 04:06:03.022013 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.021987 5882 generic.go:356] "Generic (PLEG): container finished" podID="dfa469c4-de6a-4cef-98e5-83837cc5acbb" containerID="aa7792e7b227468e1e10a81f04192448552a5807526fdf2281f2f609e14ce3ed" exitCode=0 Oct 30 04:06:03.022123 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.022028 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" Oct 30 04:06:03.022123 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.022070 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" event={"ID":"dfa469c4-de6a-4cef-98e5-83837cc5acbb","Type":"ContainerDied","Data":"aa7792e7b227468e1e10a81f04192448552a5807526fdf2281f2f609e14ce3ed"} Oct 30 04:06:03.022123 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.022098 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5" event={"ID":"dfa469c4-de6a-4cef-98e5-83837cc5acbb","Type":"ContainerDied","Data":"f0d35e5616f12e2aae0a534c26749ce323a40825c865f133802b9e0a6e34d3c4"} Oct 30 04:06:03.030001 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.029797 5882 scope.go:122] "RemoveContainer" containerID="019f55cd9bf488930db48dfa0b30520b1e36251d2ddea470fadbf52251951aa8" Oct 30 04:06:03.030451 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:06:03.030418 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"019f55cd9bf488930db48dfa0b30520b1e36251d2ddea470fadbf52251951aa8\": container with ID starting with 019f55cd9bf488930db48dfa0b30520b1e36251d2ddea470fadbf52251951aa8 not found: ID does not exist" containerID="019f55cd9bf488930db48dfa0b30520b1e36251d2ddea470fadbf52251951aa8" Oct 30 04:06:03.030551 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.030462 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"019f55cd9bf488930db48dfa0b30520b1e36251d2ddea470fadbf52251951aa8"} err="failed to get container status \"019f55cd9bf488930db48dfa0b30520b1e36251d2ddea470fadbf52251951aa8\": rpc error: code = NotFound desc = could not find container \"019f55cd9bf488930db48dfa0b30520b1e36251d2ddea470fadbf52251951aa8\": container with ID starting with 019f55cd9bf488930db48dfa0b30520b1e36251d2ddea470fadbf52251951aa8 not found: ID does not exist" Oct 30 04:06:03.030551 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.030483 5882 scope.go:122] "RemoveContainer" containerID="5ddcfd28eaf8974ecbbfbb2023244a1a8083d47abfa14421f916c294490e286f" Oct 30 04:06:03.038474 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.038458 5882 scope.go:122] "RemoveContainer" containerID="5ddcfd28eaf8974ecbbfbb2023244a1a8083d47abfa14421f916c294490e286f" Oct 30 04:06:03.038738 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:06:03.038722 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ddcfd28eaf8974ecbbfbb2023244a1a8083d47abfa14421f916c294490e286f\": container with ID starting with 5ddcfd28eaf8974ecbbfbb2023244a1a8083d47abfa14421f916c294490e286f not found: ID does not exist" containerID="5ddcfd28eaf8974ecbbfbb2023244a1a8083d47abfa14421f916c294490e286f" Oct 30 04:06:03.038832 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.038745 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ddcfd28eaf8974ecbbfbb2023244a1a8083d47abfa14421f916c294490e286f"} err="failed to get container status \"5ddcfd28eaf8974ecbbfbb2023244a1a8083d47abfa14421f916c294490e286f\": rpc error: code = NotFound desc = could not find container \"5ddcfd28eaf8974ecbbfbb2023244a1a8083d47abfa14421f916c294490e286f\": container with ID starting with 5ddcfd28eaf8974ecbbfbb2023244a1a8083d47abfa14421f916c294490e286f not found: ID does not exist" Oct 30 04:06:03.038832 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.038764 5882 scope.go:122] "RemoveContainer" containerID="aa7792e7b227468e1e10a81f04192448552a5807526fdf2281f2f609e14ce3ed" Oct 30 04:06:03.048659 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.048636 5882 scope.go:122] "RemoveContainer" containerID="aa7792e7b227468e1e10a81f04192448552a5807526fdf2281f2f609e14ce3ed" Oct 30 04:06:03.048991 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:06:03.048972 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7792e7b227468e1e10a81f04192448552a5807526fdf2281f2f609e14ce3ed\": container with ID starting with aa7792e7b227468e1e10a81f04192448552a5807526fdf2281f2f609e14ce3ed not found: ID does not exist" containerID="aa7792e7b227468e1e10a81f04192448552a5807526fdf2281f2f609e14ce3ed" Oct 30 04:06:03.049066 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.049000 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7792e7b227468e1e10a81f04192448552a5807526fdf2281f2f609e14ce3ed"} err="failed to get container status \"aa7792e7b227468e1e10a81f04192448552a5807526fdf2281f2f609e14ce3ed\": rpc error: code = NotFound desc = could not find container \"aa7792e7b227468e1e10a81f04192448552a5807526fdf2281f2f609e14ce3ed\": container with ID starting with aa7792e7b227468e1e10a81f04192448552a5807526fdf2281f2f609e14ce3ed not found: ID does not exist" Oct 30 04:06:03.074900 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.074872 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:06:03.083132 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.083105 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:06:03.120538 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.120463 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5"] Oct 30 04:06:03.125702 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.125676 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7f589477ff-4tdk5"] Oct 30 04:06:03.164570 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.164541 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:06:03.169120 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.169091 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:06:03.743990 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:03.743954 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.134:9093/-/ready\": dial tcp 10.134.0.134:9093: connect: connection refused" Oct 30 04:06:04.012746 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:04.012710 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="572e1807-f954-49fd-bb72-d1d14e2a7dd9" path="/var/lib/kubelet/pods/572e1807-f954-49fd-bb72-d1d14e2a7dd9/volumes" Oct 30 04:06:04.013098 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:04.013085 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86713a18-dde5-45e0-9c9a-0dfabea25f6d" path="/var/lib/kubelet/pods/86713a18-dde5-45e0-9c9a-0dfabea25f6d/volumes" Oct 30 04:06:04.013419 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:04.013408 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfa469c4-de6a-4cef-98e5-83837cc5acbb" path="/var/lib/kubelet/pods/dfa469c4-de6a-4cef-98e5-83837cc5acbb/volumes" Oct 30 04:06:05.326392 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.326359 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 30 04:06:05.326800 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.326776 5882 cpu_manager.go:414] "containerMap: removing container" podUID="77273182-1230-48d9-b029-5c6ee5af135f" containerName="thanos-querier" Oct 30 04:06:05.326800 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.326790 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77273182-1230-48d9-b029-5c6ee5af135f" containerName="thanos-querier" Oct 30 04:06:05.326872 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.326803 5882 cpu_manager.go:414] "containerMap: removing container" podUID="86713a18-dde5-45e0-9c9a-0dfabea25f6d" containerName="otc-container" Oct 30 04:06:05.326872 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.326810 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86713a18-dde5-45e0-9c9a-0dfabea25f6d" containerName="otc-container" Oct 30 04:06:05.326872 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.326817 5882 cpu_manager.go:414] "containerMap: removing container" podUID="572e1807-f954-49fd-bb72-d1d14e2a7dd9" containerName="perses" Oct 30 04:06:05.326872 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.326822 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="572e1807-f954-49fd-bb72-d1d14e2a7dd9" containerName="perses" Oct 30 04:06:05.326872 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.326833 5882 cpu_manager.go:414] "containerMap: removing container" podUID="dfa469c4-de6a-4cef-98e5-83837cc5acbb" containerName="ta-container" Oct 30 04:06:05.326872 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.326838 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfa469c4-de6a-4cef-98e5-83837cc5acbb" containerName="ta-container" Oct 30 04:06:05.327067 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.326898 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="77273182-1230-48d9-b029-5c6ee5af135f" containerName="thanos-querier" Oct 30 04:06:05.327067 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.326914 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="86713a18-dde5-45e0-9c9a-0dfabea25f6d" containerName="otc-container" Oct 30 04:06:05.327067 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.326922 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="572e1807-f954-49fd-bb72-d1d14e2a7dd9" containerName="perses" Oct 30 04:06:05.327067 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.326929 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfa469c4-de6a-4cef-98e5-83837cc5acbb" containerName="ta-container" Oct 30 04:06:05.350234 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.350198 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 30 04:06:05.350396 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.350312 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs" Oct 30 04:06:05.367069 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.367045 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 04:06:05.423131 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.423093 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8bdebe0d-7c01-4821-826c-70541e8c32f1-data\") pod \"seaweedfs\" (UID: \"8bdebe0d-7c01-4821-826c-70541e8c32f1\") " pod="redhat-ods-monitoring/seaweedfs" Oct 30 04:06:05.423294 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.423220 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpzgt\" (UniqueName: \"kubernetes.io/projected/8bdebe0d-7c01-4821-826c-70541e8c32f1-kube-api-access-dpzgt\") pod \"seaweedfs\" (UID: \"8bdebe0d-7c01-4821-826c-70541e8c32f1\") " pod="redhat-ods-monitoring/seaweedfs" Oct 30 04:06:05.524427 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.524372 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8bdebe0d-7c01-4821-826c-70541e8c32f1-data\") pod \"seaweedfs\" (UID: \"8bdebe0d-7c01-4821-826c-70541e8c32f1\") " pod="redhat-ods-monitoring/seaweedfs" Oct 30 04:06:05.524598 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.524500 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpzgt\" (UniqueName: \"kubernetes.io/projected/8bdebe0d-7c01-4821-826c-70541e8c32f1-kube-api-access-dpzgt\") pod \"seaweedfs\" (UID: \"8bdebe0d-7c01-4821-826c-70541e8c32f1\") " pod="redhat-ods-monitoring/seaweedfs" Oct 30 04:06:05.524867 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.524842 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8bdebe0d-7c01-4821-826c-70541e8c32f1-data\") pod \"seaweedfs\" (UID: \"8bdebe0d-7c01-4821-826c-70541e8c32f1\") " pod="redhat-ods-monitoring/seaweedfs" Oct 30 04:06:05.550033 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.550002 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpzgt\" (UniqueName: \"kubernetes.io/projected/8bdebe0d-7c01-4821-826c-70541e8c32f1-kube-api-access-dpzgt\") pod \"seaweedfs\" (UID: \"8bdebe0d-7c01-4821-826c-70541e8c32f1\") " pod="redhat-ods-monitoring/seaweedfs" Oct 30 04:06:05.674884 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.674815 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs" Oct 30 04:06:05.969336 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:05.969301 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 30 04:06:05.970164 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:06:05.970130 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bdebe0d_7c01_4821_826c_70541e8c32f1.slice/crio-0e35bbd9b544dada6fbaa43010f9ae08b1f95cb44ea472bddb6c64fcfab91a15 WatchSource:0}: Error finding container 0e35bbd9b544dada6fbaa43010f9ae08b1f95cb44ea472bddb6c64fcfab91a15: Status 404 returned error can't find the container with id 0e35bbd9b544dada6fbaa43010f9ae08b1f95cb44ea472bddb6c64fcfab91a15 Oct 30 04:06:06.040368 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:06.040335 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs" event={"ID":"8bdebe0d-7c01-4821-826c-70541e8c32f1","Type":"ContainerStarted","Data":"429af317d05c4ec72bad8fab703cfd484a7b6df11139c1be561d65004df2413d"} Oct 30 04:06:06.040368 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:06.040374 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs" event={"ID":"8bdebe0d-7c01-4821-826c-70541e8c32f1","Type":"ContainerStarted","Data":"0e35bbd9b544dada6fbaa43010f9ae08b1f95cb44ea472bddb6c64fcfab91a15"} Oct 30 04:06:06.040601 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:06.040398 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/seaweedfs" Oct 30 04:06:06.083087 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:06.083034 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/seaweedfs" podStartSLOduration=1.083018138 podStartE2EDuration="1.083018138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:06:06.08272849 +0000 UTC m=+5078.197043466" watchObservedRunningTime="2026-10-30 04:06:06.083018138 +0000 UTC m=+5078.197333114" Oct 30 04:06:08.743813 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:08.743774 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.134:9093/-/ready\": dial tcp 10.134.0.134:9093: connect: connection refused" Oct 30 04:06:11.653009 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.652980 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:06:11.676013 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.675965 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-config-volume\" (UniqueName: \"kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-config-volume\") pod \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " Oct 30 04:06:11.676118 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.676016 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-tls-assets\" (UniqueName: \"kubernetes.io/projected/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-tls-assets\") pod \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " Oct 30 04:06:11.676118 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.676050 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-alertmanager-data-science-monitoringstack-db\") pod \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " Oct 30 04:06:11.676118 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.676086 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-kube-api-access-fz5td\" (UniqueName: \"kubernetes.io/projected/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-kube-api-access-fz5td\") pod \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " Oct 30 04:06:11.676281 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.676125 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-config-out\") pod \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " Oct 30 04:06:11.676281 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.676174 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-cluster-tls-config\") pod \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " Oct 30 04:06:11.676281 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.676202 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-web-config\" (UniqueName: \"kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-web-config\") pod \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\" (UID: \"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07\") " Oct 30 04:06:11.676469 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.676439 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-alertmanager-data-science-monitoringstack-db" pod "a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" (UID: "a0c665b2-5fe7-4d2c-b3f8-136d6929bc07"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:11.678278 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.678235 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-tls-assets" pod "a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" (UID: "a0c665b2-5fe7-4d2c-b3f8-136d6929bc07"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:11.678379 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.678301 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-config-out" pod "a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" (UID: "a0c665b2-5fe7-4d2c-b3f8-136d6929bc07"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:11.678525 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.678505 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-config-volume" pod "a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" (UID: "a0c665b2-5fe7-4d2c-b3f8-136d6929bc07"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:11.678681 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.678652 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-kube-api-access-fz5td" pod "a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" (UID: "a0c665b2-5fe7-4d2c-b3f8-136d6929bc07"). InnerVolumeSpecName "kube-api-access-fz5td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:11.682285 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.682259 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-cluster-tls-config" pod "a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" (UID: "a0c665b2-5fe7-4d2c-b3f8-136d6929bc07"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:11.689047 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.689023 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-web-config" pod "a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" (UID: "a0c665b2-5fe7-4d2c-b3f8-136d6929bc07"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:11.776832 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.776803 5882 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-cluster-tls-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:06:11.776832 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.776830 5882 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-web-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:06:11.776832 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.776839 5882 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-config-volume\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:06:11.777060 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.776847 5882 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-tls-assets\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:06:11.777060 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.776856 5882 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:06:11.777060 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.776865 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz5td\" (UniqueName: \"kubernetes.io/projected/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-kube-api-access-fz5td\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:06:11.777060 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:11.776874 5882 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07-config-out\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:06:12.069140 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:12.069106 5882 generic.go:356] "Generic (PLEG): container finished" podID="a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" containerID="a725c6d735b2f41a05a7935ad2dfe9e7b0df5b4cd98fc7d669f777a7403aa89c" exitCode=0 Oct 30 04:06:12.069295 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:12.069165 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:06:12.069295 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:12.069188 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07","Type":"ContainerDied","Data":"a725c6d735b2f41a05a7935ad2dfe9e7b0df5b4cd98fc7d669f777a7403aa89c"} Oct 30 04:06:12.069295 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:12.069244 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a0c665b2-5fe7-4d2c-b3f8-136d6929bc07","Type":"ContainerDied","Data":"9ad6a4d64d798de007f3c38c1041bd1a9e2d625a3c01f080fe260093c971f6fc"} Oct 30 04:06:12.069295 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:12.069271 5882 scope.go:122] "RemoveContainer" containerID="f0e02235537a98961dc90ce8b472097e9d95581974df235e8e93cf6b94142998" Oct 30 04:06:12.077674 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:12.077654 5882 scope.go:122] "RemoveContainer" containerID="a725c6d735b2f41a05a7935ad2dfe9e7b0df5b4cd98fc7d669f777a7403aa89c" Oct 30 04:06:12.085130 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:12.085110 5882 scope.go:122] "RemoveContainer" containerID="47e9b3e35d81cf832e7bde4d962a3d0934b762883c65bdd8f714c8bee74a53e2" Oct 30 04:06:12.092312 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:12.092292 5882 scope.go:122] "RemoveContainer" containerID="f0e02235537a98961dc90ce8b472097e9d95581974df235e8e93cf6b94142998" Oct 30 04:06:12.092571 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:06:12.092552 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e02235537a98961dc90ce8b472097e9d95581974df235e8e93cf6b94142998\": container with ID starting with f0e02235537a98961dc90ce8b472097e9d95581974df235e8e93cf6b94142998 not found: ID does not exist" containerID="f0e02235537a98961dc90ce8b472097e9d95581974df235e8e93cf6b94142998" Oct 30 04:06:12.092691 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:12.092579 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e02235537a98961dc90ce8b472097e9d95581974df235e8e93cf6b94142998"} err="failed to get container status \"f0e02235537a98961dc90ce8b472097e9d95581974df235e8e93cf6b94142998\": rpc error: code = NotFound desc = could not find container \"f0e02235537a98961dc90ce8b472097e9d95581974df235e8e93cf6b94142998\": container with ID starting with f0e02235537a98961dc90ce8b472097e9d95581974df235e8e93cf6b94142998 not found: ID does not exist" Oct 30 04:06:12.092691 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:12.092597 5882 scope.go:122] "RemoveContainer" containerID="a725c6d735b2f41a05a7935ad2dfe9e7b0df5b4cd98fc7d669f777a7403aa89c" Oct 30 04:06:12.092885 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:06:12.092870 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a725c6d735b2f41a05a7935ad2dfe9e7b0df5b4cd98fc7d669f777a7403aa89c\": container with ID starting with a725c6d735b2f41a05a7935ad2dfe9e7b0df5b4cd98fc7d669f777a7403aa89c not found: ID does not exist" containerID="a725c6d735b2f41a05a7935ad2dfe9e7b0df5b4cd98fc7d669f777a7403aa89c" Oct 30 04:06:12.092931 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:12.092889 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a725c6d735b2f41a05a7935ad2dfe9e7b0df5b4cd98fc7d669f777a7403aa89c"} err="failed to get container status \"a725c6d735b2f41a05a7935ad2dfe9e7b0df5b4cd98fc7d669f777a7403aa89c\": rpc error: code = NotFound desc = could not find container \"a725c6d735b2f41a05a7935ad2dfe9e7b0df5b4cd98fc7d669f777a7403aa89c\": container with ID starting with a725c6d735b2f41a05a7935ad2dfe9e7b0df5b4cd98fc7d669f777a7403aa89c not found: ID does not exist" Oct 30 04:06:12.092931 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:12.092903 5882 scope.go:122] "RemoveContainer" containerID="47e9b3e35d81cf832e7bde4d962a3d0934b762883c65bdd8f714c8bee74a53e2" Oct 30 04:06:12.093111 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:06:12.093095 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e9b3e35d81cf832e7bde4d962a3d0934b762883c65bdd8f714c8bee74a53e2\": container with ID starting with 47e9b3e35d81cf832e7bde4d962a3d0934b762883c65bdd8f714c8bee74a53e2 not found: ID does not exist" containerID="47e9b3e35d81cf832e7bde4d962a3d0934b762883c65bdd8f714c8bee74a53e2" Oct 30 04:06:12.093152 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:12.093115 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e9b3e35d81cf832e7bde4d962a3d0934b762883c65bdd8f714c8bee74a53e2"} err="failed to get container status \"47e9b3e35d81cf832e7bde4d962a3d0934b762883c65bdd8f714c8bee74a53e2\": rpc error: code = NotFound desc = could not find container \"47e9b3e35d81cf832e7bde4d962a3d0934b762883c65bdd8f714c8bee74a53e2\": container with ID starting with 47e9b3e35d81cf832e7bde4d962a3d0934b762883c65bdd8f714c8bee74a53e2 not found: ID does not exist" Oct 30 04:06:12.127381 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:12.127354 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:06:12.140916 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:12.140890 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:06:14.009678 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:14.009642 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" path="/var/lib/kubelet/pods/a0c665b2-5fe7-4d2c-b3f8-136d6929bc07/volumes" Oct 30 04:06:17.046728 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:17.046696 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/seaweedfs" Oct 30 04:06:17.535401 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:17.535361 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 30 04:06:17.535736 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:17.535723 5882 cpu_manager.go:414] "containerMap: removing container" podUID="a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" containerName="alertmanager" Oct 30 04:06:17.535800 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:17.535737 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" containerName="alertmanager" Oct 30 04:06:17.535800 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:17.535764 5882 cpu_manager.go:414] "containerMap: removing container" podUID="a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" containerName="init-config-reloader" Oct 30 04:06:17.535800 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:17.535768 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" containerName="init-config-reloader" Oct 30 04:06:17.535800 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:17.535777 5882 cpu_manager.go:414] "containerMap: removing container" podUID="a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" containerName="config-reloader" Oct 30 04:06:17.535800 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:17.535782 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" containerName="config-reloader" Oct 30 04:06:17.535985 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:17.535832 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" containerName="alertmanager" Oct 30 04:06:17.535985 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:17.535846 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c665b2-5fe7-4d2c-b3f8-136d6929bc07" containerName="config-reloader" Oct 30 04:06:17.538980 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:17.538963 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 04:06:17.557813 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:17.557782 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 30 04:06:17.622572 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:17.622537 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9dpx\" (UniqueName: \"kubernetes.io/projected/5d2b141d-937e-42c8-a33b-884ee2617959-kube-api-access-h9dpx\") pod \"seaweedfs-bucket-creator\" (UID: \"5d2b141d-937e-42c8-a33b-884ee2617959\") " pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 04:06:17.723216 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:17.723178 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9dpx\" (UniqueName: \"kubernetes.io/projected/5d2b141d-937e-42c8-a33b-884ee2617959-kube-api-access-h9dpx\") pod \"seaweedfs-bucket-creator\" (UID: \"5d2b141d-937e-42c8-a33b-884ee2617959\") " pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 04:06:17.748670 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:17.748641 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9dpx\" (UniqueName: \"kubernetes.io/projected/5d2b141d-937e-42c8-a33b-884ee2617959-kube-api-access-h9dpx\") pod \"seaweedfs-bucket-creator\" (UID: \"5d2b141d-937e-42c8-a33b-884ee2617959\") " pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 04:06:17.850787 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:17.850707 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 04:06:18.140112 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:18.140082 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 30 04:06:18.140841 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:06:18.140814 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d2b141d_937e_42c8_a33b_884ee2617959.slice/crio-4cb4e87ad9efdb6bf830d776a7a66c16adf0d3fd9d6ac00d05625a5f9ddb00a3 WatchSource:0}: Error finding container 4cb4e87ad9efdb6bf830d776a7a66c16adf0d3fd9d6ac00d05625a5f9ddb00a3: Status 404 returned error can't find the container with id 4cb4e87ad9efdb6bf830d776a7a66c16adf0d3fd9d6ac00d05625a5f9ddb00a3 Oct 30 04:06:19.102044 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:19.101966 5882 generic.go:356] "Generic (PLEG): container finished" podID="5d2b141d-937e-42c8-a33b-884ee2617959" containerID="bba2bb1bb8ff71a09f27c6f7c65b47d925f9493b44a9fccceb05a3b75b4f1aa3" exitCode=0 Oct 30 04:06:19.102184 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:19.102053 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" event={"ID":"5d2b141d-937e-42c8-a33b-884ee2617959","Type":"ContainerDied","Data":"bba2bb1bb8ff71a09f27c6f7c65b47d925f9493b44a9fccceb05a3b75b4f1aa3"} Oct 30 04:06:19.102184 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:19.102095 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" event={"ID":"5d2b141d-937e-42c8-a33b-884ee2617959","Type":"ContainerStarted","Data":"4cb4e87ad9efdb6bf830d776a7a66c16adf0d3fd9d6ac00d05625a5f9ddb00a3"} Oct 30 04:06:20.235740 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:20.235711 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 04:06:20.345896 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:20.345863 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d2b141d-937e-42c8-a33b-884ee2617959-kube-api-access-h9dpx\" (UniqueName: \"kubernetes.io/projected/5d2b141d-937e-42c8-a33b-884ee2617959-kube-api-access-h9dpx\") pod \"5d2b141d-937e-42c8-a33b-884ee2617959\" (UID: \"5d2b141d-937e-42c8-a33b-884ee2617959\") " Oct 30 04:06:20.347911 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:20.347882 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d2b141d-937e-42c8-a33b-884ee2617959-kube-api-access-h9dpx" pod "5d2b141d-937e-42c8-a33b-884ee2617959" (UID: "5d2b141d-937e-42c8-a33b-884ee2617959"). InnerVolumeSpecName "kube-api-access-h9dpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:20.446688 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:20.446595 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9dpx\" (UniqueName: \"kubernetes.io/projected/5d2b141d-937e-42c8-a33b-884ee2617959-kube-api-access-h9dpx\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:06:21.111780 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:21.111749 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 30 04:06:21.111951 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:21.111790 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" event={"ID":"5d2b141d-937e-42c8-a33b-884ee2617959","Type":"ContainerDied","Data":"4cb4e87ad9efdb6bf830d776a7a66c16adf0d3fd9d6ac00d05625a5f9ddb00a3"} Oct 30 04:06:21.111951 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:21.111825 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cb4e87ad9efdb6bf830d776a7a66c16adf0d3fd9d6ac00d05625a5f9ddb00a3" Oct 30 04:06:23.943341 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:06:23.943301 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:06:24.355345 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.355308 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt"] Oct 30 04:06:24.355888 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.355863 5882 cpu_manager.go:414] "containerMap: removing container" podUID="5d2b141d-937e-42c8-a33b-884ee2617959" containerName="create-bucket" Oct 30 04:06:24.355888 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.355888 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d2b141d-937e-42c8-a33b-884ee2617959" containerName="create-bucket" Oct 30 04:06:24.356062 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.356007 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d2b141d-937e-42c8-a33b-884ee2617959" containerName="create-bucket" Oct 30 04:06:24.359626 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.359590 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:06:24.390403 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.390371 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ca-bundle\"" Oct 30 04:06:24.390571 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.390436 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-distributor-http\"" Oct 30 04:06:24.390706 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.390606 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-dockercfg-qxnzq\"" Oct 30 04:06:24.390706 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.390675 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-distributor-grpc\"" Oct 30 04:06:24.391718 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.391691 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-config\"" Oct 30 04:06:24.391848 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.391827 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt"] Oct 30 04:06:24.486332 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.486300 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-pk2nt\" (UID: \"32a46854-54c3-4e02-9351-02b2667b60ff\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:06:24.486535 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.486348 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/32a46854-54c3-4e02-9351-02b2667b60ff-config\") pod \"data-science-lokistack-distributor-58947679b7-pk2nt\" (UID: \"32a46854-54c3-4e02-9351-02b2667b60ff\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:06:24.486535 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.486381 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g44sx\" (UniqueName: \"kubernetes.io/projected/32a46854-54c3-4e02-9351-02b2667b60ff-kube-api-access-g44sx\") pod \"data-science-lokistack-distributor-58947679b7-pk2nt\" (UID: \"32a46854-54c3-4e02-9351-02b2667b60ff\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:06:24.486535 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.486500 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-pk2nt\" (UID: \"32a46854-54c3-4e02-9351-02b2667b60ff\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:06:24.486733 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.486555 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-pk2nt\" (UID: \"32a46854-54c3-4e02-9351-02b2667b60ff\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:06:24.587089 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.587055 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-pk2nt\" (UID: \"32a46854-54c3-4e02-9351-02b2667b60ff\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:06:24.587280 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.587133 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-pk2nt\" (UID: \"32a46854-54c3-4e02-9351-02b2667b60ff\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:06:24.587280 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.587175 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/32a46854-54c3-4e02-9351-02b2667b60ff-config\") pod \"data-science-lokistack-distributor-58947679b7-pk2nt\" (UID: \"32a46854-54c3-4e02-9351-02b2667b60ff\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:06:24.587280 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.587198 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g44sx\" (UniqueName: \"kubernetes.io/projected/32a46854-54c3-4e02-9351-02b2667b60ff-kube-api-access-g44sx\") pod \"data-science-lokistack-distributor-58947679b7-pk2nt\" (UID: \"32a46854-54c3-4e02-9351-02b2667b60ff\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:06:24.587280 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.587230 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-pk2nt\" (UID: \"32a46854-54c3-4e02-9351-02b2667b60ff\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:06:24.588340 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.588308 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-pk2nt\" (UID: \"32a46854-54c3-4e02-9351-02b2667b60ff\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:06:24.588437 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.588421 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/32a46854-54c3-4e02-9351-02b2667b60ff-config\") pod \"data-science-lokistack-distributor-58947679b7-pk2nt\" (UID: \"32a46854-54c3-4e02-9351-02b2667b60ff\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:06:24.589628 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.589594 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-pk2nt\" (UID: \"32a46854-54c3-4e02-9351-02b2667b60ff\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:06:24.589698 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.589670 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-pk2nt\" (UID: \"32a46854-54c3-4e02-9351-02b2667b60ff\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:06:24.616498 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.616430 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g44sx\" (UniqueName: \"kubernetes.io/projected/32a46854-54c3-4e02-9351-02b2667b60ff-kube-api-access-g44sx\") pod \"data-science-lokistack-distributor-58947679b7-pk2nt\" (UID: \"32a46854-54c3-4e02-9351-02b2667b60ff\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:06:24.618543 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.618512 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4"] Oct 30 04:06:24.623595 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.623571 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:24.643233 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.643192 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-shared-secret\"" Oct 30 04:06:24.644977 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.644568 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4"] Oct 30 04:06:24.646480 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.646459 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-querier-grpc\"" Oct 30 04:06:24.646599 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.646530 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-querier-http\"" Oct 30 04:06:24.671896 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.671874 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:06:24.688322 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.688170 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-test-loki-shared-secret\") pod \"data-science-lokistack-querier-84985db556-zk6g4\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:24.688322 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.688226 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-84985db556-zk6g4\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:24.688322 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.688255 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-84985db556-zk6g4\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:24.688322 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.688278 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8tvc\" (UniqueName: \"kubernetes.io/projected/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-kube-api-access-t8tvc\") pod \"data-science-lokistack-querier-84985db556-zk6g4\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:24.688557 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.688436 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-84985db556-zk6g4\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:24.688557 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.688499 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-config\") pod \"data-science-lokistack-querier-84985db556-zk6g4\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:24.780375 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.780340 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4"] Oct 30 04:06:24.784943 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.784921 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:06:24.789234 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.789204 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-84985db556-zk6g4\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:24.789362 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.789241 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8tvc\" (UniqueName: \"kubernetes.io/projected/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-kube-api-access-t8tvc\") pod \"data-science-lokistack-querier-84985db556-zk6g4\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:24.789362 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.789305 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-84985db556-zk6g4\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:24.789362 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.789354 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-config\") pod \"data-science-lokistack-querier-84985db556-zk6g4\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:24.789487 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.789400 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-test-loki-shared-secret\") pod \"data-science-lokistack-querier-84985db556-zk6g4\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:24.789487 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.789447 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-84985db556-zk6g4\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:24.790084 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.790057 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-84985db556-zk6g4\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:24.790332 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.790281 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-config\") pod \"data-science-lokistack-querier-84985db556-zk6g4\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:24.791994 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.791965 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-84985db556-zk6g4\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:24.792599 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.792573 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-84985db556-zk6g4\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:24.792748 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.792733 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-test-loki-shared-secret\") pod \"data-science-lokistack-querier-84985db556-zk6g4\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:24.807349 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.807327 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4"] Oct 30 04:06:24.819133 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.819106 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8tvc\" (UniqueName: \"kubernetes.io/projected/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-kube-api-access-t8tvc\") pod \"data-science-lokistack-querier-84985db556-zk6g4\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:24.844028 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.843995 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-query-frontend-grpc\"" Oct 30 04:06:24.844148 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.844072 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-query-frontend-http\"" Oct 30 04:06:24.890968 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.890881 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-f746fc66d-29jh4\" (UID: \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:06:24.890968 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.890931 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e43ef35-ea6c-4c3e-9099-34273546b4f4-config\") pod \"data-science-lokistack-query-frontend-f746fc66d-29jh4\" (UID: \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:06:24.890968 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.890951 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-f746fc66d-29jh4\" (UID: \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:06:24.891207 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.891057 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgqxl\" (UniqueName: \"kubernetes.io/projected/9e43ef35-ea6c-4c3e-9099-34273546b4f4-kube-api-access-dgqxl\") pod \"data-science-lokistack-query-frontend-f746fc66d-29jh4\" (UID: \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:06:24.891207 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.891115 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-f746fc66d-29jh4\" (UID: \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:06:24.935281 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.935256 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:24.975328 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.975248 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt"] Oct 30 04:06:24.976142 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:06:24.976084 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a46854_54c3_4e02_9351_02b2667b60ff.slice/crio-7095535b7c2fb0382283ad431ea5c9c01750295ff0cdd0d6d08c3e1c82b9fd0e WatchSource:0}: Error finding container 7095535b7c2fb0382283ad431ea5c9c01750295ff0cdd0d6d08c3e1c82b9fd0e: Status 404 returned error can't find the container with id 7095535b7c2fb0382283ad431ea5c9c01750295ff0cdd0d6d08c3e1c82b9fd0e Oct 30 04:06:24.978192 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.978172 5882 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:06:24.992253 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.992225 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-f746fc66d-29jh4\" (UID: \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:06:24.992361 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.992293 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-f746fc66d-29jh4\" (UID: \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:06:24.992361 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.992331 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e43ef35-ea6c-4c3e-9099-34273546b4f4-config\") pod \"data-science-lokistack-query-frontend-f746fc66d-29jh4\" (UID: \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:06:24.992875 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.992849 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-f746fc66d-29jh4\" (UID: \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:06:24.992965 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.992916 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgqxl\" (UniqueName: \"kubernetes.io/projected/9e43ef35-ea6c-4c3e-9099-34273546b4f4-kube-api-access-dgqxl\") pod \"data-science-lokistack-query-frontend-f746fc66d-29jh4\" (UID: \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:06:24.993230 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.993147 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-f746fc66d-29jh4\" (UID: \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:06:24.993230 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.993152 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e43ef35-ea6c-4c3e-9099-34273546b4f4-config\") pod \"data-science-lokistack-query-frontend-f746fc66d-29jh4\" (UID: \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:06:24.994989 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.994968 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-f746fc66d-29jh4\" (UID: \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:06:24.995361 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:24.995318 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-f746fc66d-29jh4\" (UID: \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:06:25.024156 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:25.024128 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgqxl\" (UniqueName: \"kubernetes.io/projected/9e43ef35-ea6c-4c3e-9099-34273546b4f4-kube-api-access-dgqxl\") pod \"data-science-lokistack-query-frontend-f746fc66d-29jh4\" (UID: \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:06:25.099335 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:25.099305 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:06:25.130755 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:25.130716 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" event={"ID":"32a46854-54c3-4e02-9351-02b2667b60ff","Type":"ContainerStarted","Data":"7095535b7c2fb0382283ad431ea5c9c01750295ff0cdd0d6d08c3e1c82b9fd0e"} Oct 30 04:06:25.231585 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:25.231548 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4"] Oct 30 04:06:25.401755 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:25.401724 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4"] Oct 30 04:06:25.402890 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:06:25.402852 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e43ef35_ea6c_4c3e_9099_34273546b4f4.slice/crio-afa4369a13a8c80938d1915db1f9e52cbadb957dd8fdadf5b7f63972e5e31a36 WatchSource:0}: Error finding container afa4369a13a8c80938d1915db1f9e52cbadb957dd8fdadf5b7f63972e5e31a36: Status 404 returned error can't find the container with id afa4369a13a8c80938d1915db1f9e52cbadb957dd8fdadf5b7f63972e5e31a36 Oct 30 04:06:26.138326 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:26.138286 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" event={"ID":"9e43ef35-ea6c-4c3e-9099-34273546b4f4","Type":"ContainerStarted","Data":"afa4369a13a8c80938d1915db1f9e52cbadb957dd8fdadf5b7f63972e5e31a36"} Oct 30 04:06:26.140098 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:26.140068 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" event={"ID":"0b6a3bc2-747b-4d2e-af39-5955e57ab37f","Type":"ContainerStarted","Data":"916d4d32d8694d1dff11ee27bf852c5c73455efed352e8c657a7043bdc3ac9c6"} Oct 30 04:06:28.520740 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:28.520712 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 04:06:28.537260 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:28.537221 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 04:06:30.167259 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:30.167222 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" event={"ID":"9e43ef35-ea6c-4c3e-9099-34273546b4f4","Type":"ContainerStarted","Data":"7e65e96b04357f9b5a446c0f5b4d60da08bf3933a013c4b32ce56b3c38b52c63"} Oct 30 04:06:30.167750 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:30.167334 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:06:30.169084 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:30.169053 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" event={"ID":"0b6a3bc2-747b-4d2e-af39-5955e57ab37f","Type":"ContainerStarted","Data":"4be12c08caa92d3501d4d1916992727df895ad92f4060916d32b406f3c71eeb9"} Oct 30 04:06:30.169221 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:30.169166 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:30.170639 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:30.170597 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" event={"ID":"32a46854-54c3-4e02-9351-02b2667b60ff","Type":"ContainerStarted","Data":"086f5e217c7e8dd1ab4ed65b0f01f1b6028ea0686818a0dc1affd1b2a732438a"} Oct 30 04:06:30.170757 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:30.170734 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:06:30.222297 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:30.222242 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" podStartSLOduration=1.539799742 podStartE2EDuration="6.222228075s" totalImagesPullingTime="4.682428333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:06:30.221090779 +0000 UTC m=+5102.335405764" watchObservedRunningTime="2026-10-30 04:06:30.222228075 +0000 UTC m=+5102.336543047" Oct 30 04:06:30.267650 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:30.267556 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" podStartSLOduration=1.7497190919999999 podStartE2EDuration="6.267544276s" totalImagesPullingTime="4.517825184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:06:30.266399657 +0000 UTC m=+5102.380714644" watchObservedRunningTime="2026-10-30 04:06:30.267544276 +0000 UTC m=+5102.381859244" Oct 30 04:06:30.312516 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:30.312468 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" podStartSLOduration=1.760800323 podStartE2EDuration="6.312455223s" totalImagesPullingTime="4.5516549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:06:30.311598949 +0000 UTC m=+5102.425913939" watchObservedRunningTime="2026-10-30 04:06:30.312455223 +0000 UTC m=+5102.426770198" Oct 30 04:06:51.182465 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:51.182434 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:06:51.183049 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:51.182761 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:06:51.183049 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:06:51.182915 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:06:53.942457 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:06:53.942413 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:07:23.942179 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:07:23.942101 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:07:33.685167 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.685125 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 04:07:33.688858 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.688841 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:33.719409 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.719354 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-http\"" Oct 30 04:07:33.719409 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.719361 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-grpc\"" Oct 30 04:07:33.720638 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.720603 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 04:07:33.808756 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.808724 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:33.808931 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.808765 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-test-loki-shared-secret\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:33.808931 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.808896 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvvrq\" (UniqueName: \"kubernetes.io/projected/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-kube-api-access-gvvrq\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:33.809005 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.808952 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:33.809005 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.808997 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-config\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:33.809082 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.809022 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:33.910070 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.910031 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvvrq\" (UniqueName: \"kubernetes.io/projected/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-kube-api-access-gvvrq\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:33.910277 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.910089 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:33.910277 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.910125 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-config\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:33.910277 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.910143 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:33.910277 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.910176 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:33.910277 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.910194 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-test-loki-shared-secret\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:33.911021 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.910993 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:33.911234 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.911208 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-config\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:33.912889 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.912859 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:33.912977 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.912872 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:33.913019 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.912997 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-test-loki-shared-secret\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:33.935283 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:33.935231 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvvrq\" (UniqueName: \"kubernetes.io/projected/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-kube-api-access-gvvrq\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:37.039433 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:37.039388 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:37.071878 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:37.071850 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:37.140869 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:37.140840 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:37.140996 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:37.140972 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:37.157096 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:37.157072 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") DevicePath \"csi-22fcec6f0f615271241a8359c17050656955dbc14c2774ac90939cf3f839f77f\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:37.400449 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:37.400411 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/fcf6f0536bbc97a5651ec9a8d066f70948542721519078012a300fe6ca3edfd0/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:37.426639 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:37.426595 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:38.149997 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:38.149954 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:38.182624 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:38.182579 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:38.251461 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:38.251429 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:38.251649 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:38.251534 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:38.268007 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:38.267974 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") DevicePath \"csi-40758f0eeb2ac4c8710af85cac17ce349acc25d2cc9087a81d87ac54648d93f9\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:45.589719 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:45.589684 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/be8d3d2716806c45d69e88ad453cb51178b1b3855120bf6702b7eff86c093a7e/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:45.615515 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:45.615491 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:45.700489 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:45.700460 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:46.007432 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:07:46.007404 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf526accb_9fcb_4480_bc5b_8ae927d2ed7d.slice/crio-e8b3b5e1d0f771130e59ca2e530dde8eb3e62da41b339b45429ef26f2b0a431b WatchSource:0}: Error finding container e8b3b5e1d0f771130e59ca2e530dde8eb3e62da41b339b45429ef26f2b0a431b: Status 404 returned error can't find the container with id e8b3b5e1d0f771130e59ca2e530dde8eb3e62da41b339b45429ef26f2b0a431b Oct 30 04:07:46.009823 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:46.009803 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 04:07:46.518100 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:46.518052 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"f526accb-9fcb-4480-bc5b-8ae927d2ed7d","Type":"ContainerStarted","Data":"eaefef74345dfc1371ebdfa3002812c0455558e23de6948e7ddfec166a7e5b8d"} Oct 30 04:07:46.518285 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:46.518133 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"f526accb-9fcb-4480-bc5b-8ae927d2ed7d","Type":"ContainerStarted","Data":"e8b3b5e1d0f771130e59ca2e530dde8eb3e62da41b339b45429ef26f2b0a431b"} Oct 30 04:07:46.518285 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:46.518183 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:07:46.568729 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:07:46.568685 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podStartSLOduration=16.568672628 podStartE2EDuration="16.568672628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:07:46.567925222 +0000 UTC m=+5178.682240202" watchObservedRunningTime="2026-10-30 04:07:46.568672628 +0000 UTC m=+5178.682987597" Oct 30 04:07:53.942581 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:07:53.942544 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:08:07.528765 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:08:07.528729 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="f526accb-9fcb-4480-bc5b-8ae927d2ed7d" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:08:17.527299 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:08:17.527257 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="f526accb-9fcb-4480-bc5b-8ae927d2ed7d" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:08:23.942531 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:08:23.942493 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:08:27.527914 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:08:27.527872 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="f526accb-9fcb-4480-bc5b-8ae927d2ed7d" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:08:37.527674 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:08:37.527643 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:08:53.942705 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:08:53.942667 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:09:23.943103 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:09:23.943067 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:09:53.942403 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:09:53.942366 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:10:05.229926 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:05.229890 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq"] Oct 30 04:10:05.233464 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:05.233438 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq" Oct 30 04:10:05.255199 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:05.255165 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq"] Oct 30 04:10:05.266852 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:05.266825 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-94m9t\"" Oct 30 04:10:05.322079 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:05.322052 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-6a6878d3\"" Oct 30 04:10:05.362222 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:05.362188 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv7vf\" (UniqueName: \"kubernetes.io/projected/8d389a55-b7a2-452d-9e20-a3cb31f4e946-kube-api-access-kv7vf\") pod \"data-science-usage-logs-collector-76867dbf9c-8vbhq\" (UID: \"8d389a55-b7a2-452d-9e20-a3cb31f4e946\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq" Oct 30 04:10:05.362364 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:05.362237 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8d389a55-b7a2-452d-9e20-a3cb31f4e946-otc-internal\") pod \"data-science-usage-logs-collector-76867dbf9c-8vbhq\" (UID: \"8d389a55-b7a2-452d-9e20-a3cb31f4e946\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq" Oct 30 04:10:05.463543 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:05.463511 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv7vf\" (UniqueName: \"kubernetes.io/projected/8d389a55-b7a2-452d-9e20-a3cb31f4e946-kube-api-access-kv7vf\") pod \"data-science-usage-logs-collector-76867dbf9c-8vbhq\" (UID: \"8d389a55-b7a2-452d-9e20-a3cb31f4e946\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq" Oct 30 04:10:05.463749 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:05.463552 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8d389a55-b7a2-452d-9e20-a3cb31f4e946-otc-internal\") pod \"data-science-usage-logs-collector-76867dbf9c-8vbhq\" (UID: \"8d389a55-b7a2-452d-9e20-a3cb31f4e946\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq" Oct 30 04:10:05.464255 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:05.464230 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8d389a55-b7a2-452d-9e20-a3cb31f4e946-otc-internal\") pod \"data-science-usage-logs-collector-76867dbf9c-8vbhq\" (UID: \"8d389a55-b7a2-452d-9e20-a3cb31f4e946\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq" Oct 30 04:10:05.491401 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:05.491343 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv7vf\" (UniqueName: \"kubernetes.io/projected/8d389a55-b7a2-452d-9e20-a3cb31f4e946-kube-api-access-kv7vf\") pod \"data-science-usage-logs-collector-76867dbf9c-8vbhq\" (UID: \"8d389a55-b7a2-452d-9e20-a3cb31f4e946\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq" Oct 30 04:10:05.548447 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:05.548424 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq" Oct 30 04:10:05.842605 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:05.842571 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq"] Oct 30 04:10:05.844298 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:10:05.844265 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d389a55_b7a2_452d_9e20_a3cb31f4e946.slice/crio-15da4081c9d061906cb28394c69ac9f798da55d0db9fbd71ec849319f89ba554 WatchSource:0}: Error finding container 15da4081c9d061906cb28394c69ac9f798da55d0db9fbd71ec849319f89ba554: Status 404 returned error can't find the container with id 15da4081c9d061906cb28394c69ac9f798da55d0db9fbd71ec849319f89ba554 Oct 30 04:10:05.846414 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:05.846383 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:10:05.846488 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:05.846443 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:10:06.147970 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:06.147878 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq" event={"ID":"8d389a55-b7a2-452d-9e20-a3cb31f4e946","Type":"ContainerStarted","Data":"13b81279dfcb1307573c175fb65ef72fe36b75462a9f7a3eb39617df99a5655b"} Oct 30 04:10:06.147970 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:06.147952 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq" event={"ID":"8d389a55-b7a2-452d-9e20-a3cb31f4e946","Type":"ContainerStarted","Data":"15da4081c9d061906cb28394c69ac9f798da55d0db9fbd71ec849319f89ba554"} Oct 30 04:10:06.192528 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:06.192478 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq" podStartSLOduration=1.192465754 podStartE2EDuration="1.192465754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:10:06.192350227 +0000 UTC m=+5318.306665202" watchObservedRunningTime="2026-10-30 04:10:06.192465754 +0000 UTC m=+5318.306780708" Oct 30 04:10:10.973597 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:10.973515 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-1"] Oct 30 04:10:10.977268 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:10.977250 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:11.008837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.008807 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-lifecycle-secret\"" Oct 30 04:10:11.008837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.008831 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-http\"" Oct 30 04:10:11.009056 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.008810 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-grpc\"" Oct 30 04:10:11.010131 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.010106 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-1"] Oct 30 04:10:11.115247 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.115217 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:11.115509 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.115488 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:11.115550 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.115532 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:11.115664 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.115650 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:11.115719 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.115706 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkktg\" (UniqueName: \"kubernetes.io/projected/7dc28de7-7003-4eb8-a215-ebc18c242eee-kube-api-access-kkktg\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:11.115813 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.115799 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7dc28de7-7003-4eb8-a215-ebc18c242eee-config\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:11.216776 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.216741 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:11.216983 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.216789 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:11.216983 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.216817 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:11.216983 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.216852 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkktg\" (UniqueName: \"kubernetes.io/projected/7dc28de7-7003-4eb8-a215-ebc18c242eee-kube-api-access-kkktg\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:11.216983 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.216873 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7dc28de7-7003-4eb8-a215-ebc18c242eee-config\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:11.216983 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.216907 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:11.217659 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.217631 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:11.217828 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.217802 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7dc28de7-7003-4eb8-a215-ebc18c242eee-config\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:11.219417 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.219391 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:11.219417 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.219404 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:11.219541 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.219443 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:11.241794 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:11.241746 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkktg\" (UniqueName: \"kubernetes.io/projected/7dc28de7-7003-4eb8-a215-ebc18c242eee-kube-api-access-kkktg\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:12.688303 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:12.688263 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq"] Oct 30 04:10:12.688729 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:12.688435 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq" podUID="8d389a55-b7a2-452d-9e20-a3cb31f4e946" containerName="otc-container" containerID="cri-o://13b81279dfcb1307573c175fb65ef72fe36b75462a9f7a3eb39617df99a5655b" gracePeriod=30 Oct 30 04:10:12.958016 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:12.957991 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq" Oct 30 04:10:13.033532 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:13.033496 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d389a55-b7a2-452d-9e20-a3cb31f4e946-otc-internal\" (UniqueName: \"kubernetes.io/configmap/8d389a55-b7a2-452d-9e20-a3cb31f4e946-otc-internal\") pod \"8d389a55-b7a2-452d-9e20-a3cb31f4e946\" (UID: \"8d389a55-b7a2-452d-9e20-a3cb31f4e946\") " Oct 30 04:10:13.033720 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:13.033557 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d389a55-b7a2-452d-9e20-a3cb31f4e946-kube-api-access-kv7vf\" (UniqueName: \"kubernetes.io/projected/8d389a55-b7a2-452d-9e20-a3cb31f4e946-kube-api-access-kv7vf\") pod \"8d389a55-b7a2-452d-9e20-a3cb31f4e946\" (UID: \"8d389a55-b7a2-452d-9e20-a3cb31f4e946\") " Oct 30 04:10:13.033868 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:13.033842 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d389a55-b7a2-452d-9e20-a3cb31f4e946-otc-internal" pod "8d389a55-b7a2-452d-9e20-a3cb31f4e946" (UID: "8d389a55-b7a2-452d-9e20-a3cb31f4e946"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:13.035439 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:13.035419 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d389a55-b7a2-452d-9e20-a3cb31f4e946-kube-api-access-kv7vf" pod "8d389a55-b7a2-452d-9e20-a3cb31f4e946" (UID: "8d389a55-b7a2-452d-9e20-a3cb31f4e946"). InnerVolumeSpecName "kube-api-access-kv7vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:13.135731 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:13.135700 5882 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8d389a55-b7a2-452d-9e20-a3cb31f4e946-otc-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:10:13.135731 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:13.135723 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv7vf\" (UniqueName: \"kubernetes.io/projected/8d389a55-b7a2-452d-9e20-a3cb31f4e946-kube-api-access-kv7vf\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:10:13.179686 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:13.179655 5882 generic.go:356] "Generic (PLEG): container finished" podID="8d389a55-b7a2-452d-9e20-a3cb31f4e946" containerID="13b81279dfcb1307573c175fb65ef72fe36b75462a9f7a3eb39617df99a5655b" exitCode=0 Oct 30 04:10:13.179831 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:13.179703 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq" Oct 30 04:10:13.179831 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:13.179740 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq" event={"ID":"8d389a55-b7a2-452d-9e20-a3cb31f4e946","Type":"ContainerDied","Data":"13b81279dfcb1307573c175fb65ef72fe36b75462a9f7a3eb39617df99a5655b"} Oct 30 04:10:13.179831 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:13.179791 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq" event={"ID":"8d389a55-b7a2-452d-9e20-a3cb31f4e946","Type":"ContainerDied","Data":"15da4081c9d061906cb28394c69ac9f798da55d0db9fbd71ec849319f89ba554"} Oct 30 04:10:13.179994 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:13.179865 5882 scope.go:122] "RemoveContainer" containerID="13b81279dfcb1307573c175fb65ef72fe36b75462a9f7a3eb39617df99a5655b" Oct 30 04:10:13.191731 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:13.191712 5882 scope.go:122] "RemoveContainer" containerID="13b81279dfcb1307573c175fb65ef72fe36b75462a9f7a3eb39617df99a5655b" Oct 30 04:10:13.192117 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:10:13.192090 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b81279dfcb1307573c175fb65ef72fe36b75462a9f7a3eb39617df99a5655b\": container with ID starting with 13b81279dfcb1307573c175fb65ef72fe36b75462a9f7a3eb39617df99a5655b not found: ID does not exist" containerID="13b81279dfcb1307573c175fb65ef72fe36b75462a9f7a3eb39617df99a5655b" Oct 30 04:10:13.192217 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:13.192123 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b81279dfcb1307573c175fb65ef72fe36b75462a9f7a3eb39617df99a5655b"} err="failed to get container status \"13b81279dfcb1307573c175fb65ef72fe36b75462a9f7a3eb39617df99a5655b\": rpc error: code = NotFound desc = could not find container \"13b81279dfcb1307573c175fb65ef72fe36b75462a9f7a3eb39617df99a5655b\": container with ID starting with 13b81279dfcb1307573c175fb65ef72fe36b75462a9f7a3eb39617df99a5655b not found: ID does not exist" Oct 30 04:10:13.235966 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:13.235889 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq"] Oct 30 04:10:13.244737 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:13.244699 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-8vbhq"] Oct 30 04:10:14.009484 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:14.009454 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d389a55-b7a2-452d-9e20-a3cb31f4e946" path="/var/lib/kubelet/pods/8d389a55-b7a2-452d-9e20-a3cb31f4e946/volumes" Oct 30 04:10:23.942668 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:10:23.942610 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:10:24.038446 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:24.038411 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:24.071999 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:24.071961 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:24.139389 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:24.139355 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:24.139565 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:24.139498 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:24.155810 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:24.155778 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") DevicePath \"csi-51b362136bc4ecc02770f4e1d3595c1c59eb604f2b3d12558a9a3323d6a31529\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:24.263993 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:24.263961 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/66d42b06118341040bdf4acaf0eeae99e9dd1cda49be064756edb60338b51e13/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:24.292210 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:24.292185 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:24.489376 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:24.489340 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:24.791218 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:24.791188 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-1"] Oct 30 04:10:24.792064 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:10:24.792041 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc28de7_7003_4eb8_a215_ebc18c242eee.slice/crio-a334a5684b7783273e135430e8ae876c18ffe822bffcffacb302bfc50053794e WatchSource:0}: Error finding container a334a5684b7783273e135430e8ae876c18ffe822bffcffacb302bfc50053794e: Status 404 returned error can't find the container with id a334a5684b7783273e135430e8ae876c18ffe822bffcffacb302bfc50053794e Oct 30 04:10:25.243718 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:25.243604 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" event={"ID":"7dc28de7-7003-4eb8-a215-ebc18c242eee","Type":"ContainerStarted","Data":"0d41dffb6b2a56bec89032258ec53dc56dc2a89bd7d537f95791a9549814a11d"} Oct 30 04:10:25.244163 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:25.243721 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:25.244163 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:25.243742 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" event={"ID":"7dc28de7-7003-4eb8-a215-ebc18c242eee","Type":"ContainerStarted","Data":"a334a5684b7783273e135430e8ae876c18ffe822bffcffacb302bfc50053794e"} Oct 30 04:10:25.295629 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:25.295568 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" podStartSLOduration=15.295554497 podStartE2EDuration="15.295554497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:10:25.289731644 +0000 UTC m=+5337.404046620" watchObservedRunningTime="2026-10-30 04:10:25.295554497 +0000 UTC m=+5337.409869462" Oct 30 04:10:29.241172 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:29.241140 5882 scope.go:122] "RemoveContainer" containerID="750aabaf98dfb1874010480f237b38347e552312bf901a0de7c3edfd1bf5465d" Oct 30 04:10:29.249131 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:29.249100 5882 scope.go:122] "RemoveContainer" containerID="5a6dcfb8edb5dcfb7bf284c026727347038dae408594f262e1c9fb89e06c7d32" Oct 30 04:10:29.256660 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:29.256588 5882 scope.go:122] "RemoveContainer" containerID="cf1ea8207257a5f95369a1b252a2bbdee742d9245527643574a918e08fad7cf8" Oct 30 04:10:29.264369 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:29.264348 5882 scope.go:122] "RemoveContainer" containerID="9a2b2cd48c6c111448d7df6a4363a539a48f19c86cb49db527c70508c490b4e4" Oct 30 04:10:29.272965 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:29.272890 5882 scope.go:122] "RemoveContainer" containerID="5da9d414265dddbaa98db024d92d49def1a1af787f2e7006531d003729b0e7fb" Oct 30 04:10:32.685242 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.685209 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4"] Oct 30 04:10:32.685651 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.685411 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" podUID="0b6a3bc2-747b-4d2e-af39-5955e57ab37f" containerName="loki-querier" containerID="cri-o://4be12c08caa92d3501d4d1916992727df895ad92f4060916d32b406f3c71eeb9" gracePeriod=30 Oct 30 04:10:32.736932 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.736898 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb"] Oct 30 04:10:32.737279 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.737267 5882 cpu_manager.go:414] "containerMap: removing container" podUID="8d389a55-b7a2-452d-9e20-a3cb31f4e946" containerName="otc-container" Oct 30 04:10:32.737322 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.737280 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d389a55-b7a2-452d-9e20-a3cb31f4e946" containerName="otc-container" Oct 30 04:10:32.737357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.737341 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d389a55-b7a2-452d-9e20-a3cb31f4e946" containerName="otc-container" Oct 30 04:10:32.742227 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.742208 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:32.762162 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.762132 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb"] Oct 30 04:10:32.815740 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.815710 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-s4ctb\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:32.815924 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.815830 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-s4ctb\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:32.815924 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.815875 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phvd6\" (UniqueName: \"kubernetes.io/projected/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-kube-api-access-phvd6\") pod \"data-science-lokistack-querier-56ffbd5fdc-s4ctb\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:32.815924 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.815899 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-s4ctb\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:32.815924 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.815919 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-s4ctb\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:32.816095 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.816049 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-s4ctb\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:32.917217 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.917180 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-s4ctb\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:32.917217 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.917218 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-s4ctb\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:32.917443 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.917284 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-s4ctb\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:32.917443 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.917307 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-s4ctb\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:32.917443 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.917371 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-s4ctb\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:32.917443 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.917404 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phvd6\" (UniqueName: \"kubernetes.io/projected/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-kube-api-access-phvd6\") pod \"data-science-lokistack-querier-56ffbd5fdc-s4ctb\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:32.918039 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.918011 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-s4ctb\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:32.918201 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.918176 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-s4ctb\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:32.919914 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.919886 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-s4ctb\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:32.920009 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.919989 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-s4ctb\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:32.920159 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.920140 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-s4ctb\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:32.944608 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:32.944556 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phvd6\" (UniqueName: \"kubernetes.io/projected/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-kube-api-access-phvd6\") pod \"data-science-lokistack-querier-56ffbd5fdc-s4ctb\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:33.053324 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.053294 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:33.348546 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.348517 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb"] Oct 30 04:10:33.349535 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:10:33.349504 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffdc3a67_2671_40c6_a23a_7b4bd7d32b3e.slice/crio-666215ff701d40e4a888a181d108d72dcee3496537cc909fb113006329a77693 WatchSource:0}: Error finding container 666215ff701d40e4a888a181d108d72dcee3496537cc909fb113006329a77693: Status 404 returned error can't find the container with id 666215ff701d40e4a888a181d108d72dcee3496537cc909fb113006329a77693 Oct 30 04:10:33.441724 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.441696 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:10:33.523735 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.523694 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-ca-bundle\") pod \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " Oct 30 04:10:33.523897 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.523751 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-querier-grpc\") pod \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " Oct 30 04:10:33.523897 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.523800 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-kube-api-access-t8tvc\" (UniqueName: \"kubernetes.io/projected/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-kube-api-access-t8tvc\") pod \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " Oct 30 04:10:33.523897 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.523839 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-test-loki-shared-secret\") pod \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " Oct 30 04:10:33.523897 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.523867 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-config\" (UniqueName: \"kubernetes.io/configmap/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-config\") pod \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " Oct 30 04:10:33.524131 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.523907 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-querier-http\") pod \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\" (UID: \"0b6a3bc2-747b-4d2e-af39-5955e57ab37f\") " Oct 30 04:10:33.524192 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.524117 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-ca-bundle" pod "0b6a3bc2-747b-4d2e-af39-5955e57ab37f" (UID: "0b6a3bc2-747b-4d2e-af39-5955e57ab37f"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:33.524434 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.524408 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-ca-bundle\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:10:33.524562 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.524472 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-config" pod "0b6a3bc2-747b-4d2e-af39-5955e57ab37f" (UID: "0b6a3bc2-747b-4d2e-af39-5955e57ab37f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:33.526029 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.525998 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-kube-api-access-t8tvc" pod "0b6a3bc2-747b-4d2e-af39-5955e57ab37f" (UID: "0b6a3bc2-747b-4d2e-af39-5955e57ab37f"). InnerVolumeSpecName "kube-api-access-t8tvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:33.526395 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.526365 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-querier-grpc" pod "0b6a3bc2-747b-4d2e-af39-5955e57ab37f" (UID: "0b6a3bc2-747b-4d2e-af39-5955e57ab37f"). InnerVolumeSpecName "data-science-lokistack-querier-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:10:33.526395 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.526382 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-test-loki-shared-secret" pod "0b6a3bc2-747b-4d2e-af39-5955e57ab37f" (UID: "0b6a3bc2-747b-4d2e-af39-5955e57ab37f"). InnerVolumeSpecName "test-loki-shared-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:10:33.526521 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.526433 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-querier-http" pod "0b6a3bc2-747b-4d2e-af39-5955e57ab37f" (UID: "0b6a3bc2-747b-4d2e-af39-5955e57ab37f"). InnerVolumeSpecName "data-science-lokistack-querier-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:10:33.625879 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.625845 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-querier-grpc\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:10:33.625879 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.625873 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8tvc\" (UniqueName: \"kubernetes.io/projected/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-kube-api-access-t8tvc\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:10:33.625879 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.625885 5882 reconciler_common.go:299] "Volume detached for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-test-loki-shared-secret\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:10:33.626099 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.625894 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:10:33.626099 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:33.625903 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/0b6a3bc2-747b-4d2e-af39-5955e57ab37f-data-science-lokistack-querier-http\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:10:34.289941 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:34.289907 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" event={"ID":"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e","Type":"ContainerStarted","Data":"9318038aff90fb922740199622517243196d094fd2c8d878b213db8e46e6eda3"} Oct 30 04:10:34.290364 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:34.289961 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" event={"ID":"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e","Type":"ContainerStarted","Data":"666215ff701d40e4a888a181d108d72dcee3496537cc909fb113006329a77693"} Oct 30 04:10:34.290364 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:34.290025 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:34.291354 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:34.291329 5882 generic.go:356] "Generic (PLEG): container finished" podID="0b6a3bc2-747b-4d2e-af39-5955e57ab37f" containerID="4be12c08caa92d3501d4d1916992727df895ad92f4060916d32b406f3c71eeb9" exitCode=0 Oct 30 04:10:34.291453 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:34.291364 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" Oct 30 04:10:34.291453 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:34.291394 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" event={"ID":"0b6a3bc2-747b-4d2e-af39-5955e57ab37f","Type":"ContainerDied","Data":"4be12c08caa92d3501d4d1916992727df895ad92f4060916d32b406f3c71eeb9"} Oct 30 04:10:34.291453 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:34.291428 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4" event={"ID":"0b6a3bc2-747b-4d2e-af39-5955e57ab37f","Type":"ContainerDied","Data":"916d4d32d8694d1dff11ee27bf852c5c73455efed352e8c657a7043bdc3ac9c6"} Oct 30 04:10:34.291453 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:34.291449 5882 scope.go:122] "RemoveContainer" containerID="4be12c08caa92d3501d4d1916992727df895ad92f4060916d32b406f3c71eeb9" Oct 30 04:10:34.300053 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:34.300037 5882 scope.go:122] "RemoveContainer" containerID="4be12c08caa92d3501d4d1916992727df895ad92f4060916d32b406f3c71eeb9" Oct 30 04:10:34.300331 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:10:34.300310 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4be12c08caa92d3501d4d1916992727df895ad92f4060916d32b406f3c71eeb9\": container with ID starting with 4be12c08caa92d3501d4d1916992727df895ad92f4060916d32b406f3c71eeb9 not found: ID does not exist" containerID="4be12c08caa92d3501d4d1916992727df895ad92f4060916d32b406f3c71eeb9" Oct 30 04:10:34.300423 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:34.300336 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be12c08caa92d3501d4d1916992727df895ad92f4060916d32b406f3c71eeb9"} err="failed to get container status \"4be12c08caa92d3501d4d1916992727df895ad92f4060916d32b406f3c71eeb9\": rpc error: code = NotFound desc = could not find container \"4be12c08caa92d3501d4d1916992727df895ad92f4060916d32b406f3c71eeb9\": container with ID starting with 4be12c08caa92d3501d4d1916992727df895ad92f4060916d32b406f3c71eeb9 not found: ID does not exist" Oct 30 04:10:34.336379 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:34.336340 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" podStartSLOduration=2.336329412 podStartE2EDuration="2.336329412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:10:34.33580979 +0000 UTC m=+5346.450124762" watchObservedRunningTime="2026-10-30 04:10:34.336329412 +0000 UTC m=+5346.450644387" Oct 30 04:10:34.374930 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:34.374905 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4"] Oct 30 04:10:34.379156 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:34.379134 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-zk6g4"] Oct 30 04:10:36.010082 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:36.010048 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6a3bc2-747b-4d2e-af39-5955e57ab37f" path="/var/lib/kubelet/pods/0b6a3bc2-747b-4d2e-af39-5955e57ab37f/volumes" Oct 30 04:10:46.254037 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:46.254006 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:53.941969 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:10:53.941929 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:10:55.301794 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:55.301765 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:10:56.705915 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:56.705880 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 04:10:56.706343 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:56.706109 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="f526accb-9fcb-4480-bc5b-8ae927d2ed7d" containerName="loki-ingester" containerID="cri-o://eaefef74345dfc1371ebdfa3002812c0455558e23de6948e7ddfec166a7e5b8d" gracePeriod=600 Oct 30 04:10:57.527733 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:57.527691 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="f526accb-9fcb-4480-bc5b-8ae927d2ed7d" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:10:58.968443 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:58.968418 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:10:59.165051 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.164969 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-kube-api-access-gvvrq\" (UniqueName: \"kubernetes.io/projected/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-kube-api-access-gvvrq\") pod \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " Oct 30 04:10:59.165051 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.165030 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ingester-grpc\") pod \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " Oct 30 04:10:59.165293 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.165067 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ca-bundle\") pod \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " Oct 30 04:10:59.165293 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.165096 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-config\" (UniqueName: \"kubernetes.io/configmap/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-config\") pod \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " Oct 30 04:10:59.165293 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.165193 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " Oct 30 04:10:59.165460 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.165296 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-test-loki-shared-secret\") pod \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " Oct 30 04:10:59.165460 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.165401 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " Oct 30 04:10:59.165582 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.165461 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ingester-http\") pod \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\" (UID: \"f526accb-9fcb-4480-bc5b-8ae927d2ed7d\") " Oct 30 04:10:59.165582 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.165527 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ca-bundle" pod "f526accb-9fcb-4480-bc5b-8ae927d2ed7d" (UID: "f526accb-9fcb-4480-bc5b-8ae927d2ed7d"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:59.165582 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.165551 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-config" pod "f526accb-9fcb-4480-bc5b-8ae927d2ed7d" (UID: "f526accb-9fcb-4480-bc5b-8ae927d2ed7d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:59.165916 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.165892 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ca-bundle\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:10:59.166035 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.165921 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:10:59.167727 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.167688 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-kube-api-access-gvvrq" pod "f526accb-9fcb-4480-bc5b-8ae927d2ed7d" (UID: "f526accb-9fcb-4480-bc5b-8ae927d2ed7d"). InnerVolumeSpecName "kube-api-access-gvvrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:59.167851 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.167772 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ingester-grpc" pod "f526accb-9fcb-4480-bc5b-8ae927d2ed7d" (UID: "f526accb-9fcb-4480-bc5b-8ae927d2ed7d"). InnerVolumeSpecName "data-science-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:10:59.168075 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.168048 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-test-loki-shared-secret" pod "f526accb-9fcb-4480-bc5b-8ae927d2ed7d" (UID: "f526accb-9fcb-4480-bc5b-8ae927d2ed7d"). InnerVolumeSpecName "test-loki-shared-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:10:59.168161 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.168115 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ingester-http" pod "f526accb-9fcb-4480-bc5b-8ae927d2ed7d" (UID: "f526accb-9fcb-4480-bc5b-8ae927d2ed7d"). InnerVolumeSpecName "data-science-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:10:59.168590 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.168560 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735" pod "f526accb-9fcb-4480-bc5b-8ae927d2ed7d" (UID: "f526accb-9fcb-4480-bc5b-8ae927d2ed7d"). InnerVolumeSpecName "pvc-ea00ca9c-db2e-4885-bfa1-004970abec93". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:10:59.168756 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.168737 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950" pod "f526accb-9fcb-4480-bc5b-8ae927d2ed7d" (UID: "f526accb-9fcb-4480-bc5b-8ae927d2ed7d"). InnerVolumeSpecName "pvc-199f9790-eb98-4bad-8977-a99a3b174eed". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:10:59.266940 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.266907 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ingester-grpc\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:10:59.267104 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.266964 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:10:59.267104 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.266981 5882 reconciler_common.go:299] "Volume detached for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-test-loki-shared-secret\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:10:59.267104 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.267005 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:10:59.267104 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.267020 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-data-science-lokistack-ingester-http\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:10:59.267104 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.267033 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvvrq\" (UniqueName: \"kubernetes.io/projected/f526accb-9fcb-4480-bc5b-8ae927d2ed7d-kube-api-access-gvvrq\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:10:59.279118 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.279100 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-ea00ca9c-db2e-4885-bfa1-004970abec93" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:10:59.281272 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.281256 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-199f9790-eb98-4bad-8977-a99a3b174eed" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:10:59.370942 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.370905 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-40758f0eeb2ac4c8710af85cac17ce349acc25d2cc9087a81d87ac54648d93f9\"" Oct 30 04:10:59.370942 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.370937 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-22fcec6f0f615271241a8359c17050656955dbc14c2774ac90939cf3f839f77f\"" Oct 30 04:10:59.416420 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.416335 5882 generic.go:356] "Generic (PLEG): container finished" podID="f526accb-9fcb-4480-bc5b-8ae927d2ed7d" containerID="eaefef74345dfc1371ebdfa3002812c0455558e23de6948e7ddfec166a7e5b8d" exitCode=0 Oct 30 04:10:59.416420 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.416393 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:10:59.416420 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.416404 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"f526accb-9fcb-4480-bc5b-8ae927d2ed7d","Type":"ContainerDied","Data":"eaefef74345dfc1371ebdfa3002812c0455558e23de6948e7ddfec166a7e5b8d"} Oct 30 04:10:59.416673 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.416465 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"f526accb-9fcb-4480-bc5b-8ae927d2ed7d","Type":"ContainerDied","Data":"e8b3b5e1d0f771130e59ca2e530dde8eb3e62da41b339b45429ef26f2b0a431b"} Oct 30 04:10:59.416673 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.416485 5882 scope.go:122] "RemoveContainer" containerID="eaefef74345dfc1371ebdfa3002812c0455558e23de6948e7ddfec166a7e5b8d" Oct 30 04:10:59.425947 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.425927 5882 scope.go:122] "RemoveContainer" containerID="eaefef74345dfc1371ebdfa3002812c0455558e23de6948e7ddfec166a7e5b8d" Oct 30 04:10:59.426213 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:10:59.426194 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaefef74345dfc1371ebdfa3002812c0455558e23de6948e7ddfec166a7e5b8d\": container with ID starting with eaefef74345dfc1371ebdfa3002812c0455558e23de6948e7ddfec166a7e5b8d not found: ID does not exist" containerID="eaefef74345dfc1371ebdfa3002812c0455558e23de6948e7ddfec166a7e5b8d" Oct 30 04:10:59.426271 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.426218 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaefef74345dfc1371ebdfa3002812c0455558e23de6948e7ddfec166a7e5b8d"} err="failed to get container status \"eaefef74345dfc1371ebdfa3002812c0455558e23de6948e7ddfec166a7e5b8d\": rpc error: code = NotFound desc = could not find container \"eaefef74345dfc1371ebdfa3002812c0455558e23de6948e7ddfec166a7e5b8d\": container with ID starting with eaefef74345dfc1371ebdfa3002812c0455558e23de6948e7ddfec166a7e5b8d not found: ID does not exist" Oct 30 04:10:59.473061 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.473031 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 04:10:59.486979 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.486950 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 04:10:59.517517 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.517488 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 04:10:59.517904 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.517890 5882 cpu_manager.go:414] "containerMap: removing container" podUID="0b6a3bc2-747b-4d2e-af39-5955e57ab37f" containerName="loki-querier" Oct 30 04:10:59.517965 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.517905 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6a3bc2-747b-4d2e-af39-5955e57ab37f" containerName="loki-querier" Oct 30 04:10:59.517965 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.517933 5882 cpu_manager.go:414] "containerMap: removing container" podUID="f526accb-9fcb-4480-bc5b-8ae927d2ed7d" containerName="loki-ingester" Oct 30 04:10:59.517965 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.517938 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f526accb-9fcb-4480-bc5b-8ae927d2ed7d" containerName="loki-ingester" Oct 30 04:10:59.518059 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.517999 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f526accb-9fcb-4480-bc5b-8ae927d2ed7d" containerName="loki-ingester" Oct 30 04:10:59.518059 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.518010 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6a3bc2-747b-4d2e-af39-5955e57ab37f" containerName="loki-querier" Oct 30 04:10:59.522765 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.522743 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:10:59.541636 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.541591 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-grpc\"" Oct 30 04:10:59.541820 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.541661 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-http\"" Oct 30 04:10:59.543173 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.543143 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 04:10:59.674050 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.673966 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:10:59.674207 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.674089 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxjbs\" (UniqueName: \"kubernetes.io/projected/c5e00d57-8d8a-4050-832c-0ead88622be6-kube-api-access-rxjbs\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:10:59.674207 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.674115 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:10:59.674280 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.674206 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:10:59.674280 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.674230 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:10:59.674280 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.674247 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5e00d57-8d8a-4050-832c-0ead88622be6-config\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:10:59.775216 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.775171 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:10:59.775418 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.775235 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:10:59.775418 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.775265 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5e00d57-8d8a-4050-832c-0ead88622be6-config\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:10:59.775418 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.775331 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:10:59.775418 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.775418 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxjbs\" (UniqueName: \"kubernetes.io/projected/c5e00d57-8d8a-4050-832c-0ead88622be6-kube-api-access-rxjbs\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:10:59.775661 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.775447 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:10:59.776010 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.775945 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:10:59.776142 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.776068 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5e00d57-8d8a-4050-832c-0ead88622be6-config\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:10:59.778387 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.778364 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:10:59.778525 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.778500 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:10:59.778733 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.778713 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:10:59.800952 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:10:59.800923 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxjbs\" (UniqueName: \"kubernetes.io/projected/c5e00d57-8d8a-4050-832c-0ead88622be6-kube-api-access-rxjbs\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:11:00.010172 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:00.010137 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f526accb-9fcb-4480-bc5b-8ae927d2ed7d" path="/var/lib/kubelet/pods/f526accb-9fcb-4480-bc5b-8ae927d2ed7d/volumes" Oct 30 04:11:00.986383 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:00.986337 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:11:00.986383 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:00.986388 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:11:01.033541 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:01.033508 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:11:01.033937 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:01.033554 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:11:01.087741 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:01.087699 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:11:01.087741 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:01.087748 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:11:01.088006 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:01.087833 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:11:01.088006 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:01.087874 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:11:01.104441 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:01.104414 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") DevicePath \"csi-40758f0eeb2ac4c8710af85cac17ce349acc25d2cc9087a81d87ac54648d93f9\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:11:01.104556 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:01.104414 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") DevicePath \"csi-22fcec6f0f615271241a8359c17050656955dbc14c2774ac90939cf3f839f77f\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:11:01.206934 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:01.206896 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/fcf6f0536bbc97a5651ec9a8d066f70948542721519078012a300fe6ca3edfd0/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:11:01.218548 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:01.218514 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/be8d3d2716806c45d69e88ad453cb51178b1b3855120bf6702b7eff86c093a7e/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:11:01.240131 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:01.240071 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:11:01.244479 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:01.244460 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:11:01.334626 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:01.334585 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:11:01.639106 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:01.639065 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 04:11:01.639503 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:11:01.639472 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e00d57_8d8a_4050_832c_0ead88622be6.slice/crio-c2df32a887230f7305f6b1fd3920cf68e1bcc867a5ddf713726f143638aa1fda WatchSource:0}: Error finding container c2df32a887230f7305f6b1fd3920cf68e1bcc867a5ddf713726f143638aa1fda: Status 404 returned error can't find the container with id c2df32a887230f7305f6b1fd3920cf68e1bcc867a5ddf713726f143638aa1fda Oct 30 04:11:02.433142 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:02.433107 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"c5e00d57-8d8a-4050-832c-0ead88622be6","Type":"ContainerStarted","Data":"6184ec6b84508ce16e94e03a6444c29510cd6a1bc2ba24f77c16bf016dfce47c"} Oct 30 04:11:02.433538 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:02.433165 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"c5e00d57-8d8a-4050-832c-0ead88622be6","Type":"ContainerStarted","Data":"c2df32a887230f7305f6b1fd3920cf68e1bcc867a5ddf713726f143638aa1fda"} Oct 30 04:11:02.433538 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:02.433273 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:11:02.485626 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:02.485577 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podStartSLOduration=3.485560558 podStartE2EDuration="3.485560558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:11:02.479745715 +0000 UTC m=+5374.594060700" watchObservedRunningTime="2026-10-30 04:11:02.485560558 +0000 UTC m=+5374.599875534" Oct 30 04:11:23.443269 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:23.443223 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="c5e00d57-8d8a-4050-832c-0ead88622be6" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:11:23.942237 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:11:23.942190 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:11:28.558598 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:28.558473 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 04:11:28.572098 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:28.572074 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 04:11:33.442558 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:33.442515 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="c5e00d57-8d8a-4050-832c-0ead88622be6" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:11:43.442650 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:11:43.442565 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:11:53.942646 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:11:53.942600 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:12:23.942381 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:12:23.942342 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:12:53.943207 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:12:53.943168 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:12:57.574676 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.574636 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 04:12:57.575232 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.574879 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="c5e00d57-8d8a-4050-832c-0ead88622be6" containerName="loki-ingester" containerID="cri-o://6184ec6b84508ce16e94e03a6444c29510cd6a1bc2ba24f77c16bf016dfce47c" gracePeriod=600 Oct 30 04:12:57.593800 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.593761 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-1"] Oct 30 04:12:57.593978 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.593959 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" podUID="7dc28de7-7003-4eb8-a215-ebc18c242eee" containerName="loki-index-gateway" containerID="cri-o://0d41dffb6b2a56bec89032258ec53dc56dc2a89bd7d537f95791a9549814a11d" gracePeriod=30 Oct 30 04:12:57.634286 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.634253 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt"] Oct 30 04:12:57.634543 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.634514 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" podUID="32a46854-54c3-4e02-9351-02b2667b60ff" containerName="loki-distributor" containerID="cri-o://086f5e217c7e8dd1ab4ed65b0f01f1b6028ea0686818a0dc1affd1b2a732438a" gracePeriod=30 Oct 30 04:12:57.646254 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.646224 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb"] Oct 30 04:12:57.646432 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.646412 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" podUID="ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e" containerName="loki-querier" containerID="cri-o://9318038aff90fb922740199622517243196d094fd2c8d878b213db8e46e6eda3" gracePeriod=30 Oct 30 04:12:57.652873 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.652845 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4"] Oct 30 04:12:57.653033 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.653013 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" podUID="9e43ef35-ea6c-4c3e-9099-34273546b4f4" containerName="loki-query-frontend" containerID="cri-o://7e65e96b04357f9b5a446c0f5b4d60da08bf3933a013c4b32ce56b3c38b52c63" gracePeriod=30 Oct 30 04:12:57.933064 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.933032 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:12:57.952702 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.952675 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:12:57.959425 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.959397 5882 generic.go:356] "Generic (PLEG): container finished" podID="9e43ef35-ea6c-4c3e-9099-34273546b4f4" containerID="7e65e96b04357f9b5a446c0f5b4d60da08bf3933a013c4b32ce56b3c38b52c63" exitCode=0 Oct 30 04:12:57.959550 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.959439 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" Oct 30 04:12:57.959550 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.959472 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" event={"ID":"9e43ef35-ea6c-4c3e-9099-34273546b4f4","Type":"ContainerDied","Data":"7e65e96b04357f9b5a446c0f5b4d60da08bf3933a013c4b32ce56b3c38b52c63"} Oct 30 04:12:57.959550 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.959525 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4" event={"ID":"9e43ef35-ea6c-4c3e-9099-34273546b4f4","Type":"ContainerDied","Data":"afa4369a13a8c80938d1915db1f9e52cbadb957dd8fdadf5b7f63972e5e31a36"} Oct 30 04:12:57.959550 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.959546 5882 scope.go:122] "RemoveContainer" containerID="7e65e96b04357f9b5a446c0f5b4d60da08bf3933a013c4b32ce56b3c38b52c63" Oct 30 04:12:57.960938 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.960920 5882 generic.go:356] "Generic (PLEG): container finished" podID="7dc28de7-7003-4eb8-a215-ebc18c242eee" containerID="0d41dffb6b2a56bec89032258ec53dc56dc2a89bd7d537f95791a9549814a11d" exitCode=0 Oct 30 04:12:57.961025 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.960949 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" event={"ID":"7dc28de7-7003-4eb8-a215-ebc18c242eee","Type":"ContainerDied","Data":"0d41dffb6b2a56bec89032258ec53dc56dc2a89bd7d537f95791a9549814a11d"} Oct 30 04:12:57.961025 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.960962 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:12:57.961025 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:57.960974 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" event={"ID":"7dc28de7-7003-4eb8-a215-ebc18c242eee","Type":"ContainerDied","Data":"a334a5684b7783273e135430e8ae876c18ffe822bffcffacb302bfc50053794e"} Oct 30 04:12:58.058884 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.058752 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-index-gateway-grpc\") pod \"7dc28de7-7003-4eb8-a215-ebc18c242eee\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " Oct 30 04:12:58.058884 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.058809 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-query-frontend-http\") pod \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\" (UID: \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\") " Oct 30 04:12:58.058884 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.058843 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-ca-bundle\") pod \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\" (UID: \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\") " Oct 30 04:12:58.059074 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.058911 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-index-gateway-http\") pod \"7dc28de7-7003-4eb8-a215-ebc18c242eee\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " Oct 30 04:12:58.059074 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.058946 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e43ef35-ea6c-4c3e-9099-34273546b4f4-config\" (UniqueName: \"kubernetes.io/configmap/9e43ef35-ea6c-4c3e-9099-34273546b4f4-config\") pod \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\" (UID: \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\") " Oct 30 04:12:58.059074 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.058973 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-test-loki-lifecycle-secret\") pod \"7dc28de7-7003-4eb8-a215-ebc18c242eee\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " Oct 30 04:12:58.059074 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.059024 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-ca-bundle\") pod \"7dc28de7-7003-4eb8-a215-ebc18c242eee\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " Oct 30 04:12:58.059074 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.059068 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc28de7-7003-4eb8-a215-ebc18c242eee-config\" (UniqueName: \"kubernetes.io/configmap/7dc28de7-7003-4eb8-a215-ebc18c242eee-config\") pod \"7dc28de7-7003-4eb8-a215-ebc18c242eee\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " Oct 30 04:12:58.059319 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.059098 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e43ef35-ea6c-4c3e-9099-34273546b4f4-kube-api-access-dgqxl\" (UniqueName: \"kubernetes.io/projected/9e43ef35-ea6c-4c3e-9099-34273546b4f4-kube-api-access-dgqxl\") pod \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\" (UID: \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\") " Oct 30 04:12:58.059319 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.059227 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"7dc28de7-7003-4eb8-a215-ebc18c242eee\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " Oct 30 04:12:58.059319 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.059268 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc28de7-7003-4eb8-a215-ebc18c242eee-kube-api-access-kkktg\" (UniqueName: \"kubernetes.io/projected/7dc28de7-7003-4eb8-a215-ebc18c242eee-kube-api-access-kkktg\") pod \"7dc28de7-7003-4eb8-a215-ebc18c242eee\" (UID: \"7dc28de7-7003-4eb8-a215-ebc18c242eee\") " Oct 30 04:12:58.059319 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.059302 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-query-frontend-grpc\") pod \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\" (UID: \"9e43ef35-ea6c-4c3e-9099-34273546b4f4\") " Oct 30 04:12:58.060465 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.060227 5882 scope.go:122] "RemoveContainer" containerID="7e65e96b04357f9b5a446c0f5b4d60da08bf3933a013c4b32ce56b3c38b52c63" Oct 30 04:12:58.060572 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.060490 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-ca-bundle" pod "9e43ef35-ea6c-4c3e-9099-34273546b4f4" (UID: "9e43ef35-ea6c-4c3e-9099-34273546b4f4"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:12:58.060883 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.060852 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-ca-bundle" pod "7dc28de7-7003-4eb8-a215-ebc18c242eee" (UID: "7dc28de7-7003-4eb8-a215-ebc18c242eee"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:12:58.061176 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.061026 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc28de7-7003-4eb8-a215-ebc18c242eee-config" pod "7dc28de7-7003-4eb8-a215-ebc18c242eee" (UID: "7dc28de7-7003-4eb8-a215-ebc18c242eee"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:12:58.061463 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.061118 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e43ef35-ea6c-4c3e-9099-34273546b4f4-config" pod "9e43ef35-ea6c-4c3e-9099-34273546b4f4" (UID: "9e43ef35-ea6c-4c3e-9099-34273546b4f4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:12:58.062298 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:12:58.062138 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e65e96b04357f9b5a446c0f5b4d60da08bf3933a013c4b32ce56b3c38b52c63\": container with ID starting with 7e65e96b04357f9b5a446c0f5b4d60da08bf3933a013c4b32ce56b3c38b52c63 not found: ID does not exist" containerID="7e65e96b04357f9b5a446c0f5b4d60da08bf3933a013c4b32ce56b3c38b52c63" Oct 30 04:12:58.062298 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.062176 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e65e96b04357f9b5a446c0f5b4d60da08bf3933a013c4b32ce56b3c38b52c63"} err="failed to get container status \"7e65e96b04357f9b5a446c0f5b4d60da08bf3933a013c4b32ce56b3c38b52c63\": rpc error: code = NotFound desc = could not find container \"7e65e96b04357f9b5a446c0f5b4d60da08bf3933a013c4b32ce56b3c38b52c63\": container with ID starting with 7e65e96b04357f9b5a446c0f5b4d60da08bf3933a013c4b32ce56b3c38b52c63 not found: ID does not exist" Oct 30 04:12:58.062298 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.062199 5882 scope.go:122] "RemoveContainer" containerID="0d41dffb6b2a56bec89032258ec53dc56dc2a89bd7d537f95791a9549814a11d" Oct 30 04:12:58.063992 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.063933 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-query-frontend-grpc" pod "9e43ef35-ea6c-4c3e-9099-34273546b4f4" (UID: "9e43ef35-ea6c-4c3e-9099-34273546b4f4"). InnerVolumeSpecName "data-science-lokistack-query-frontend-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:58.064414 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.064389 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-test-loki-lifecycle-secret" pod "7dc28de7-7003-4eb8-a215-ebc18c242eee" (UID: "7dc28de7-7003-4eb8-a215-ebc18c242eee"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:58.064522 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.064410 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-index-gateway-grpc" pod "7dc28de7-7003-4eb8-a215-ebc18c242eee" (UID: "7dc28de7-7003-4eb8-a215-ebc18c242eee"). InnerVolumeSpecName "data-science-lokistack-index-gateway-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:58.064522 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.064484 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc28de7-7003-4eb8-a215-ebc18c242eee-kube-api-access-kkktg" pod "7dc28de7-7003-4eb8-a215-ebc18c242eee" (UID: "7dc28de7-7003-4eb8-a215-ebc18c242eee"). InnerVolumeSpecName "kube-api-access-kkktg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:12:58.064668 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.064579 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-query-frontend-http" pod "9e43ef35-ea6c-4c3e-9099-34273546b4f4" (UID: "9e43ef35-ea6c-4c3e-9099-34273546b4f4"). InnerVolumeSpecName "data-science-lokistack-query-frontend-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:58.064668 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.064659 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-index-gateway-http" pod "7dc28de7-7003-4eb8-a215-ebc18c242eee" (UID: "7dc28de7-7003-4eb8-a215-ebc18c242eee"). InnerVolumeSpecName "data-science-lokistack-index-gateway-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:58.064872 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.064800 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00" pod "7dc28de7-7003-4eb8-a215-ebc18c242eee" (UID: "7dc28de7-7003-4eb8-a215-ebc18c242eee"). InnerVolumeSpecName "pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:12:58.064872 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.064804 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e43ef35-ea6c-4c3e-9099-34273546b4f4-kube-api-access-dgqxl" pod "9e43ef35-ea6c-4c3e-9099-34273546b4f4" (UID: "9e43ef35-ea6c-4c3e-9099-34273546b4f4"). InnerVolumeSpecName "kube-api-access-dgqxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:12:58.071880 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.071860 5882 scope.go:122] "RemoveContainer" containerID="0d41dffb6b2a56bec89032258ec53dc56dc2a89bd7d537f95791a9549814a11d" Oct 30 04:12:58.072154 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:12:58.072131 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d41dffb6b2a56bec89032258ec53dc56dc2a89bd7d537f95791a9549814a11d\": container with ID starting with 0d41dffb6b2a56bec89032258ec53dc56dc2a89bd7d537f95791a9549814a11d not found: ID does not exist" containerID="0d41dffb6b2a56bec89032258ec53dc56dc2a89bd7d537f95791a9549814a11d" Oct 30 04:12:58.072217 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.072165 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d41dffb6b2a56bec89032258ec53dc56dc2a89bd7d537f95791a9549814a11d"} err="failed to get container status \"0d41dffb6b2a56bec89032258ec53dc56dc2a89bd7d537f95791a9549814a11d\": rpc error: code = NotFound desc = could not find container \"0d41dffb6b2a56bec89032258ec53dc56dc2a89bd7d537f95791a9549814a11d\": container with ID starting with 0d41dffb6b2a56bec89032258ec53dc56dc2a89bd7d537f95791a9549814a11d not found: ID does not exist" Oct 30 04:12:58.161318 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.160836 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:12:58.161318 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.161051 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkktg\" (UniqueName: \"kubernetes.io/projected/7dc28de7-7003-4eb8-a215-ebc18c242eee-kube-api-access-kkktg\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.161318 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.161133 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-query-frontend-grpc\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.161318 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.161160 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-index-gateway-grpc\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.161318 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.161177 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-query-frontend-http\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.161318 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.161195 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e43ef35-ea6c-4c3e-9099-34273546b4f4-data-science-lokistack-ca-bundle\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.161318 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.161212 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-index-gateway-http\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.161318 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.161229 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e43ef35-ea6c-4c3e-9099-34273546b4f4-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.161318 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.161246 5882 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/7dc28de7-7003-4eb8-a215-ebc18c242eee-test-loki-lifecycle-secret\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.161318 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.161261 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc28de7-7003-4eb8-a215-ebc18c242eee-data-science-lokistack-ca-bundle\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.161318 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.161277 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7dc28de7-7003-4eb8-a215-ebc18c242eee-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.161318 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.161290 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgqxl\" (UniqueName: \"kubernetes.io/projected/9e43ef35-ea6c-4c3e-9099-34273546b4f4-kube-api-access-dgqxl\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.178977 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.178954 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:12:58.261892 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.261846 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-51b362136bc4ecc02770f4e1d3595c1c59eb604f2b3d12558a9a3323d6a31529\"" Oct 30 04:12:58.292116 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.292093 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:12:58.319688 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.319652 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4"] Oct 30 04:12:58.324098 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.324063 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-29jh4"] Oct 30 04:12:58.409245 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.409174 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-1"] Oct 30 04:12:58.413671 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.413644 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-1"] Oct 30 04:12:58.433577 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.433557 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:12:58.463249 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.463221 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-ca-bundle\") pod \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " Oct 30 04:12:58.463363 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.463321 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-test-loki-lifecycle-secret\") pod \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " Oct 30 04:12:58.463432 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.463363 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-kube-api-access-phvd6\" (UniqueName: \"kubernetes.io/projected/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-kube-api-access-phvd6\") pod \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " Oct 30 04:12:58.463432 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.463395 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-querier-grpc\") pod \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " Oct 30 04:12:58.463602 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.463478 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-querier-http\") pod \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " Oct 30 04:12:58.463602 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.463512 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-config\" (UniqueName: \"kubernetes.io/configmap/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-config\") pod \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\" (UID: \"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e\") " Oct 30 04:12:58.463602 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.463554 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-ca-bundle" pod "ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e" (UID: "ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:12:58.463850 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.463830 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-ca-bundle\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.464067 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.464039 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-config" pod "ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e" (UID: "ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:12:58.465565 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.465539 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-kube-api-access-phvd6" pod "ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e" (UID: "ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e"). InnerVolumeSpecName "kube-api-access-phvd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:12:58.465565 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.465550 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-querier-http" pod "ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e" (UID: "ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e"). InnerVolumeSpecName "data-science-lokistack-querier-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:58.465960 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.465942 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-test-loki-lifecycle-secret" pod "ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e" (UID: "ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:58.466004 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.465959 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-querier-grpc" pod "ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e" (UID: "ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e"). InnerVolumeSpecName "data-science-lokistack-querier-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:58.564122 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.564092 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e00d57-8d8a-4050-832c-0ead88622be6-kube-api-access-rxjbs\" (UniqueName: \"kubernetes.io/projected/c5e00d57-8d8a-4050-832c-0ead88622be6-kube-api-access-rxjbs\") pod \"c5e00d57-8d8a-4050-832c-0ead88622be6\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " Oct 30 04:12:58.564122 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.564126 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-test-loki-lifecycle-secret\") pod \"c5e00d57-8d8a-4050-832c-0ead88622be6\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " Oct 30 04:12:58.564324 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.564155 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ca-bundle\") pod \"c5e00d57-8d8a-4050-832c-0ead88622be6\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " Oct 30 04:12:58.564324 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.564287 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"c5e00d57-8d8a-4050-832c-0ead88622be6\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " Oct 30 04:12:58.564402 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.564361 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ingester-grpc\") pod \"c5e00d57-8d8a-4050-832c-0ead88622be6\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " Oct 30 04:12:58.564453 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.564400 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ingester-http\") pod \"c5e00d57-8d8a-4050-832c-0ead88622be6\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " Oct 30 04:12:58.564453 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.564445 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5e00d57-8d8a-4050-832c-0ead88622be6-config\" (UniqueName: \"kubernetes.io/configmap/c5e00d57-8d8a-4050-832c-0ead88622be6-config\") pod \"c5e00d57-8d8a-4050-832c-0ead88622be6\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " Oct 30 04:12:58.564567 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.564524 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"c5e00d57-8d8a-4050-832c-0ead88622be6\" (UID: \"c5e00d57-8d8a-4050-832c-0ead88622be6\") " Oct 30 04:12:58.564675 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.564649 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ca-bundle" pod "c5e00d57-8d8a-4050-832c-0ead88622be6" (UID: "c5e00d57-8d8a-4050-832c-0ead88622be6"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:12:58.565306 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.564916 5882 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-test-loki-lifecycle-secret\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.565306 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.564938 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phvd6\" (UniqueName: \"kubernetes.io/projected/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-kube-api-access-phvd6\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.565306 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.564954 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-querier-grpc\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.565306 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.564969 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-data-science-lokistack-querier-http\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.565306 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.564984 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.565306 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.564911 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e00d57-8d8a-4050-832c-0ead88622be6-config" pod "c5e00d57-8d8a-4050-832c-0ead88622be6" (UID: "c5e00d57-8d8a-4050-832c-0ead88622be6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:12:58.565306 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.564996 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ca-bundle\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.566692 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.566666 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ingester-grpc" pod "c5e00d57-8d8a-4050-832c-0ead88622be6" (UID: "c5e00d57-8d8a-4050-832c-0ead88622be6"). InnerVolumeSpecName "data-science-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:58.566867 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.566839 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e00d57-8d8a-4050-832c-0ead88622be6-kube-api-access-rxjbs" pod "c5e00d57-8d8a-4050-832c-0ead88622be6" (UID: "c5e00d57-8d8a-4050-832c-0ead88622be6"). InnerVolumeSpecName "kube-api-access-rxjbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:12:58.566960 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.566926 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-test-loki-lifecycle-secret" pod "c5e00d57-8d8a-4050-832c-0ead88622be6" (UID: "c5e00d57-8d8a-4050-832c-0ead88622be6"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:58.567013 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.566976 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ingester-http" pod "c5e00d57-8d8a-4050-832c-0ead88622be6" (UID: "c5e00d57-8d8a-4050-832c-0ead88622be6"). InnerVolumeSpecName "data-science-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:58.567466 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.567449 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950" pod "c5e00d57-8d8a-4050-832c-0ead88622be6" (UID: "c5e00d57-8d8a-4050-832c-0ead88622be6"). InnerVolumeSpecName "pvc-199f9790-eb98-4bad-8977-a99a3b174eed". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:12:58.567768 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.567744 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735" pod "c5e00d57-8d8a-4050-832c-0ead88622be6" (UID: "c5e00d57-8d8a-4050-832c-0ead88622be6"). InnerVolumeSpecName "pvc-ea00ca9c-db2e-4885-bfa1-004970abec93". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:12:58.666279 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.666217 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxjbs\" (UniqueName: \"kubernetes.io/projected/c5e00d57-8d8a-4050-832c-0ead88622be6-kube-api-access-rxjbs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.666279 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.666243 5882 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-test-loki-lifecycle-secret\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.666279 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.666276 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:12:58.666827 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.666286 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ingester-grpc\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.666827 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.666295 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/c5e00d57-8d8a-4050-832c-0ead88622be6-data-science-lokistack-ingester-http\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.666827 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.666305 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5e00d57-8d8a-4050-832c-0ead88622be6-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.666827 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.666319 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:12:58.675889 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.675871 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-199f9790-eb98-4bad-8977-a99a3b174eed" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:12:58.678636 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.678602 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-ea00ca9c-db2e-4885-bfa1-004970abec93" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:12:58.767290 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.767251 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-22fcec6f0f615271241a8359c17050656955dbc14c2774ac90939cf3f839f77f\"" Oct 30 04:12:58.767290 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.767284 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-40758f0eeb2ac4c8710af85cac17ce349acc25d2cc9087a81d87ac54648d93f9\"" Oct 30 04:12:58.966707 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.966663 5882 generic.go:356] "Generic (PLEG): container finished" podID="c5e00d57-8d8a-4050-832c-0ead88622be6" containerID="6184ec6b84508ce16e94e03a6444c29510cd6a1bc2ba24f77c16bf016dfce47c" exitCode=0 Oct 30 04:12:58.966837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.966718 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" Oct 30 04:12:58.966837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.966727 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"c5e00d57-8d8a-4050-832c-0ead88622be6","Type":"ContainerDied","Data":"6184ec6b84508ce16e94e03a6444c29510cd6a1bc2ba24f77c16bf016dfce47c"} Oct 30 04:12:58.966837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.966806 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"c5e00d57-8d8a-4050-832c-0ead88622be6","Type":"ContainerDied","Data":"c2df32a887230f7305f6b1fd3920cf68e1bcc867a5ddf713726f143638aa1fda"} Oct 30 04:12:58.966837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.966829 5882 scope.go:122] "RemoveContainer" containerID="6184ec6b84508ce16e94e03a6444c29510cd6a1bc2ba24f77c16bf016dfce47c" Oct 30 04:12:58.968502 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.968483 5882 generic.go:356] "Generic (PLEG): container finished" podID="32a46854-54c3-4e02-9351-02b2667b60ff" containerID="086f5e217c7e8dd1ab4ed65b0f01f1b6028ea0686818a0dc1affd1b2a732438a" exitCode=0 Oct 30 04:12:58.968629 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.968558 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" event={"ID":"32a46854-54c3-4e02-9351-02b2667b60ff","Type":"ContainerDied","Data":"086f5e217c7e8dd1ab4ed65b0f01f1b6028ea0686818a0dc1affd1b2a732438a"} Oct 30 04:12:58.972096 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.972074 5882 generic.go:356] "Generic (PLEG): container finished" podID="ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e" containerID="9318038aff90fb922740199622517243196d094fd2c8d878b213db8e46e6eda3" exitCode=0 Oct 30 04:12:58.972204 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.972105 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" Oct 30 04:12:58.972204 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.972123 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" event={"ID":"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e","Type":"ContainerDied","Data":"9318038aff90fb922740199622517243196d094fd2c8d878b213db8e46e6eda3"} Oct 30 04:12:58.972204 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.972153 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb" event={"ID":"ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e","Type":"ContainerDied","Data":"666215ff701d40e4a888a181d108d72dcee3496537cc909fb113006329a77693"} Oct 30 04:12:58.984141 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.984120 5882 scope.go:122] "RemoveContainer" containerID="6184ec6b84508ce16e94e03a6444c29510cd6a1bc2ba24f77c16bf016dfce47c" Oct 30 04:12:58.984420 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:12:58.984403 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6184ec6b84508ce16e94e03a6444c29510cd6a1bc2ba24f77c16bf016dfce47c\": container with ID starting with 6184ec6b84508ce16e94e03a6444c29510cd6a1bc2ba24f77c16bf016dfce47c not found: ID does not exist" containerID="6184ec6b84508ce16e94e03a6444c29510cd6a1bc2ba24f77c16bf016dfce47c" Oct 30 04:12:58.984468 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.984427 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6184ec6b84508ce16e94e03a6444c29510cd6a1bc2ba24f77c16bf016dfce47c"} err="failed to get container status \"6184ec6b84508ce16e94e03a6444c29510cd6a1bc2ba24f77c16bf016dfce47c\": rpc error: code = NotFound desc = could not find container \"6184ec6b84508ce16e94e03a6444c29510cd6a1bc2ba24f77c16bf016dfce47c\": container with ID starting with 6184ec6b84508ce16e94e03a6444c29510cd6a1bc2ba24f77c16bf016dfce47c not found: ID does not exist" Oct 30 04:12:58.984468 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.984443 5882 scope.go:122] "RemoveContainer" containerID="9318038aff90fb922740199622517243196d094fd2c8d878b213db8e46e6eda3" Oct 30 04:12:59.000753 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.993718 5882 scope.go:122] "RemoveContainer" containerID="9318038aff90fb922740199622517243196d094fd2c8d878b213db8e46e6eda3" Oct 30 04:12:59.000753 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:12:58.994004 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9318038aff90fb922740199622517243196d094fd2c8d878b213db8e46e6eda3\": container with ID starting with 9318038aff90fb922740199622517243196d094fd2c8d878b213db8e46e6eda3 not found: ID does not exist" containerID="9318038aff90fb922740199622517243196d094fd2c8d878b213db8e46e6eda3" Oct 30 04:12:59.000753 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:58.994031 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9318038aff90fb922740199622517243196d094fd2c8d878b213db8e46e6eda3"} err="failed to get container status \"9318038aff90fb922740199622517243196d094fd2c8d878b213db8e46e6eda3\": rpc error: code = NotFound desc = could not find container \"9318038aff90fb922740199622517243196d094fd2c8d878b213db8e46e6eda3\": container with ID starting with 9318038aff90fb922740199622517243196d094fd2c8d878b213db8e46e6eda3 not found: ID does not exist" Oct 30 04:12:59.025096 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.025068 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 04:12:59.028883 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.028858 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"] Oct 30 04:12:59.066204 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.066174 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb"] Oct 30 04:12:59.071956 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.071931 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-s4ctb"] Oct 30 04:12:59.091712 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.091690 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:12:59.271194 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.271164 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-ca-bundle\") pod \"32a46854-54c3-4e02-9351-02b2667b60ff\" (UID: \"32a46854-54c3-4e02-9351-02b2667b60ff\") " Oct 30 04:12:59.271335 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.271233 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-distributor-http\") pod \"32a46854-54c3-4e02-9351-02b2667b60ff\" (UID: \"32a46854-54c3-4e02-9351-02b2667b60ff\") " Oct 30 04:12:59.271335 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.271292 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a46854-54c3-4e02-9351-02b2667b60ff-kube-api-access-g44sx\" (UniqueName: \"kubernetes.io/projected/32a46854-54c3-4e02-9351-02b2667b60ff-kube-api-access-g44sx\") pod \"32a46854-54c3-4e02-9351-02b2667b60ff\" (UID: \"32a46854-54c3-4e02-9351-02b2667b60ff\") " Oct 30 04:12:59.271400 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.271345 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-distributor-grpc\") pod \"32a46854-54c3-4e02-9351-02b2667b60ff\" (UID: \"32a46854-54c3-4e02-9351-02b2667b60ff\") " Oct 30 04:12:59.271400 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.271384 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32a46854-54c3-4e02-9351-02b2667b60ff-config\" (UniqueName: \"kubernetes.io/configmap/32a46854-54c3-4e02-9351-02b2667b60ff-config\") pod \"32a46854-54c3-4e02-9351-02b2667b60ff\" (UID: \"32a46854-54c3-4e02-9351-02b2667b60ff\") " Oct 30 04:12:59.271745 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.271610 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-ca-bundle" pod "32a46854-54c3-4e02-9351-02b2667b60ff" (UID: "32a46854-54c3-4e02-9351-02b2667b60ff"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:12:59.271844 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.271819 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a46854-54c3-4e02-9351-02b2667b60ff-config" pod "32a46854-54c3-4e02-9351-02b2667b60ff" (UID: "32a46854-54c3-4e02-9351-02b2667b60ff"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:12:59.273230 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.273211 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a46854-54c3-4e02-9351-02b2667b60ff-kube-api-access-g44sx" pod "32a46854-54c3-4e02-9351-02b2667b60ff" (UID: "32a46854-54c3-4e02-9351-02b2667b60ff"). InnerVolumeSpecName "kube-api-access-g44sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:12:59.273732 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.273711 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-distributor-http" pod "32a46854-54c3-4e02-9351-02b2667b60ff" (UID: "32a46854-54c3-4e02-9351-02b2667b60ff"). InnerVolumeSpecName "data-science-lokistack-distributor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:59.273794 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.273724 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-distributor-grpc" pod "32a46854-54c3-4e02-9351-02b2667b60ff" (UID: "32a46854-54c3-4e02-9351-02b2667b60ff"). InnerVolumeSpecName "data-science-lokistack-distributor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:59.372235 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.372206 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g44sx\" (UniqueName: \"kubernetes.io/projected/32a46854-54c3-4e02-9351-02b2667b60ff-kube-api-access-g44sx\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:59.372235 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.372233 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-distributor-grpc\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:59.372235 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.372244 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/32a46854-54c3-4e02-9351-02b2667b60ff-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:59.372453 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.372253 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-ca-bundle\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:59.372453 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.372262 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/32a46854-54c3-4e02-9351-02b2667b60ff-data-science-lokistack-distributor-http\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:12:59.980341 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.980258 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" event={"ID":"32a46854-54c3-4e02-9351-02b2667b60ff","Type":"ContainerDied","Data":"7095535b7c2fb0382283ad431ea5c9c01750295ff0cdd0d6d08c3e1c82b9fd0e"} Oct 30 04:12:59.980341 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.980281 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt" Oct 30 04:12:59.980341 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:12:59.980314 5882 scope.go:122] "RemoveContainer" containerID="086f5e217c7e8dd1ab4ed65b0f01f1b6028ea0686818a0dc1affd1b2a732438a" Oct 30 04:13:00.009188 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:00.009155 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dc28de7-7003-4eb8-a215-ebc18c242eee" path="/var/lib/kubelet/pods/7dc28de7-7003-4eb8-a215-ebc18c242eee/volumes" Oct 30 04:13:00.009632 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:00.009593 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e43ef35-ea6c-4c3e-9099-34273546b4f4" path="/var/lib/kubelet/pods/9e43ef35-ea6c-4c3e-9099-34273546b4f4/volumes" Oct 30 04:13:00.010050 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:00.010033 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e00d57-8d8a-4050-832c-0ead88622be6" path="/var/lib/kubelet/pods/c5e00d57-8d8a-4050-832c-0ead88622be6/volumes" Oct 30 04:13:00.010755 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:00.010735 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e" path="/var/lib/kubelet/pods/ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e/volumes" Oct 30 04:13:00.035777 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:00.035744 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt"] Oct 30 04:13:00.049544 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:00.049522 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-pk2nt"] Oct 30 04:13:02.009281 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:02.009244 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32a46854-54c3-4e02-9351-02b2667b60ff" path="/var/lib/kubelet/pods/32a46854-54c3-4e02-9351-02b2667b60ff/volumes" Oct 30 04:13:03.228492 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.228459 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8"] Oct 30 04:13:03.229006 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.228831 5882 cpu_manager.go:414] "containerMap: removing container" podUID="7dc28de7-7003-4eb8-a215-ebc18c242eee" containerName="loki-index-gateway" Oct 30 04:13:03.229006 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.228843 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc28de7-7003-4eb8-a215-ebc18c242eee" containerName="loki-index-gateway" Oct 30 04:13:03.229006 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.228851 5882 cpu_manager.go:414] "containerMap: removing container" podUID="ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e" containerName="loki-querier" Oct 30 04:13:03.229006 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.228856 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e" containerName="loki-querier" Oct 30 04:13:03.229006 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.228869 5882 cpu_manager.go:414] "containerMap: removing container" podUID="32a46854-54c3-4e02-9351-02b2667b60ff" containerName="loki-distributor" Oct 30 04:13:03.229006 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.228875 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a46854-54c3-4e02-9351-02b2667b60ff" containerName="loki-distributor" Oct 30 04:13:03.229006 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.228889 5882 cpu_manager.go:414] "containerMap: removing container" podUID="c5e00d57-8d8a-4050-832c-0ead88622be6" containerName="loki-ingester" Oct 30 04:13:03.229006 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.228894 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e00d57-8d8a-4050-832c-0ead88622be6" containerName="loki-ingester" Oct 30 04:13:03.229006 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.228910 5882 cpu_manager.go:414] "containerMap: removing container" podUID="9e43ef35-ea6c-4c3e-9099-34273546b4f4" containerName="loki-query-frontend" Oct 30 04:13:03.229006 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.228914 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e43ef35-ea6c-4c3e-9099-34273546b4f4" containerName="loki-query-frontend" Oct 30 04:13:03.229006 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.228963 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e00d57-8d8a-4050-832c-0ead88622be6" containerName="loki-ingester" Oct 30 04:13:03.229006 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.228971 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e43ef35-ea6c-4c3e-9099-34273546b4f4" containerName="loki-query-frontend" Oct 30 04:13:03.229006 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.228979 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc28de7-7003-4eb8-a215-ebc18c242eee" containerName="loki-index-gateway" Oct 30 04:13:03.229006 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.228984 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffdc3a67-2671-40c6-a23a-7b4bd7d32b3e" containerName="loki-querier" Oct 30 04:13:03.229006 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.228990 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a46854-54c3-4e02-9351-02b2667b60ff" containerName="loki-distributor" Oct 30 04:13:03.233829 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.233804 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:13:03.253488 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.253453 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-distributor-grpc\"" Oct 30 04:13:03.253488 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.253469 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ca-bundle\"" Oct 30 04:13:03.254477 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.254447 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8"] Oct 30 04:13:03.256258 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.256234 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-distributor-http\"" Oct 30 04:13:03.256358 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.256296 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-dockercfg-vft9z\"" Oct 30 04:13:03.256982 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.256963 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-config\"" Oct 30 04:13:03.306275 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.306248 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-mjcm8\" (UID: \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:13:03.306431 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.306306 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnc9j\" (UniqueName: \"kubernetes.io/projected/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-kube-api-access-wnc9j\") pod \"data-science-lokistack-distributor-58947679b7-mjcm8\" (UID: \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:13:03.306431 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.306404 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-mjcm8\" (UID: \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:13:03.306524 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.306457 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-mjcm8\" (UID: \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:13:03.306524 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.306488 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-config\") pod \"data-science-lokistack-distributor-58947679b7-mjcm8\" (UID: \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:13:03.407560 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.407525 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-mjcm8\" (UID: \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:13:03.407750 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.407588 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnc9j\" (UniqueName: \"kubernetes.io/projected/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-kube-api-access-wnc9j\") pod \"data-science-lokistack-distributor-58947679b7-mjcm8\" (UID: \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:13:03.407750 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.407637 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-mjcm8\" (UID: \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:13:03.407750 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.407685 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-mjcm8\" (UID: \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:13:03.407750 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.407718 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-config\") pod \"data-science-lokistack-distributor-58947679b7-mjcm8\" (UID: \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:13:03.408507 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.408475 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-mjcm8\" (UID: \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:13:03.408653 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.408549 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-config\") pod \"data-science-lokistack-distributor-58947679b7-mjcm8\" (UID: \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:13:03.410243 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.410218 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-mjcm8\" (UID: \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:13:03.410340 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.410267 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-mjcm8\" (UID: \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:13:03.413137 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.413113 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9"] Oct 30 04:13:03.417278 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.417258 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:03.436167 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.436142 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnc9j\" (UniqueName: \"kubernetes.io/projected/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-kube-api-access-wnc9j\") pod \"data-science-lokistack-distributor-58947679b7-mjcm8\" (UID: \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:13:03.441726 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.441700 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9"] Oct 30 04:13:03.453451 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.453430 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-querier-http\"" Oct 30 04:13:03.453577 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.453564 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-lifecycle-secret\"" Oct 30 04:13:03.453630 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.453569 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-querier-grpc\"" Oct 30 04:13:03.508084 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.508057 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-4w2d9\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:03.508262 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.508091 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4ffd19f-279b-4393-94fc-2fd82bc53f69-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-4w2d9\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:03.508262 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.508111 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-4w2d9\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:03.508262 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.508247 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plsvq\" (UniqueName: \"kubernetes.io/projected/e4ffd19f-279b-4393-94fc-2fd82bc53f69-kube-api-access-plsvq\") pod \"data-science-lokistack-querier-56ffbd5fdc-4w2d9\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:03.508405 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.508280 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-4w2d9\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:03.508405 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.508298 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-4w2d9\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:03.546223 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.546197 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:13:03.609317 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.609269 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-4w2d9\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:03.609463 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.609367 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plsvq\" (UniqueName: \"kubernetes.io/projected/e4ffd19f-279b-4393-94fc-2fd82bc53f69-kube-api-access-plsvq\") pod \"data-science-lokistack-querier-56ffbd5fdc-4w2d9\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:03.609463 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.609409 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-4w2d9\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:03.609463 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.609440 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-4w2d9\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:03.609652 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.609519 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-4w2d9\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:03.609652 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.609540 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4ffd19f-279b-4393-94fc-2fd82bc53f69-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-4w2d9\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:03.612535 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.610414 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4ffd19f-279b-4393-94fc-2fd82bc53f69-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-4w2d9\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:03.614103 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.613972 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-4w2d9\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:03.614103 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.614013 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-4w2d9\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:03.616456 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.616430 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d"] Oct 30 04:13:03.616707 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.616609 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-4w2d9\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:03.619509 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.619469 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-4w2d9\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:03.622191 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.621964 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:13:03.640657 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.640632 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plsvq\" (UniqueName: \"kubernetes.io/projected/e4ffd19f-279b-4393-94fc-2fd82bc53f69-kube-api-access-plsvq\") pod \"data-science-lokistack-querier-56ffbd5fdc-4w2d9\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:03.647078 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.647050 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d"] Oct 30 04:13:03.654884 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.654864 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-query-frontend-grpc\"" Oct 30 04:13:03.654976 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.654939 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-query-frontend-http\"" Oct 30 04:13:03.710181 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.710153 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-f746fc66d-fpf2d\" (UID: \"95e2a939-2c53-428a-958a-b92aeb6782d2\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:13:03.710353 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.710188 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk77d\" (UniqueName: \"kubernetes.io/projected/95e2a939-2c53-428a-958a-b92aeb6782d2-kube-api-access-kk77d\") pod \"data-science-lokistack-query-frontend-f746fc66d-fpf2d\" (UID: \"95e2a939-2c53-428a-958a-b92aeb6782d2\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:13:03.710353 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.710294 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95e2a939-2c53-428a-958a-b92aeb6782d2-config\") pod \"data-science-lokistack-query-frontend-f746fc66d-fpf2d\" (UID: \"95e2a939-2c53-428a-958a-b92aeb6782d2\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:13:03.710353 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.710313 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-f746fc66d-fpf2d\" (UID: \"95e2a939-2c53-428a-958a-b92aeb6782d2\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:13:03.710353 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.710345 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-f746fc66d-fpf2d\" (UID: \"95e2a939-2c53-428a-958a-b92aeb6782d2\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:13:03.735026 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.734995 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:03.811198 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.811168 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95e2a939-2c53-428a-958a-b92aeb6782d2-config\") pod \"data-science-lokistack-query-frontend-f746fc66d-fpf2d\" (UID: \"95e2a939-2c53-428a-958a-b92aeb6782d2\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:13:03.811399 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.811203 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-f746fc66d-fpf2d\" (UID: \"95e2a939-2c53-428a-958a-b92aeb6782d2\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:13:03.811399 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.811247 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-f746fc66d-fpf2d\" (UID: \"95e2a939-2c53-428a-958a-b92aeb6782d2\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:13:03.811399 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.811314 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-f746fc66d-fpf2d\" (UID: \"95e2a939-2c53-428a-958a-b92aeb6782d2\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:13:03.811399 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.811339 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk77d\" (UniqueName: \"kubernetes.io/projected/95e2a939-2c53-428a-958a-b92aeb6782d2-kube-api-access-kk77d\") pod \"data-science-lokistack-query-frontend-f746fc66d-fpf2d\" (UID: \"95e2a939-2c53-428a-958a-b92aeb6782d2\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:13:03.812088 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.812058 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-f746fc66d-fpf2d\" (UID: \"95e2a939-2c53-428a-958a-b92aeb6782d2\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:13:03.812193 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.812116 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95e2a939-2c53-428a-958a-b92aeb6782d2-config\") pod \"data-science-lokistack-query-frontend-f746fc66d-fpf2d\" (UID: \"95e2a939-2c53-428a-958a-b92aeb6782d2\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:13:03.813752 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.813729 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-f746fc66d-fpf2d\" (UID: \"95e2a939-2c53-428a-958a-b92aeb6782d2\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:13:03.813853 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.813833 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-f746fc66d-fpf2d\" (UID: \"95e2a939-2c53-428a-958a-b92aeb6782d2\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:13:03.846028 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.846002 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk77d\" (UniqueName: \"kubernetes.io/projected/95e2a939-2c53-428a-958a-b92aeb6782d2-kube-api-access-kk77d\") pod \"data-science-lokistack-query-frontend-f746fc66d-fpf2d\" (UID: \"95e2a939-2c53-428a-958a-b92aeb6782d2\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:13:03.850293 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.850271 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8"] Oct 30 04:13:03.850992 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:13:03.850969 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a5bf67f_05b6_43f6_ad63_db2328fbfffe.slice/crio-23edcc3117c0af21483d619ebc369a99067a63663c124365cae3e9681247dda7 WatchSource:0}: Error finding container 23edcc3117c0af21483d619ebc369a99067a63663c124365cae3e9681247dda7: Status 404 returned error can't find the container with id 23edcc3117c0af21483d619ebc369a99067a63663c124365cae3e9681247dda7 Oct 30 04:13:03.934275 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:03.934102 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:13:04.017191 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:04.017065 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:13:04.017191 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:04.017125 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" event={"ID":"7a5bf67f-05b6-43f6-ad63-db2328fbfffe","Type":"ContainerStarted","Data":"0696d621e061924134f69cfc62ffdc9f074b6dc351591cf8f43709f1eda3f4b0"} Oct 30 04:13:04.017191 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:04.017155 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" event={"ID":"7a5bf67f-05b6-43f6-ad63-db2328fbfffe","Type":"ContainerStarted","Data":"23edcc3117c0af21483d619ebc369a99067a63663c124365cae3e9681247dda7"} Oct 30 04:13:04.043334 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:04.043301 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9"] Oct 30 04:13:04.043710 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:13:04.043677 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ffd19f_279b_4393_94fc_2fd82bc53f69.slice/crio-61ec69ae2a39ff46b1f6d818f6c619f532ae9b0ab6d148374ba353be709fc455 WatchSource:0}: Error finding container 61ec69ae2a39ff46b1f6d818f6c619f532ae9b0ab6d148374ba353be709fc455: Status 404 returned error can't find the container with id 61ec69ae2a39ff46b1f6d818f6c619f532ae9b0ab6d148374ba353be709fc455 Oct 30 04:13:04.072372 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:04.072324 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" podStartSLOduration=1.072310039 podStartE2EDuration="1.072310039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:04.071899739 +0000 UTC m=+5496.186214726" watchObservedRunningTime="2026-10-30 04:13:04.072310039 +0000 UTC m=+5496.186625014" Oct 30 04:13:04.237476 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:04.237442 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d"] Oct 30 04:13:04.238099 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:13:04.238068 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e2a939_2c53_428a_958a_b92aeb6782d2.slice/crio-7bf180bcfcdbdb8644c3ea0498e76901324b418dfbb7796569749427a4a82607 WatchSource:0}: Error finding container 7bf180bcfcdbdb8644c3ea0498e76901324b418dfbb7796569749427a4a82607: Status 404 returned error can't find the container with id 7bf180bcfcdbdb8644c3ea0498e76901324b418dfbb7796569749427a4a82607 Oct 30 04:13:05.022357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:05.022317 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" event={"ID":"95e2a939-2c53-428a-958a-b92aeb6782d2","Type":"ContainerStarted","Data":"eca9ed1070b2e23c6fe1a7805b112f44aff5ad3e297ee3ab453d03f7fe3380d7"} Oct 30 04:13:05.022563 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:05.022410 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" event={"ID":"95e2a939-2c53-428a-958a-b92aeb6782d2","Type":"ContainerStarted","Data":"7bf180bcfcdbdb8644c3ea0498e76901324b418dfbb7796569749427a4a82607"} Oct 30 04:13:05.022563 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:05.022443 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:13:05.023936 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:05.023907 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" event={"ID":"e4ffd19f-279b-4393-94fc-2fd82bc53f69","Type":"ContainerStarted","Data":"67af549ceb8b5f65feec86617e1ab73073958a49b0b8101aaa1d8a61070782d8"} Oct 30 04:13:05.024059 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:05.023948 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:05.024059 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:05.023962 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" event={"ID":"e4ffd19f-279b-4393-94fc-2fd82bc53f69","Type":"ContainerStarted","Data":"61ec69ae2a39ff46b1f6d818f6c619f532ae9b0ab6d148374ba353be709fc455"} Oct 30 04:13:05.069444 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:05.069394 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" podStartSLOduration=2.069383069 podStartE2EDuration="2.069383069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:05.06860351 +0000 UTC m=+5497.182918499" watchObservedRunningTime="2026-10-30 04:13:05.069383069 +0000 UTC m=+5497.183698038" Oct 30 04:13:05.115659 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:05.115588 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" podStartSLOduration=2.115574694 podStartE2EDuration="2.115574694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:05.114455492 +0000 UTC m=+5497.228770467" watchObservedRunningTime="2026-10-30 04:13:05.115574694 +0000 UTC m=+5497.229889668" Oct 30 04:13:23.941870 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:13:23.941791 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:13:25.029933 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:25.029900 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:13:26.033131 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:26.033101 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:13:26.033518 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:26.033172 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:13:30.499712 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.499671 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-1"] Oct 30 04:13:30.504633 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.504597 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:30.538003 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.537975 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-grpc\"" Oct 30 04:13:30.538162 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.538028 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-http\"" Oct 30 04:13:30.538879 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.538856 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-1"] Oct 30 04:13:30.649205 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.649172 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19706d32-1881-4e8c-b647-323fbec3a23c-config\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:30.649375 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.649218 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:30.649375 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.649238 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:30.649375 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.649334 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:30.649375 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.649357 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mjrk\" (UniqueName: \"kubernetes.io/projected/19706d32-1881-4e8c-b647-323fbec3a23c-kube-api-access-5mjrk\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:30.649537 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.649434 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:30.750650 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.750548 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:30.750650 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.750610 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19706d32-1881-4e8c-b647-323fbec3a23c-config\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:30.750885 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.750658 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:30.750885 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.750675 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:30.750885 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.750696 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:30.750885 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.750713 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mjrk\" (UniqueName: \"kubernetes.io/projected/19706d32-1881-4e8c-b647-323fbec3a23c-kube-api-access-5mjrk\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:30.751417 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.751387 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:30.751599 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.751575 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19706d32-1881-4e8c-b647-323fbec3a23c-config\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:30.753118 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.753096 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:30.753353 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.753329 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:30.753493 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.753478 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:30.776206 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:30.776181 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mjrk\" (UniqueName: \"kubernetes.io/projected/19706d32-1881-4e8c-b647-323fbec3a23c-kube-api-access-5mjrk\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:34.280987 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:34.280955 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:34.313652 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:34.313612 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:34.382134 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:34.382091 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:34.382305 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:34.382229 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:34.398461 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:34.398441 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") DevicePath \"csi-51b362136bc4ecc02770f4e1d3595c1c59eb604f2b3d12558a9a3323d6a31529\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:34.504416 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:34.504385 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/66d42b06118341040bdf4acaf0eeae99e9dd1cda49be064756edb60338b51e13/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:34.532753 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:34.532695 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:34.715286 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:34.715248 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:35.009753 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:35.009722 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-1"] Oct 30 04:13:35.009899 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:13:35.009759 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19706d32_1881_4e8c_b647_323fbec3a23c.slice/crio-8c06c92db682115b0811c1f088e58a43bcc2ede4670703d36347ab9a1bb60d12 WatchSource:0}: Error finding container 8c06c92db682115b0811c1f088e58a43bcc2ede4670703d36347ab9a1bb60d12: Status 404 returned error can't find the container with id 8c06c92db682115b0811c1f088e58a43bcc2ede4670703d36347ab9a1bb60d12 Oct 30 04:13:35.155532 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:35.155496 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" event={"ID":"19706d32-1881-4e8c-b647-323fbec3a23c","Type":"ContainerStarted","Data":"616619d6a458468afef41b1f89b6dba4ec9b17dbf5fba4ac6a62dcae991477f1"} Oct 30 04:13:35.155854 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:35.155553 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" event={"ID":"19706d32-1881-4e8c-b647-323fbec3a23c","Type":"ContainerStarted","Data":"8c06c92db682115b0811c1f088e58a43bcc2ede4670703d36347ab9a1bb60d12"} Oct 30 04:13:35.155854 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:35.155651 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:13:35.204389 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:35.204340 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" podStartSLOduration=5.204325475 podStartE2EDuration="5.204325475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:35.203236587 +0000 UTC m=+5527.317551563" watchObservedRunningTime="2026-10-30 04:13:35.204325475 +0000 UTC m=+5527.318640444" Oct 30 04:13:53.942830 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:13:53.942794 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:13:56.165216 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:13:56.165181 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:14:23.942477 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:14:23.942439 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:14:53.942814 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:14:53.942773 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:15:00.796829 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:00.796794 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"] Oct 30 04:15:00.800647 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:00.800606 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:00.831140 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:00.831113 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-http\"" Oct 30 04:15:00.831270 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:00.831168 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-grpc\"" Oct 30 04:15:00.832371 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:00.832352 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"] Oct 30 04:15:00.948658 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:00.948601 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acde17fb-6ac1-4f56-a720-8739190ba5e0-config\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:00.948874 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:00.948682 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:00.948874 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:00.948785 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:00.948874 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:00.948824 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:00.948996 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:00.948934 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhgp7\" (UniqueName: \"kubernetes.io/projected/acde17fb-6ac1-4f56-a720-8739190ba5e0-kube-api-access-rhgp7\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:00.948996 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:00.948975 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:01.049520 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:01.049432 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:01.049520 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:01.049492 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhgp7\" (UniqueName: \"kubernetes.io/projected/acde17fb-6ac1-4f56-a720-8739190ba5e0-kube-api-access-rhgp7\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:01.049520 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:01.049522 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:01.049844 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:01.049559 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acde17fb-6ac1-4f56-a720-8739190ba5e0-config\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:01.049844 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:01.049735 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:01.049844 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:01.049791 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:01.050253 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:01.050223 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:01.050374 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:01.050355 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acde17fb-6ac1-4f56-a720-8739190ba5e0-config\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:01.052270 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:01.052240 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:01.052587 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:01.052567 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:01.053002 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:01.052980 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:01.075365 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:01.075330 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhgp7\" (UniqueName: \"kubernetes.io/projected/acde17fb-6ac1-4f56-a720-8739190ba5e0-kube-api-access-rhgp7\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:06.002598 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:06.002565 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:06.003092 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:06.002606 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:06.035985 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:06.035957 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:06.036141 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:06.036087 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:06.103769 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:06.103735 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:06.103917 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:06.103773 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:06.103917 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:06.103863 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:06.104008 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:06.103924 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:06.120630 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:06.120580 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") DevicePath \"csi-7f1fa6d16725b1a70ce6c688c15e8c4930aef3140041d1c62d191453190dbfc4\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:06.120718 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:06.120646 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") DevicePath \"csi-a91c9f77820c34a5a63d608a96e7cabde299f4f622113cc391c9b782e45fe5b4\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:06.226997 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:06.226959 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b5a18b684db142dd3a35cb91f8150351c9e560e77f88e25963e593bd05a144db/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:06.235720 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:06.235692 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/393980d41d60e4cbec658b78e9ee48927cfc5faa7d50a5969ab6365fd37edfe9/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:06.259168 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:06.259111 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:06.263502 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:06.263478 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:06.512126 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:06.512046 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:06.817202 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:06.817169 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"] Oct 30 04:15:06.817987 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:15:06.817959 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacde17fb_6ac1_4f56_a720_8739190ba5e0.slice/crio-76a9d8a3e7ace681a56567dc7a36947f08d6113959f568304f11329efbb8162a WatchSource:0}: Error finding container 76a9d8a3e7ace681a56567dc7a36947f08d6113959f568304f11329efbb8162a: Status 404 returned error can't find the container with id 76a9d8a3e7ace681a56567dc7a36947f08d6113959f568304f11329efbb8162a Oct 30 04:15:07.551151 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:07.551115 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" event={"ID":"acde17fb-6ac1-4f56-a720-8739190ba5e0","Type":"ContainerStarted","Data":"e0c231b460de9d1d277f4b04e245131adcc65187fc524c6b6bc9077188528aef"} Oct 30 04:15:07.551151 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:07.551160 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" event={"ID":"acde17fb-6ac1-4f56-a720-8739190ba5e0","Type":"ContainerStarted","Data":"76a9d8a3e7ace681a56567dc7a36947f08d6113959f568304f11329efbb8162a"} Oct 30 04:15:07.551721 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:07.551187 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:07.603116 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:07.603068 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" podStartSLOduration=7.603054376 podStartE2EDuration="7.603054376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:07.597271315 +0000 UTC m=+5619.711586290" watchObservedRunningTime="2026-10-30 04:15:07.603054376 +0000 UTC m=+5619.717369343" Oct 30 04:15:23.942574 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:15:23.942538 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:15:28.561841 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:28.561800 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" podUID="acde17fb-6ac1-4f56-a720-8739190ba5e0" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:15:38.560024 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:38.559983 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" podUID="acde17fb-6ac1-4f56-a720-8739190ba5e0" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:15:48.560003 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:15:48.559971 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:15:53.942520 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:15:53.942482 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:16:16.164477 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.164395 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-1"] Oct 30 04:16:16.165324 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.165294 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" podUID="19706d32-1881-4e8c-b647-323fbec3a23c" containerName="loki-index-gateway" containerID="cri-o://616619d6a458468afef41b1f89b6dba4ec9b17dbf5fba4ac6a62dcae991477f1" gracePeriod=30 Oct 30 04:16:16.206640 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.206594 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"] Oct 30 04:16:16.207125 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.207087 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" podUID="acde17fb-6ac1-4f56-a720-8739190ba5e0" containerName="loki-ingester" containerID="cri-o://e0c231b460de9d1d277f4b04e245131adcc65187fc524c6b6bc9077188528aef" gracePeriod=600 Oct 30 04:16:16.211391 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.211356 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8"] Oct 30 04:16:16.211795 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.211760 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" podUID="7a5bf67f-05b6-43f6-ad63-db2328fbfffe" containerName="loki-distributor" containerID="cri-o://0696d621e061924134f69cfc62ffdc9f074b6dc351591cf8f43709f1eda3f4b0" gracePeriod=30 Oct 30 04:16:16.226704 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.226675 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9"] Oct 30 04:16:16.226915 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.226892 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" podUID="e4ffd19f-279b-4393-94fc-2fd82bc53f69" containerName="loki-querier" containerID="cri-o://67af549ceb8b5f65feec86617e1ab73073958a49b0b8101aaa1d8a61070782d8" gracePeriod=30 Oct 30 04:16:16.236937 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.236906 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d"] Oct 30 04:16:16.237207 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.237144 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" podUID="95e2a939-2c53-428a-958a-b92aeb6782d2" containerName="loki-query-frontend" containerID="cri-o://eca9ed1070b2e23c6fe1a7805b112f44aff5ad3e297ee3ab453d03f7fe3380d7" gracePeriod=30 Oct 30 04:16:16.565432 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.565398 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:16:16.568568 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.568549 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:16:16.655648 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.655598 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-query-frontend-http\") pod \"95e2a939-2c53-428a-958a-b92aeb6782d2\" (UID: \"95e2a939-2c53-428a-958a-b92aeb6782d2\") " Oct 30 04:16:16.655785 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.655662 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-ca-bundle\") pod \"19706d32-1881-4e8c-b647-323fbec3a23c\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " Oct 30 04:16:16.655785 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.655705 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-query-frontend-grpc\") pod \"95e2a939-2c53-428a-958a-b92aeb6782d2\" (UID: \"95e2a939-2c53-428a-958a-b92aeb6782d2\") " Oct 30 04:16:16.655885 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.655811 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19706d32-1881-4e8c-b647-323fbec3a23c-kube-api-access-5mjrk\" (UniqueName: \"kubernetes.io/projected/19706d32-1881-4e8c-b647-323fbec3a23c-kube-api-access-5mjrk\") pod \"19706d32-1881-4e8c-b647-323fbec3a23c\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " Oct 30 04:16:16.655885 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.655855 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95e2a939-2c53-428a-958a-b92aeb6782d2-config\" (UniqueName: \"kubernetes.io/configmap/95e2a939-2c53-428a-958a-b92aeb6782d2-config\") pod \"95e2a939-2c53-428a-958a-b92aeb6782d2\" (UID: \"95e2a939-2c53-428a-958a-b92aeb6782d2\") " Oct 30 04:16:16.655990 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.655886 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-index-gateway-http\") pod \"19706d32-1881-4e8c-b647-323fbec3a23c\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " Oct 30 04:16:16.655990 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.655922 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95e2a939-2c53-428a-958a-b92aeb6782d2-kube-api-access-kk77d\" (UniqueName: \"kubernetes.io/projected/95e2a939-2c53-428a-958a-b92aeb6782d2-kube-api-access-kk77d\") pod \"95e2a939-2c53-428a-958a-b92aeb6782d2\" (UID: \"95e2a939-2c53-428a-958a-b92aeb6782d2\") " Oct 30 04:16:16.655990 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.655960 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-ca-bundle\") pod \"95e2a939-2c53-428a-958a-b92aeb6782d2\" (UID: \"95e2a939-2c53-428a-958a-b92aeb6782d2\") " Oct 30 04:16:16.656152 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.656049 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"19706d32-1881-4e8c-b647-323fbec3a23c\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " Oct 30 04:16:16.656152 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.656092 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-test-loki-lifecycle-secret\") pod \"19706d32-1881-4e8c-b647-323fbec3a23c\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " Oct 30 04:16:16.656386 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.656358 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-ca-bundle" pod "19706d32-1881-4e8c-b647-323fbec3a23c" (UID: "19706d32-1881-4e8c-b647-323fbec3a23c"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:16.656471 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.656399 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95e2a939-2c53-428a-958a-b92aeb6782d2-config" pod "95e2a939-2c53-428a-958a-b92aeb6782d2" (UID: "95e2a939-2c53-428a-958a-b92aeb6782d2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:16.656856 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.656784 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-ca-bundle" pod "95e2a939-2c53-428a-958a-b92aeb6782d2" (UID: "95e2a939-2c53-428a-958a-b92aeb6782d2"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:16.659772 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.659187 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-test-loki-lifecycle-secret" pod "19706d32-1881-4e8c-b647-323fbec3a23c" (UID: "19706d32-1881-4e8c-b647-323fbec3a23c"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:16.660286 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.660238 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00" pod "19706d32-1881-4e8c-b647-323fbec3a23c" (UID: "19706d32-1881-4e8c-b647-323fbec3a23c"). InnerVolumeSpecName "pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:16:16.660286 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.660242 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-query-frontend-http" pod "95e2a939-2c53-428a-958a-b92aeb6782d2" (UID: "95e2a939-2c53-428a-958a-b92aeb6782d2"). InnerVolumeSpecName "data-science-lokistack-query-frontend-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:16.660286 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.660245 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-index-gateway-http" pod "19706d32-1881-4e8c-b647-323fbec3a23c" (UID: "19706d32-1881-4e8c-b647-323fbec3a23c"). InnerVolumeSpecName "data-science-lokistack-index-gateway-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:16.660286 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.660271 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e2a939-2c53-428a-958a-b92aeb6782d2-kube-api-access-kk77d" pod "95e2a939-2c53-428a-958a-b92aeb6782d2" (UID: "95e2a939-2c53-428a-958a-b92aeb6782d2"). InnerVolumeSpecName "kube-api-access-kk77d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:16:16.661312 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.661267 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19706d32-1881-4e8c-b647-323fbec3a23c-kube-api-access-5mjrk" pod "19706d32-1881-4e8c-b647-323fbec3a23c" (UID: "19706d32-1881-4e8c-b647-323fbec3a23c"). InnerVolumeSpecName "kube-api-access-5mjrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:16:16.662016 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.661981 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-query-frontend-grpc" pod "95e2a939-2c53-428a-958a-b92aeb6782d2" (UID: "95e2a939-2c53-428a-958a-b92aeb6782d2"). InnerVolumeSpecName "data-science-lokistack-query-frontend-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:16.756875 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.756842 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19706d32-1881-4e8c-b647-323fbec3a23c-config\" (UniqueName: \"kubernetes.io/configmap/19706d32-1881-4e8c-b647-323fbec3a23c-config\") pod \"19706d32-1881-4e8c-b647-323fbec3a23c\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " Oct 30 04:16:16.757050 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.756888 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-index-gateway-grpc\") pod \"19706d32-1881-4e8c-b647-323fbec3a23c\" (UID: \"19706d32-1881-4e8c-b647-323fbec3a23c\") " Oct 30 04:16:16.757227 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.757185 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mjrk\" (UniqueName: \"kubernetes.io/projected/19706d32-1881-4e8c-b647-323fbec3a23c-kube-api-access-5mjrk\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.757227 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.757211 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95e2a939-2c53-428a-958a-b92aeb6782d2-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.757399 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.757227 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-index-gateway-http\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.757399 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.757242 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk77d\" (UniqueName: \"kubernetes.io/projected/95e2a939-2c53-428a-958a-b92aeb6782d2-kube-api-access-kk77d\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.757399 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.757244 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19706d32-1881-4e8c-b647-323fbec3a23c-config" pod "19706d32-1881-4e8c-b647-323fbec3a23c" (UID: "19706d32-1881-4e8c-b647-323fbec3a23c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:16.757399 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.757258 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-ca-bundle\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.757399 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.757314 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:16:16.757399 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.757334 5882 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-test-loki-lifecycle-secret\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.757399 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.757350 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-query-frontend-http\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.757399 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.757367 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-ca-bundle\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.757399 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.757384 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/95e2a939-2c53-428a-958a-b92aeb6782d2-data-science-lokistack-query-frontend-grpc\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.759916 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.759884 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-index-gateway-grpc" pod "19706d32-1881-4e8c-b647-323fbec3a23c" (UID: "19706d32-1881-4e8c-b647-323fbec3a23c"). InnerVolumeSpecName "data-science-lokistack-index-gateway-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:16.767361 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.767339 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:16:16.803228 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.803205 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:16:16.849279 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.849238 5882 generic.go:356] "Generic (PLEG): container finished" podID="95e2a939-2c53-428a-958a-b92aeb6782d2" containerID="eca9ed1070b2e23c6fe1a7805b112f44aff5ad3e297ee3ab453d03f7fe3380d7" exitCode=0 Oct 30 04:16:16.849342 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.849285 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" Oct 30 04:16:16.849342 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.849294 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" event={"ID":"95e2a939-2c53-428a-958a-b92aeb6782d2","Type":"ContainerDied","Data":"eca9ed1070b2e23c6fe1a7805b112f44aff5ad3e297ee3ab453d03f7fe3380d7"} Oct 30 04:16:16.849429 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.849352 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d" event={"ID":"95e2a939-2c53-428a-958a-b92aeb6782d2","Type":"ContainerDied","Data":"7bf180bcfcdbdb8644c3ea0498e76901324b418dfbb7796569749427a4a82607"} Oct 30 04:16:16.849429 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.849380 5882 scope.go:122] "RemoveContainer" containerID="eca9ed1070b2e23c6fe1a7805b112f44aff5ad3e297ee3ab453d03f7fe3380d7" Oct 30 04:16:16.851565 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.851536 5882 generic.go:356] "Generic (PLEG): container finished" podID="e4ffd19f-279b-4393-94fc-2fd82bc53f69" containerID="67af549ceb8b5f65feec86617e1ab73073958a49b0b8101aaa1d8a61070782d8" exitCode=0 Oct 30 04:16:16.851730 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.851711 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" event={"ID":"e4ffd19f-279b-4393-94fc-2fd82bc53f69","Type":"ContainerDied","Data":"67af549ceb8b5f65feec86617e1ab73073958a49b0b8101aaa1d8a61070782d8"} Oct 30 04:16:16.851789 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.851756 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" event={"ID":"e4ffd19f-279b-4393-94fc-2fd82bc53f69","Type":"ContainerDied","Data":"61ec69ae2a39ff46b1f6d818f6c619f532ae9b0ab6d148374ba353be709fc455"} Oct 30 04:16:16.851837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.851829 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9" Oct 30 04:16:16.854114 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.854089 5882 generic.go:356] "Generic (PLEG): container finished" podID="19706d32-1881-4e8c-b647-323fbec3a23c" containerID="616619d6a458468afef41b1f89b6dba4ec9b17dbf5fba4ac6a62dcae991477f1" exitCode=0 Oct 30 04:16:16.854220 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.854130 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:16:16.854220 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.854130 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" event={"ID":"19706d32-1881-4e8c-b647-323fbec3a23c","Type":"ContainerDied","Data":"616619d6a458468afef41b1f89b6dba4ec9b17dbf5fba4ac6a62dcae991477f1"} Oct 30 04:16:16.854326 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.854253 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" event={"ID":"19706d32-1881-4e8c-b647-323fbec3a23c","Type":"ContainerDied","Data":"8c06c92db682115b0811c1f088e58a43bcc2ede4670703d36347ab9a1bb60d12"} Oct 30 04:16:16.857807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.857785 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4ffd19f-279b-4393-94fc-2fd82bc53f69-config\" (UniqueName: \"kubernetes.io/configmap/e4ffd19f-279b-4393-94fc-2fd82bc53f69-config\") pod \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " Oct 30 04:16:16.857912 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.857842 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-querier-grpc\") pod \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " Oct 30 04:16:16.857912 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.857888 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-ca-bundle\") pod \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " Oct 30 04:16:16.858029 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.857953 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-querier-http\") pod \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " Oct 30 04:16:16.858029 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.858005 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-test-loki-lifecycle-secret\") pod \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " Oct 30 04:16:16.858140 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.858060 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ffd19f-279b-4393-94fc-2fd82bc53f69-kube-api-access-plsvq\" (UniqueName: \"kubernetes.io/projected/e4ffd19f-279b-4393-94fc-2fd82bc53f69-kube-api-access-plsvq\") pod \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\" (UID: \"e4ffd19f-279b-4393-94fc-2fd82bc53f69\") " Oct 30 04:16:16.858193 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.858163 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ffd19f-279b-4393-94fc-2fd82bc53f69-config" pod "e4ffd19f-279b-4393-94fc-2fd82bc53f69" (UID: "e4ffd19f-279b-4393-94fc-2fd82bc53f69"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:16.858355 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.858334 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4ffd19f-279b-4393-94fc-2fd82bc53f69-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.858425 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.858360 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-51b362136bc4ecc02770f4e1d3595c1c59eb604f2b3d12558a9a3323d6a31529\"" Oct 30 04:16:16.858425 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.858377 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19706d32-1881-4e8c-b647-323fbec3a23c-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.858532 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.858392 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/19706d32-1881-4e8c-b647-323fbec3a23c-data-science-lokistack-index-gateway-grpc\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.858593 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.858571 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-ca-bundle" pod "e4ffd19f-279b-4393-94fc-2fd82bc53f69" (UID: "e4ffd19f-279b-4393-94fc-2fd82bc53f69"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:16.860318 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.860296 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-querier-http" pod "e4ffd19f-279b-4393-94fc-2fd82bc53f69" (UID: "e4ffd19f-279b-4393-94fc-2fd82bc53f69"). InnerVolumeSpecName "data-science-lokistack-querier-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:16.860714 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.860598 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ffd19f-279b-4393-94fc-2fd82bc53f69-kube-api-access-plsvq" pod "e4ffd19f-279b-4393-94fc-2fd82bc53f69" (UID: "e4ffd19f-279b-4393-94fc-2fd82bc53f69"). InnerVolumeSpecName "kube-api-access-plsvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:16:16.860879 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.860853 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-querier-grpc" pod "e4ffd19f-279b-4393-94fc-2fd82bc53f69" (UID: "e4ffd19f-279b-4393-94fc-2fd82bc53f69"). InnerVolumeSpecName "data-science-lokistack-querier-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:16.860953 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.860863 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-test-loki-lifecycle-secret" pod "e4ffd19f-279b-4393-94fc-2fd82bc53f69" (UID: "e4ffd19f-279b-4393-94fc-2fd82bc53f69"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:16.924085 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.924061 5882 scope.go:122] "RemoveContainer" containerID="eca9ed1070b2e23c6fe1a7805b112f44aff5ad3e297ee3ab453d03f7fe3380d7" Oct 30 04:16:16.924387 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:16:16.924370 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eca9ed1070b2e23c6fe1a7805b112f44aff5ad3e297ee3ab453d03f7fe3380d7\": container with ID starting with eca9ed1070b2e23c6fe1a7805b112f44aff5ad3e297ee3ab453d03f7fe3380d7 not found: ID does not exist" containerID="eca9ed1070b2e23c6fe1a7805b112f44aff5ad3e297ee3ab453d03f7fe3380d7" Oct 30 04:16:16.924444 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.924395 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca9ed1070b2e23c6fe1a7805b112f44aff5ad3e297ee3ab453d03f7fe3380d7"} err="failed to get container status \"eca9ed1070b2e23c6fe1a7805b112f44aff5ad3e297ee3ab453d03f7fe3380d7\": rpc error: code = NotFound desc = could not find container \"eca9ed1070b2e23c6fe1a7805b112f44aff5ad3e297ee3ab453d03f7fe3380d7\": container with ID starting with eca9ed1070b2e23c6fe1a7805b112f44aff5ad3e297ee3ab453d03f7fe3380d7 not found: ID does not exist" Oct 30 04:16:16.924444 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.924411 5882 scope.go:122] "RemoveContainer" containerID="67af549ceb8b5f65feec86617e1ab73073958a49b0b8101aaa1d8a61070782d8" Oct 30 04:16:16.932173 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.932151 5882 scope.go:122] "RemoveContainer" containerID="67af549ceb8b5f65feec86617e1ab73073958a49b0b8101aaa1d8a61070782d8" Oct 30 04:16:16.932434 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:16:16.932406 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67af549ceb8b5f65feec86617e1ab73073958a49b0b8101aaa1d8a61070782d8\": container with ID starting with 67af549ceb8b5f65feec86617e1ab73073958a49b0b8101aaa1d8a61070782d8 not found: ID does not exist" containerID="67af549ceb8b5f65feec86617e1ab73073958a49b0b8101aaa1d8a61070782d8" Oct 30 04:16:16.932530 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.932434 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67af549ceb8b5f65feec86617e1ab73073958a49b0b8101aaa1d8a61070782d8"} err="failed to get container status \"67af549ceb8b5f65feec86617e1ab73073958a49b0b8101aaa1d8a61070782d8\": rpc error: code = NotFound desc = could not find container \"67af549ceb8b5f65feec86617e1ab73073958a49b0b8101aaa1d8a61070782d8\": container with ID starting with 67af549ceb8b5f65feec86617e1ab73073958a49b0b8101aaa1d8a61070782d8 not found: ID does not exist" Oct 30 04:16:16.932530 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.932450 5882 scope.go:122] "RemoveContainer" containerID="616619d6a458468afef41b1f89b6dba4ec9b17dbf5fba4ac6a62dcae991477f1" Oct 30 04:16:16.939141 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.939119 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d"] Oct 30 04:16:16.942102 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.942078 5882 scope.go:122] "RemoveContainer" containerID="616619d6a458468afef41b1f89b6dba4ec9b17dbf5fba4ac6a62dcae991477f1" Oct 30 04:16:16.942384 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:16:16.942362 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"616619d6a458468afef41b1f89b6dba4ec9b17dbf5fba4ac6a62dcae991477f1\": container with ID starting with 616619d6a458468afef41b1f89b6dba4ec9b17dbf5fba4ac6a62dcae991477f1 not found: ID does not exist" containerID="616619d6a458468afef41b1f89b6dba4ec9b17dbf5fba4ac6a62dcae991477f1" Oct 30 04:16:16.942459 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.942390 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616619d6a458468afef41b1f89b6dba4ec9b17dbf5fba4ac6a62dcae991477f1"} err="failed to get container status \"616619d6a458468afef41b1f89b6dba4ec9b17dbf5fba4ac6a62dcae991477f1\": rpc error: code = NotFound desc = could not find container \"616619d6a458468afef41b1f89b6dba4ec9b17dbf5fba4ac6a62dcae991477f1\": container with ID starting with 616619d6a458468afef41b1f89b6dba4ec9b17dbf5fba4ac6a62dcae991477f1 not found: ID does not exist" Oct 30 04:16:16.943092 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.943071 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-fpf2d"] Oct 30 04:16:16.959175 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.959047 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-querier-grpc\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.959175 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.959079 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-ca-bundle\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.959175 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.959096 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-data-science-lokistack-querier-http\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.959175 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.959111 5882 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/e4ffd19f-279b-4393-94fc-2fd82bc53f69-test-loki-lifecycle-secret\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.959175 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.959126 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plsvq\" (UniqueName: \"kubernetes.io/projected/e4ffd19f-279b-4393-94fc-2fd82bc53f69-kube-api-access-plsvq\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.981611 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.981579 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-1"] Oct 30 04:16:16.986411 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:16.986383 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-1"] Oct 30 04:16:17.068314 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.068283 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:16:17.143668 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.143641 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:16:17.160361 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.160338 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-distributor-grpc\") pod \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\" (UID: \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\") " Oct 30 04:16:17.160468 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.160371 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-ca-bundle\") pod \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\" (UID: \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\") " Oct 30 04:16:17.160468 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.160410 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ca-bundle\") pod \"acde17fb-6ac1-4f56-a720-8739190ba5e0\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " Oct 30 04:16:17.160468 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.160443 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-distributor-http\") pod \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\" (UID: \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\") " Oct 30 04:16:17.160665 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.160530 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"acde17fb-6ac1-4f56-a720-8739190ba5e0\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " Oct 30 04:16:17.160665 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.160573 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acde17fb-6ac1-4f56-a720-8739190ba5e0-config\" (UniqueName: \"kubernetes.io/configmap/acde17fb-6ac1-4f56-a720-8739190ba5e0-config\") pod \"acde17fb-6ac1-4f56-a720-8739190ba5e0\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " Oct 30 04:16:17.160665 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.160609 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ingester-grpc\") pod \"acde17fb-6ac1-4f56-a720-8739190ba5e0\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " Oct 30 04:16:17.160837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.160675 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-config\" (UniqueName: \"kubernetes.io/configmap/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-config\") pod \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\" (UID: \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\") " Oct 30 04:16:17.160837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.160714 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-test-loki-lifecycle-secret\") pod \"acde17fb-6ac1-4f56-a720-8739190ba5e0\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " Oct 30 04:16:17.160837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.160745 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acde17fb-6ac1-4f56-a720-8739190ba5e0-kube-api-access-rhgp7\" (UniqueName: \"kubernetes.io/projected/acde17fb-6ac1-4f56-a720-8739190ba5e0-kube-api-access-rhgp7\") pod \"acde17fb-6ac1-4f56-a720-8739190ba5e0\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " Oct 30 04:16:17.160837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.160772 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-ca-bundle" pod "7a5bf67f-05b6-43f6-ad63-db2328fbfffe" (UID: "7a5bf67f-05b6-43f6-ad63-db2328fbfffe"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:17.160837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.160788 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ingester-http\") pod \"acde17fb-6ac1-4f56-a720-8739190ba5e0\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " Oct 30 04:16:17.161086 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.160950 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ca-bundle" pod "acde17fb-6ac1-4f56-a720-8739190ba5e0" (UID: "acde17fb-6ac1-4f56-a720-8739190ba5e0"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:17.162047 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.161240 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-kube-api-access-wnc9j\" (UniqueName: \"kubernetes.io/projected/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-kube-api-access-wnc9j\") pod \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\" (UID: \"7a5bf67f-05b6-43f6-ad63-db2328fbfffe\") " Oct 30 04:16:17.162047 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.161356 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"acde17fb-6ac1-4f56-a720-8739190ba5e0\" (UID: \"acde17fb-6ac1-4f56-a720-8739190ba5e0\") " Oct 30 04:16:17.162047 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.161680 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-ca-bundle\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.162047 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.161701 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ca-bundle\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.162047 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.161717 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-config" pod "7a5bf67f-05b6-43f6-ad63-db2328fbfffe" (UID: "7a5bf67f-05b6-43f6-ad63-db2328fbfffe"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:17.162047 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.161884 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acde17fb-6ac1-4f56-a720-8739190ba5e0-config" pod "acde17fb-6ac1-4f56-a720-8739190ba5e0" (UID: "acde17fb-6ac1-4f56-a720-8739190ba5e0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:17.163183 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.162969 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-distributor-grpc" pod "7a5bf67f-05b6-43f6-ad63-db2328fbfffe" (UID: "7a5bf67f-05b6-43f6-ad63-db2328fbfffe"). InnerVolumeSpecName "data-science-lokistack-distributor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:17.163358 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.163328 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-distributor-http" pod "7a5bf67f-05b6-43f6-ad63-db2328fbfffe" (UID: "7a5bf67f-05b6-43f6-ad63-db2328fbfffe"). InnerVolumeSpecName "data-science-lokistack-distributor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:17.163358 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.163342 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ingester-http" pod "acde17fb-6ac1-4f56-a720-8739190ba5e0" (UID: "acde17fb-6ac1-4f56-a720-8739190ba5e0"). InnerVolumeSpecName "data-science-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:17.163877 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.163853 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ingester-grpc" pod "acde17fb-6ac1-4f56-a720-8739190ba5e0" (UID: "acde17fb-6ac1-4f56-a720-8739190ba5e0"). InnerVolumeSpecName "data-science-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:17.164776 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.164746 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-test-loki-lifecycle-secret" pod "acde17fb-6ac1-4f56-a720-8739190ba5e0" (UID: "acde17fb-6ac1-4f56-a720-8739190ba5e0"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:17.165189 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.165166 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050" pod "acde17fb-6ac1-4f56-a720-8739190ba5e0" (UID: "acde17fb-6ac1-4f56-a720-8739190ba5e0"). InnerVolumeSpecName "pvc-5441af3d-1475-431a-b556-4acba3990893". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:16:17.165278 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.165192 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61" pod "acde17fb-6ac1-4f56-a720-8739190ba5e0" (UID: "acde17fb-6ac1-4f56-a720-8739190ba5e0"). InnerVolumeSpecName "pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:16:17.165278 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.165231 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-kube-api-access-wnc9j" pod "7a5bf67f-05b6-43f6-ad63-db2328fbfffe" (UID: "7a5bf67f-05b6-43f6-ad63-db2328fbfffe"). InnerVolumeSpecName "kube-api-access-wnc9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:16:17.167775 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.167758 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acde17fb-6ac1-4f56-a720-8739190ba5e0-kube-api-access-rhgp7" pod "acde17fb-6ac1-4f56-a720-8739190ba5e0" (UID: "acde17fb-6ac1-4f56-a720-8739190ba5e0"). InnerVolumeSpecName "kube-api-access-rhgp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:16:17.230436 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.230402 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9"] Oct 30 04:16:17.234843 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.234815 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-4w2d9"] Oct 30 04:16:17.262192 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.262154 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ingester-http\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.262192 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.262187 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnc9j\" (UniqueName: \"kubernetes.io/projected/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-kube-api-access-wnc9j\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.262361 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.262215 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:16:17.262361 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.262226 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-distributor-grpc\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.262361 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.262237 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-data-science-lokistack-distributor-http\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.262361 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.262260 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:16:17.262361 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.262270 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acde17fb-6ac1-4f56-a720-8739190ba5e0-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.262361 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.262279 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-data-science-lokistack-ingester-grpc\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.262361 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.262287 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7a5bf67f-05b6-43f6-ad63-db2328fbfffe-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.262361 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.262296 5882 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/acde17fb-6ac1-4f56-a720-8739190ba5e0-test-loki-lifecycle-secret\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.262361 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.262305 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhgp7\" (UniqueName: \"kubernetes.io/projected/acde17fb-6ac1-4f56-a720-8739190ba5e0-kube-api-access-rhgp7\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.273492 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.273469 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:16:17.277100 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.277054 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-5441af3d-1475-431a-b556-4acba3990893" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:16:17.363590 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.363554 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-a91c9f77820c34a5a63d608a96e7cabde299f4f622113cc391c9b782e45fe5b4\"" Oct 30 04:16:17.363590 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.363586 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-7f1fa6d16725b1a70ce6c688c15e8c4930aef3140041d1c62d191453190dbfc4\"" Oct 30 04:16:17.859869 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.859835 5882 generic.go:356] "Generic (PLEG): container finished" podID="acde17fb-6ac1-4f56-a720-8739190ba5e0" containerID="e0c231b460de9d1d277f4b04e245131adcc65187fc524c6b6bc9077188528aef" exitCode=0 Oct 30 04:16:17.860031 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.859881 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" Oct 30 04:16:17.860031 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.859923 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" event={"ID":"acde17fb-6ac1-4f56-a720-8739190ba5e0","Type":"ContainerDied","Data":"e0c231b460de9d1d277f4b04e245131adcc65187fc524c6b6bc9077188528aef"} Oct 30 04:16:17.860031 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.859976 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" event={"ID":"acde17fb-6ac1-4f56-a720-8739190ba5e0","Type":"ContainerDied","Data":"76a9d8a3e7ace681a56567dc7a36947f08d6113959f568304f11329efbb8162a"} Oct 30 04:16:17.860031 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.860004 5882 scope.go:122] "RemoveContainer" containerID="e0c231b460de9d1d277f4b04e245131adcc65187fc524c6b6bc9077188528aef" Oct 30 04:16:17.864107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.864084 5882 generic.go:356] "Generic (PLEG): container finished" podID="7a5bf67f-05b6-43f6-ad63-db2328fbfffe" containerID="0696d621e061924134f69cfc62ffdc9f074b6dc351591cf8f43709f1eda3f4b0" exitCode=0 Oct 30 04:16:17.864221 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.864140 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" Oct 30 04:16:17.864221 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.864142 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" event={"ID":"7a5bf67f-05b6-43f6-ad63-db2328fbfffe","Type":"ContainerDied","Data":"0696d621e061924134f69cfc62ffdc9f074b6dc351591cf8f43709f1eda3f4b0"} Oct 30 04:16:17.864297 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.864249 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8" event={"ID":"7a5bf67f-05b6-43f6-ad63-db2328fbfffe","Type":"ContainerDied","Data":"23edcc3117c0af21483d619ebc369a99067a63663c124365cae3e9681247dda7"} Oct 30 04:16:17.869772 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.869691 5882 scope.go:122] "RemoveContainer" containerID="e0c231b460de9d1d277f4b04e245131adcc65187fc524c6b6bc9077188528aef" Oct 30 04:16:17.870110 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:16:17.870077 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c231b460de9d1d277f4b04e245131adcc65187fc524c6b6bc9077188528aef\": container with ID starting with e0c231b460de9d1d277f4b04e245131adcc65187fc524c6b6bc9077188528aef not found: ID does not exist" containerID="e0c231b460de9d1d277f4b04e245131adcc65187fc524c6b6bc9077188528aef" Oct 30 04:16:17.870216 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.870105 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c231b460de9d1d277f4b04e245131adcc65187fc524c6b6bc9077188528aef"} err="failed to get container status \"e0c231b460de9d1d277f4b04e245131adcc65187fc524c6b6bc9077188528aef\": rpc error: code = NotFound desc = could not find container \"e0c231b460de9d1d277f4b04e245131adcc65187fc524c6b6bc9077188528aef\": container with ID starting with e0c231b460de9d1d277f4b04e245131adcc65187fc524c6b6bc9077188528aef not found: ID does not exist" Oct 30 04:16:17.870216 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.870127 5882 scope.go:122] "RemoveContainer" containerID="0696d621e061924134f69cfc62ffdc9f074b6dc351591cf8f43709f1eda3f4b0" Oct 30 04:16:17.877764 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.877747 5882 scope.go:122] "RemoveContainer" containerID="0696d621e061924134f69cfc62ffdc9f074b6dc351591cf8f43709f1eda3f4b0" Oct 30 04:16:17.878014 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:16:17.877999 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0696d621e061924134f69cfc62ffdc9f074b6dc351591cf8f43709f1eda3f4b0\": container with ID starting with 0696d621e061924134f69cfc62ffdc9f074b6dc351591cf8f43709f1eda3f4b0 not found: ID does not exist" containerID="0696d621e061924134f69cfc62ffdc9f074b6dc351591cf8f43709f1eda3f4b0" Oct 30 04:16:17.878068 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.878019 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0696d621e061924134f69cfc62ffdc9f074b6dc351591cf8f43709f1eda3f4b0"} err="failed to get container status \"0696d621e061924134f69cfc62ffdc9f074b6dc351591cf8f43709f1eda3f4b0\": rpc error: code = NotFound desc = could not find container \"0696d621e061924134f69cfc62ffdc9f074b6dc351591cf8f43709f1eda3f4b0\": container with ID starting with 0696d621e061924134f69cfc62ffdc9f074b6dc351591cf8f43709f1eda3f4b0 not found: ID does not exist" Oct 30 04:16:17.915788 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.915765 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"] Oct 30 04:16:17.919716 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.919694 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"] Oct 30 04:16:17.957888 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.957855 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8"] Oct 30 04:16:17.960121 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:17.960094 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-mjcm8"] Oct 30 04:16:18.014737 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:18.014696 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19706d32-1881-4e8c-b647-323fbec3a23c" path="/var/lib/kubelet/pods/19706d32-1881-4e8c-b647-323fbec3a23c/volumes" Oct 30 04:16:18.015198 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:18.015178 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a5bf67f-05b6-43f6-ad63-db2328fbfffe" path="/var/lib/kubelet/pods/7a5bf67f-05b6-43f6-ad63-db2328fbfffe/volumes" Oct 30 04:16:18.015606 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:18.015589 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95e2a939-2c53-428a-958a-b92aeb6782d2" path="/var/lib/kubelet/pods/95e2a939-2c53-428a-958a-b92aeb6782d2/volumes" Oct 30 04:16:18.016462 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:18.016438 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acde17fb-6ac1-4f56-a720-8739190ba5e0" path="/var/lib/kubelet/pods/acde17fb-6ac1-4f56-a720-8739190ba5e0/volumes" Oct 30 04:16:18.016854 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:18.016837 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4ffd19f-279b-4393-94fc-2fd82bc53f69" path="/var/lib/kubelet/pods/e4ffd19f-279b-4393-94fc-2fd82bc53f69/volumes" Oct 30 04:16:18.736354 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:18.736320 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 30 04:16:18.740546 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:18.740521 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 30 04:16:18.774227 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:18.774203 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 30 04:16:18.774417 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:18.774384 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/seaweedfs" podUID="8bdebe0d-7c01-4821-826c-70541e8c32f1" containerName="seaweedfs" containerID="cri-o://429af317d05c4ec72bad8fab703cfd484a7b6df11139c1be561d65004df2413d" gracePeriod=30 Oct 30 04:16:20.009115 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:20.009078 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d2b141d-937e-42c8-a33b-884ee2617959" path="/var/lib/kubelet/pods/5d2b141d-937e-42c8-a33b-884ee2617959/volumes" Oct 30 04:16:23.942445 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:16:23.942409 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:16:28.587356 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:28.587239 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 04:16:28.620151 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:28.601197 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 04:16:29.368399 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:29.368369 5882 scope.go:122] "RemoveContainer" containerID="bba2bb1bb8ff71a09f27c6f7c65b47d925f9493b44a9fccceb05a3b75b4f1aa3" Oct 30 04:16:44.118768 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:44.118744 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs" Oct 30 04:16:44.206057 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:44.205973 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bdebe0d-7c01-4821-826c-70541e8c32f1-kube-api-access-dpzgt\" (UniqueName: \"kubernetes.io/projected/8bdebe0d-7c01-4821-826c-70541e8c32f1-kube-api-access-dpzgt\") pod \"8bdebe0d-7c01-4821-826c-70541e8c32f1\" (UID: \"8bdebe0d-7c01-4821-826c-70541e8c32f1\") " Oct 30 04:16:44.206464 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:44.206098 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdebe0d-7c01-4821-826c-70541e8c32f1-data\" (UniqueName: \"kubernetes.io/empty-dir/8bdebe0d-7c01-4821-826c-70541e8c32f1-data\") pod \"8bdebe0d-7c01-4821-826c-70541e8c32f1\" (UID: \"8bdebe0d-7c01-4821-826c-70541e8c32f1\") " Oct 30 04:16:44.207469 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:44.207441 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdebe0d-7c01-4821-826c-70541e8c32f1-data" pod "8bdebe0d-7c01-4821-826c-70541e8c32f1" (UID: "8bdebe0d-7c01-4821-826c-70541e8c32f1"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:16:44.207921 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:44.207894 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bdebe0d-7c01-4821-826c-70541e8c32f1-kube-api-access-dpzgt" pod "8bdebe0d-7c01-4821-826c-70541e8c32f1" (UID: "8bdebe0d-7c01-4821-826c-70541e8c32f1"). InnerVolumeSpecName "kube-api-access-dpzgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:16:44.307282 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:44.307241 5882 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8bdebe0d-7c01-4821-826c-70541e8c32f1-data\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:44.307282 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:44.307279 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpzgt\" (UniqueName: \"kubernetes.io/projected/8bdebe0d-7c01-4821-826c-70541e8c32f1-kube-api-access-dpzgt\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:44.979424 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:44.979385 5882 generic.go:356] "Generic (PLEG): container finished" podID="8bdebe0d-7c01-4821-826c-70541e8c32f1" containerID="429af317d05c4ec72bad8fab703cfd484a7b6df11139c1be561d65004df2413d" exitCode=0 Oct 30 04:16:44.979649 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:44.979447 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs" Oct 30 04:16:44.979649 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:44.979461 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs" event={"ID":"8bdebe0d-7c01-4821-826c-70541e8c32f1","Type":"ContainerDied","Data":"429af317d05c4ec72bad8fab703cfd484a7b6df11139c1be561d65004df2413d"} Oct 30 04:16:44.979649 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:44.979502 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs" event={"ID":"8bdebe0d-7c01-4821-826c-70541e8c32f1","Type":"ContainerDied","Data":"0e35bbd9b544dada6fbaa43010f9ae08b1f95cb44ea472bddb6c64fcfab91a15"} Oct 30 04:16:44.979649 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:44.979518 5882 scope.go:122] "RemoveContainer" containerID="429af317d05c4ec72bad8fab703cfd484a7b6df11139c1be561d65004df2413d" Oct 30 04:16:44.988679 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:44.988665 5882 scope.go:122] "RemoveContainer" containerID="429af317d05c4ec72bad8fab703cfd484a7b6df11139c1be561d65004df2413d" Oct 30 04:16:44.988914 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:16:44.988895 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"429af317d05c4ec72bad8fab703cfd484a7b6df11139c1be561d65004df2413d\": container with ID starting with 429af317d05c4ec72bad8fab703cfd484a7b6df11139c1be561d65004df2413d not found: ID does not exist" containerID="429af317d05c4ec72bad8fab703cfd484a7b6df11139c1be561d65004df2413d" Oct 30 04:16:44.988965 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:44.988920 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429af317d05c4ec72bad8fab703cfd484a7b6df11139c1be561d65004df2413d"} err="failed to get container status \"429af317d05c4ec72bad8fab703cfd484a7b6df11139c1be561d65004df2413d\": rpc error: code = NotFound desc = could not find container \"429af317d05c4ec72bad8fab703cfd484a7b6df11139c1be561d65004df2413d\": container with ID starting with 429af317d05c4ec72bad8fab703cfd484a7b6df11139c1be561d65004df2413d not found: ID does not exist" Oct 30 04:16:45.034807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:45.034772 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 30 04:16:45.038212 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:45.038188 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 30 04:16:46.008598 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:46.008566 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bdebe0d-7c01-4821-826c-70541e8c32f1" path="/var/lib/kubelet/pods/8bdebe0d-7c01-4821-826c-70541e8c32f1/volumes" Oct 30 04:16:50.648892 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.648861 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9"] Oct 30 04:16:50.649357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.649216 5882 cpu_manager.go:414] "containerMap: removing container" podUID="acde17fb-6ac1-4f56-a720-8739190ba5e0" containerName="loki-ingester" Oct 30 04:16:50.649357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.649227 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acde17fb-6ac1-4f56-a720-8739190ba5e0" containerName="loki-ingester" Oct 30 04:16:50.649357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.649246 5882 cpu_manager.go:414] "containerMap: removing container" podUID="e4ffd19f-279b-4393-94fc-2fd82bc53f69" containerName="loki-querier" Oct 30 04:16:50.649357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.649251 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ffd19f-279b-4393-94fc-2fd82bc53f69" containerName="loki-querier" Oct 30 04:16:50.649357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.649267 5882 cpu_manager.go:414] "containerMap: removing container" podUID="8bdebe0d-7c01-4821-826c-70541e8c32f1" containerName="seaweedfs" Oct 30 04:16:50.649357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.649272 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bdebe0d-7c01-4821-826c-70541e8c32f1" containerName="seaweedfs" Oct 30 04:16:50.649357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.649283 5882 cpu_manager.go:414] "containerMap: removing container" podUID="7a5bf67f-05b6-43f6-ad63-db2328fbfffe" containerName="loki-distributor" Oct 30 04:16:50.649357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.649289 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a5bf67f-05b6-43f6-ad63-db2328fbfffe" containerName="loki-distributor" Oct 30 04:16:50.649357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.649296 5882 cpu_manager.go:414] "containerMap: removing container" podUID="95e2a939-2c53-428a-958a-b92aeb6782d2" containerName="loki-query-frontend" Oct 30 04:16:50.649357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.649301 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95e2a939-2c53-428a-958a-b92aeb6782d2" containerName="loki-query-frontend" Oct 30 04:16:50.649357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.649307 5882 cpu_manager.go:414] "containerMap: removing container" podUID="19706d32-1881-4e8c-b647-323fbec3a23c" containerName="loki-index-gateway" Oct 30 04:16:50.649357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.649312 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19706d32-1881-4e8c-b647-323fbec3a23c" containerName="loki-index-gateway" Oct 30 04:16:50.649357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.649363 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a5bf67f-05b6-43f6-ad63-db2328fbfffe" containerName="loki-distributor" Oct 30 04:16:50.649779 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.649372 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="95e2a939-2c53-428a-958a-b92aeb6782d2" containerName="loki-query-frontend" Oct 30 04:16:50.649779 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.649378 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="19706d32-1881-4e8c-b647-323fbec3a23c" containerName="loki-index-gateway" Oct 30 04:16:50.649779 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.649385 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="acde17fb-6ac1-4f56-a720-8739190ba5e0" containerName="loki-ingester" Oct 30 04:16:50.649779 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.649392 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ffd19f-279b-4393-94fc-2fd82bc53f69" containerName="loki-querier" Oct 30 04:16:50.649779 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.649399 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bdebe0d-7c01-4821-826c-70541e8c32f1" containerName="seaweedfs" Oct 30 04:16:50.652335 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.652315 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9" Oct 30 04:16:50.676811 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.676788 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 04:16:50.676950 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.676837 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 04:16:50.677096 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.677080 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 04:16:50.677960 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.677935 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9"] Oct 30 04:16:50.765302 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.765265 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79bcb\" (UniqueName: \"kubernetes.io/projected/13fb3a4f-721a-477f-bcce-6279c2b5e3fe-kube-api-access-79bcb\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9\" (UID: \"13fb3a4f-721a-477f-bcce-6279c2b5e3fe\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9" Oct 30 04:16:50.866570 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.866536 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79bcb\" (UniqueName: \"kubernetes.io/projected/13fb3a4f-721a-477f-bcce-6279c2b5e3fe-kube-api-access-79bcb\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9\" (UID: \"13fb3a4f-721a-477f-bcce-6279c2b5e3fe\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9" Oct 30 04:16:50.892284 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.892248 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79bcb\" (UniqueName: \"kubernetes.io/projected/13fb3a4f-721a-477f-bcce-6279c2b5e3fe-kube-api-access-79bcb\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9\" (UID: \"13fb3a4f-721a-477f-bcce-6279c2b5e3fe\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9" Oct 30 04:16:50.974211 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:50.974129 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9" Oct 30 04:16:51.269043 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.269010 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9"] Oct 30 04:16:51.269329 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:16:51.269307 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13fb3a4f_721a_477f_bcce_6279c2b5e3fe.slice/crio-c0dc99ebdaaeca8138c85a24421b7442c85e1736e8fd4436bc3418f188dd3bd0 WatchSource:0}: Error finding container c0dc99ebdaaeca8138c85a24421b7442c85e1736e8fd4436bc3418f188dd3bd0: Status 404 returned error can't find the container with id c0dc99ebdaaeca8138c85a24421b7442c85e1736e8fd4436bc3418f188dd3bd0 Oct 30 04:16:51.692301 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.692204 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:16:51.695708 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.695689 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:16:51.717075 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.717048 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:16:51.720319 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.720294 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:16:51.730269 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.730248 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:16:51.731070 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.731049 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:16:51.753135 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.749636 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:16:51.782825 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.782800 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 30 04:16:51.786917 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.786882 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:16:51.786917 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.786898 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-m9xwj\"" Oct 30 04:16:51.787099 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.787003 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:16:51.787099 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.787061 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:16:51.798950 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.798917 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z"] Oct 30 04:16:51.802423 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.802402 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" Oct 30 04:16:51.823258 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.823234 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z"] Oct 30 04:16:51.848511 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.848484 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 30 04:16:51.848698 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.848680 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-7qlx9\"" Oct 30 04:16:51.875170 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.875142 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"fc860910-960e-4e65-9d52-529517d1cb95\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:16:51.875302 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.875180 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9nfb\" (UniqueName: \"kubernetes.io/projected/fc860910-960e-4e65-9d52-529517d1cb95-kube-api-access-s9nfb\") pod \"data-science-collector-collector-1\" (UID: \"fc860910-960e-4e65-9d52-529517d1cb95\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:16:51.875302 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.875264 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fc860910-960e-4e65-9d52-529517d1cb95-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"fc860910-960e-4e65-9d52-529517d1cb95\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:16:51.875428 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.875385 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-plugins\") pod \"data-science-perses-0\" (UID: \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:16:51.875428 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.875421 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fc860910-960e-4e65-9d52-529517d1cb95\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:16:51.875518 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.875449 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-config\") pod \"data-science-perses-0\" (UID: \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:16:51.875549 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.875536 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn7zk\" (UniqueName: \"kubernetes.io/projected/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-kube-api-access-wn7zk\") pod \"data-science-perses-0\" (UID: \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:16:51.875582 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.875561 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fc860910-960e-4e65-9d52-529517d1cb95\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:16:51.882272 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.882256 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 30 04:16:51.976736 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.976665 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fc860910-960e-4e65-9d52-529517d1cb95\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:16:51.976736 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.976712 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/93ab81ec-b834-429b-b4f8-047abf18fa22-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-866546d5d5-8ft4z\" (UID: \"93ab81ec-b834-429b-b4f8-047abf18fa22\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" Oct 30 04:16:51.976942 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.976833 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"fc860910-960e-4e65-9d52-529517d1cb95\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:16:51.976942 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.976932 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9nfb\" (UniqueName: \"kubernetes.io/projected/fc860910-960e-4e65-9d52-529517d1cb95-kube-api-access-s9nfb\") pod \"data-science-collector-collector-1\" (UID: \"fc860910-960e-4e65-9d52-529517d1cb95\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:16:51.977056 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.976993 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fc860910-960e-4e65-9d52-529517d1cb95-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"fc860910-960e-4e65-9d52-529517d1cb95\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:16:51.977114 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.977060 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-plugins\") pod \"data-science-perses-0\" (UID: \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:16:51.977114 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.977091 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fc860910-960e-4e65-9d52-529517d1cb95\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:16:51.977210 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.977116 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-config\") pod \"data-science-perses-0\" (UID: \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:16:51.977210 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.977151 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/93ab81ec-b834-429b-b4f8-047abf18fa22-ta-internal\") pod \"data-science-collector-targetallocator-866546d5d5-8ft4z\" (UID: \"93ab81ec-b834-429b-b4f8-047abf18fa22\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" Oct 30 04:16:51.977210 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.977177 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8b5j\" (UniqueName: \"kubernetes.io/projected/93ab81ec-b834-429b-b4f8-047abf18fa22-kube-api-access-m8b5j\") pod \"data-science-collector-targetallocator-866546d5d5-8ft4z\" (UID: \"93ab81ec-b834-429b-b4f8-047abf18fa22\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" Oct 30 04:16:51.977367 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.977215 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn7zk\" (UniqueName: \"kubernetes.io/projected/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-kube-api-access-wn7zk\") pod \"data-science-perses-0\" (UID: \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:16:51.977675 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.977655 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-plugins\") pod \"data-science-perses-0\" (UID: \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:16:51.977943 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.977918 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fc860910-960e-4e65-9d52-529517d1cb95-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"fc860910-960e-4e65-9d52-529517d1cb95\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:16:51.978061 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.978038 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-config\") pod \"data-science-perses-0\" (UID: \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:16:51.979459 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.979423 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fc860910-960e-4e65-9d52-529517d1cb95\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:16:51.979555 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.979500 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"fc860910-960e-4e65-9d52-529517d1cb95\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:16:51.979555 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:51.979528 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fc860910-960e-4e65-9d52-529517d1cb95\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:16:52.005841 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:52.005817 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9nfb\" (UniqueName: \"kubernetes.io/projected/fc860910-960e-4e65-9d52-529517d1cb95-kube-api-access-s9nfb\") pod \"data-science-collector-collector-1\" (UID: \"fc860910-960e-4e65-9d52-529517d1cb95\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:16:52.006075 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:52.006048 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn7zk\" (UniqueName: \"kubernetes.io/projected/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-kube-api-access-wn7zk\") pod \"data-science-perses-0\" (UID: \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:16:52.008330 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:52.008312 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:16:52.012233 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:52.012200 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9" event={"ID":"13fb3a4f-721a-477f-bcce-6279c2b5e3fe","Type":"ContainerStarted","Data":"e24de3392d96cd6fc7beaf18200d2fb8ff34c4e9e89e371aee09518737609d56"} Oct 30 04:16:52.012336 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:52.012242 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9" event={"ID":"13fb3a4f-721a-477f-bcce-6279c2b5e3fe","Type":"ContainerStarted","Data":"c0dc99ebdaaeca8138c85a24421b7442c85e1736e8fd4436bc3418f188dd3bd0"} Oct 30 04:16:52.056648 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:52.056521 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9" podStartSLOduration=2.056507299 podStartE2EDuration="2.056507299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:52.056358401 +0000 UTC m=+5724.170673377" watchObservedRunningTime="2026-10-30 04:16:52.056507299 +0000 UTC m=+5724.170822273" Oct 30 04:16:52.078176 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:52.078145 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/93ab81ec-b834-429b-b4f8-047abf18fa22-ta-internal\") pod \"data-science-collector-targetallocator-866546d5d5-8ft4z\" (UID: \"93ab81ec-b834-429b-b4f8-047abf18fa22\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" Oct 30 04:16:52.078295 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:52.078184 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8b5j\" (UniqueName: \"kubernetes.io/projected/93ab81ec-b834-429b-b4f8-047abf18fa22-kube-api-access-m8b5j\") pod \"data-science-collector-targetallocator-866546d5d5-8ft4z\" (UID: \"93ab81ec-b834-429b-b4f8-047abf18fa22\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" Oct 30 04:16:52.078295 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:52.078222 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/93ab81ec-b834-429b-b4f8-047abf18fa22-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-866546d5d5-8ft4z\" (UID: \"93ab81ec-b834-429b-b4f8-047abf18fa22\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" Oct 30 04:16:52.079014 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:52.078990 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/93ab81ec-b834-429b-b4f8-047abf18fa22-ta-internal\") pod \"data-science-collector-targetallocator-866546d5d5-8ft4z\" (UID: \"93ab81ec-b834-429b-b4f8-047abf18fa22\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" Oct 30 04:16:52.080899 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:52.080881 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/93ab81ec-b834-429b-b4f8-047abf18fa22-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-866546d5d5-8ft4z\" (UID: \"93ab81ec-b834-429b-b4f8-047abf18fa22\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" Oct 30 04:16:52.104298 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:52.104269 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8b5j\" (UniqueName: \"kubernetes.io/projected/93ab81ec-b834-429b-b4f8-047abf18fa22-kube-api-access-m8b5j\") pod \"data-science-collector-targetallocator-866546d5d5-8ft4z\" (UID: \"93ab81ec-b834-429b-b4f8-047abf18fa22\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" Oct 30 04:16:52.114007 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:52.113978 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" Oct 30 04:16:52.316752 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:52.316722 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:16:52.317068 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:16:52.317048 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc860910_960e_4e65_9d52_529517d1cb95.slice/crio-84f0ec802f4bba468f3374b0c660485ec059b443ccdf18932b3bcec02d1fbf6e WatchSource:0}: Error finding container 84f0ec802f4bba468f3374b0c660485ec059b443ccdf18932b3bcec02d1fbf6e: Status 404 returned error can't find the container with id 84f0ec802f4bba468f3374b0c660485ec059b443ccdf18932b3bcec02d1fbf6e Oct 30 04:16:52.319403 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:52.319372 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:16:52.319489 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:52.319431 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:16:52.405525 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:52.405493 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z"] Oct 30 04:16:52.406748 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:16:52.406718 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93ab81ec_b834_429b_b4f8_047abf18fa22.slice/crio-796a512750e1c58ef3e116a6a2716817f0bf86eb5949122ebae7db1348b54d8f WatchSource:0}: Error finding container 796a512750e1c58ef3e116a6a2716817f0bf86eb5949122ebae7db1348b54d8f: Status 404 returned error can't find the container with id 796a512750e1c58ef3e116a6a2716817f0bf86eb5949122ebae7db1348b54d8f Oct 30 04:16:52.409010 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:52.408981 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:16:52.409090 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:52.409038 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:16:53.018828 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:53.018787 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fc860910-960e-4e65-9d52-529517d1cb95","Type":"ContainerStarted","Data":"f679b9e9b7668632171f2840f1c506ab8ada39e6a8a0aceaa0d71a278494588b"} Oct 30 04:16:53.019254 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:53.018836 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fc860910-960e-4e65-9d52-529517d1cb95","Type":"ContainerStarted","Data":"84f0ec802f4bba468f3374b0c660485ec059b443ccdf18932b3bcec02d1fbf6e"} Oct 30 04:16:53.020496 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:53.020473 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" event={"ID":"93ab81ec-b834-429b-b4f8-047abf18fa22","Type":"ContainerStarted","Data":"25333cbe79691dca0fcbbaeaf38f1220624ac69355712fce43f73bec966db0db"} Oct 30 04:16:53.020596 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:53.020505 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" event={"ID":"93ab81ec-b834-429b-b4f8-047abf18fa22","Type":"ContainerStarted","Data":"796a512750e1c58ef3e116a6a2716817f0bf86eb5949122ebae7db1348b54d8f"} Oct 30 04:16:53.020678 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:53.020657 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" Oct 30 04:16:53.022387 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:53.022369 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" Oct 30 04:16:53.068089 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:53.068039 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.068026268 podStartE2EDuration="2.068026268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:53.064189006 +0000 UTC m=+5725.178503982" watchObservedRunningTime="2026-10-30 04:16:53.068026268 +0000 UTC m=+5725.182341242" Oct 30 04:16:53.109892 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:53.109833 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" podStartSLOduration=2.109815528 podStartE2EDuration="2.109815528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:53.108710233 +0000 UTC m=+5725.223025208" watchObservedRunningTime="2026-10-30 04:16:53.109815528 +0000 UTC m=+5725.224130503" Oct 30 04:16:53.942851 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:16:53.942812 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:16:55.298269 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.298238 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:16:55.302108 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.302087 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.319696 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.319671 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 04:16:55.319696 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.319678 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 04:16:55.319870 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.319706 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:16:55.319870 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.319760 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 04:16:55.333077 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.333051 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-5vht9\"" Oct 30 04:16:55.342902 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.342874 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:16:55.360544 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.360506 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:16:55.360726 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.360693 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.375184 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.375158 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:16:55.409929 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.409887 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kx72\" (UniqueName: \"kubernetes.io/projected/4ef93eb8-40a1-4388-873f-e363c354abee-kube-api-access-6kx72\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.410086 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.409940 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.410171 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.410074 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.410171 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.410118 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.410171 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.410150 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4ef93eb8-40a1-4388-873f-e363c354abee-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.410313 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.410267 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4ef93eb8-40a1-4388-873f-e363c354abee-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.410359 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.410309 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4ef93eb8-40a1-4388-873f-e363c354abee-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.419522 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.419494 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 04:16:55.419522 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.419506 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 04:16:55.419786 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.419532 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 04:16:55.419786 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.419772 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:16:55.419786 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.419777 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 04:16:55.419952 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.419793 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-vt8ws\"" Oct 30 04:16:55.419952 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.419823 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 04:16:55.429294 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.429264 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9"] Oct 30 04:16:55.429520 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.429492 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9" podUID="13fb3a4f-721a-477f-bcce-6279c2b5e3fe" containerName="thanos-querier" containerID="cri-o://e24de3392d96cd6fc7beaf18200d2fb8ff34c4e9e89e371aee09518737609d56" gracePeriod=30 Oct 30 04:16:55.433138 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.433114 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 04:16:55.433235 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.433162 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 04:16:55.433287 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.433235 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 04:16:55.511785 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.511751 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.511942 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.511791 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kx72\" (UniqueName: \"kubernetes.io/projected/4ef93eb8-40a1-4388-873f-e363c354abee-kube-api-access-6kx72\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.511942 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.511815 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.511942 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.511913 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.512049 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.511953 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.512084 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.512051 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.512165 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.512139 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.512303 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.512185 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.512303 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.512278 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6c6l\" (UniqueName: \"kubernetes.io/projected/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-kube-api-access-r6c6l\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.512403 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.512319 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.512403 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.512370 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.512403 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.512397 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.512535 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.512417 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4ef93eb8-40a1-4388-873f-e363c354abee-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.512535 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.512454 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.512535 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.512485 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.512736 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.512565 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.512736 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.512589 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.512736 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.512610 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4ef93eb8-40a1-4388-873f-e363c354abee-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.512736 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.512720 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4ef93eb8-40a1-4388-873f-e363c354abee-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.513150 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.513106 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4ef93eb8-40a1-4388-873f-e363c354abee-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.514559 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.514531 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.514935 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.514906 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.515024 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.514996 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4ef93eb8-40a1-4388-873f-e363c354abee-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.515024 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.515004 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4ef93eb8-40a1-4388-873f-e363c354abee-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.515274 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.515259 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.537201 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.537170 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kx72\" (UniqueName: \"kubernetes.io/projected/4ef93eb8-40a1-4388-873f-e363c354abee-kube-api-access-6kx72\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.614174 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.614141 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.614338 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.614184 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.614338 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.614206 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.614338 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.614231 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.614338 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.614256 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.614338 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.614288 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.614338 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.614325 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6c6l\" (UniqueName: \"kubernetes.io/projected/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-kube-api-access-r6c6l\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.614681 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.614451 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.614681 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.614579 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.614681 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.614659 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.614845 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.614700 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.614845 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.614757 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.614962 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.614892 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.615464 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.615419 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.616958 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.616922 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.617360 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.617333 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.618271 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.618240 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.618368 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.618292 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.619064 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.619038 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.619211 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.619186 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:16:55.619800 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.619772 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.619915 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.619876 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.619977 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.619910 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.620402 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.620380 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.643505 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.643463 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6c6l\" (UniqueName: \"kubernetes.io/projected/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-kube-api-access-r6c6l\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:55.665579 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.665555 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9" Oct 30 04:16:55.715837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.715809 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13fb3a4f-721a-477f-bcce-6279c2b5e3fe-kube-api-access-79bcb\" (UniqueName: \"kubernetes.io/projected/13fb3a4f-721a-477f-bcce-6279c2b5e3fe-kube-api-access-79bcb\") pod \"13fb3a4f-721a-477f-bcce-6279c2b5e3fe\" (UID: \"13fb3a4f-721a-477f-bcce-6279c2b5e3fe\") " Oct 30 04:16:55.717864 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.717838 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13fb3a4f-721a-477f-bcce-6279c2b5e3fe-kube-api-access-79bcb" pod "13fb3a4f-721a-477f-bcce-6279c2b5e3fe" (UID: "13fb3a4f-721a-477f-bcce-6279c2b5e3fe"). InnerVolumeSpecName "kube-api-access-79bcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:16:55.816702 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.816670 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79bcb\" (UniqueName: \"kubernetes.io/projected/13fb3a4f-721a-477f-bcce-6279c2b5e3fe-kube-api-access-79bcb\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:16:55.928591 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:55.928559 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:16:55.928834 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:16:55.928810 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef93eb8_40a1_4388_873f_e363c354abee.slice/crio-7bb089bc15ef7db93a4332c22d8dc65a48a4484d01d3988e9d8cac2d7100f3bd WatchSource:0}: Error finding container 7bb089bc15ef7db93a4332c22d8dc65a48a4484d01d3988e9d8cac2d7100f3bd: Status 404 returned error can't find the container with id 7bb089bc15ef7db93a4332c22d8dc65a48a4484d01d3988e9d8cac2d7100f3bd Oct 30 04:16:56.034833 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:56.034802 5882 generic.go:356] "Generic (PLEG): container finished" podID="13fb3a4f-721a-477f-bcce-6279c2b5e3fe" containerID="e24de3392d96cd6fc7beaf18200d2fb8ff34c4e9e89e371aee09518737609d56" exitCode=0 Oct 30 04:16:56.034998 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:56.034856 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9" Oct 30 04:16:56.034998 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:56.034889 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9" event={"ID":"13fb3a4f-721a-477f-bcce-6279c2b5e3fe","Type":"ContainerDied","Data":"e24de3392d96cd6fc7beaf18200d2fb8ff34c4e9e89e371aee09518737609d56"} Oct 30 04:16:56.034998 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:56.034921 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9" event={"ID":"13fb3a4f-721a-477f-bcce-6279c2b5e3fe","Type":"ContainerDied","Data":"c0dc99ebdaaeca8138c85a24421b7442c85e1736e8fd4436bc3418f188dd3bd0"} Oct 30 04:16:56.034998 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:56.034947 5882 scope.go:122] "RemoveContainer" containerID="e24de3392d96cd6fc7beaf18200d2fb8ff34c4e9e89e371aee09518737609d56" Oct 30 04:16:56.036295 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:56.036268 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"4ef93eb8-40a1-4388-873f-e363c354abee","Type":"ContainerStarted","Data":"7bb089bc15ef7db93a4332c22d8dc65a48a4484d01d3988e9d8cac2d7100f3bd"} Oct 30 04:16:56.042924 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:56.042904 5882 scope.go:122] "RemoveContainer" containerID="e24de3392d96cd6fc7beaf18200d2fb8ff34c4e9e89e371aee09518737609d56" Oct 30 04:16:56.043207 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:16:56.043189 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24de3392d96cd6fc7beaf18200d2fb8ff34c4e9e89e371aee09518737609d56\": container with ID starting with e24de3392d96cd6fc7beaf18200d2fb8ff34c4e9e89e371aee09518737609d56 not found: ID does not exist" containerID="e24de3392d96cd6fc7beaf18200d2fb8ff34c4e9e89e371aee09518737609d56" Oct 30 04:16:56.043254 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:56.043217 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24de3392d96cd6fc7beaf18200d2fb8ff34c4e9e89e371aee09518737609d56"} err="failed to get container status \"e24de3392d96cd6fc7beaf18200d2fb8ff34c4e9e89e371aee09518737609d56\": rpc error: code = NotFound desc = could not find container \"e24de3392d96cd6fc7beaf18200d2fb8ff34c4e9e89e371aee09518737609d56\": container with ID starting with e24de3392d96cd6fc7beaf18200d2fb8ff34c4e9e89e371aee09518737609d56 not found: ID does not exist" Oct 30 04:16:56.090396 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:56.090363 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9"] Oct 30 04:16:56.094560 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:56.094534 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-d9pp9"] Oct 30 04:16:57.042101 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:57.042071 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"4ef93eb8-40a1-4388-873f-e363c354abee","Type":"ContainerStarted","Data":"0a65e7fadf28c3aacff6c0b1181d436fc836ca3a40f692b429e59605fbdf4f77"} Oct 30 04:16:58.009464 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:58.009431 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13fb3a4f-721a-477f-bcce-6279c2b5e3fe" path="/var/lib/kubelet/pods/13fb3a4f-721a-477f-bcce-6279c2b5e3fe/volumes" Oct 30 04:16:58.443482 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:58.443405 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:16:58.475893 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:58.475862 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:16:58.544449 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:58.544409 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:16:58.544652 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:58.544544 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:16:58.560953 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:58.560925 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\") DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:16:58.665355 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:58.665328 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2c422b71c1e0b8fdf34659f2cef7846e06b4353ed88027dc99ade5d7fc29e9d6/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:16:58.699519 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:58.699454 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:16:58.932315 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:58.932276 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:16:59.049319 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:59.049279 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:59.081500 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:59.081471 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:59.150073 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:59.150030 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:59.150273 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:59.150190 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:59.166359 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:59.166325 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") DevicePath \"csi-927d18ef2eb99137c68fbe5e0bb8ea629bb5219ed56cdede72da7c0f045be95e\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:59.226986 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:59.226955 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:16:59.227416 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:16:59.227391 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07adcc73_ec51_42e3_bf3d_115b9cbc34fd.slice/crio-8c7d1d81834bc65a09f1bc07b5f02c3e15fbef824ed2f1b1448d78b9b7f1506a WatchSource:0}: Error finding container 8c7d1d81834bc65a09f1bc07b5f02c3e15fbef824ed2f1b1448d78b9b7f1506a: Status 404 returned error can't find the container with id 8c7d1d81834bc65a09f1bc07b5f02c3e15fbef824ed2f1b1448d78b9b7f1506a Oct 30 04:16:59.270227 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:59.270199 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a535375d22e374ba333968465471c7604a06f9d95a513910fe4bb083b629e986/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:59.301673 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:59.301637 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:59.583391 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:59.583353 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:16:59.913475 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:16:59.913443 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae29c8fd_02d3_40c0_9520_3c2a0a2f88e6.slice/crio-6769dd53fcd3758493f69b382be998d7ac188816be1d40e868cb7fd074a32a59 WatchSource:0}: Error finding container 6769dd53fcd3758493f69b382be998d7ac188816be1d40e868cb7fd074a32a59: Status 404 returned error can't find the container with id 6769dd53fcd3758493f69b382be998d7ac188816be1d40e868cb7fd074a32a59 Oct 30 04:16:59.924922 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:16:59.924895 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:17:00.058502 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:00.058464 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6","Type":"ContainerStarted","Data":"6769dd53fcd3758493f69b382be998d7ac188816be1d40e868cb7fd074a32a59"} Oct 30 04:17:00.060027 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:00.059990 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"07adcc73-ec51-42e3-bf3d-115b9cbc34fd","Type":"ContainerStarted","Data":"3552a0ee9d2c25d5ec1d317a30fd3ccc2aca5f34520ddf1dc914c76e97dbe08f"} Oct 30 04:17:00.060170 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:00.060036 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"07adcc73-ec51-42e3-bf3d-115b9cbc34fd","Type":"ContainerStarted","Data":"8c7d1d81834bc65a09f1bc07b5f02c3e15fbef824ed2f1b1448d78b9b7f1506a"} Oct 30 04:17:00.060253 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:00.060184 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:17:00.061924 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:00.061901 5882 generic.go:356] "Generic (PLEG): container finished" podID="4ef93eb8-40a1-4388-873f-e363c354abee" containerID="0a65e7fadf28c3aacff6c0b1181d436fc836ca3a40f692b429e59605fbdf4f77" exitCode=0 Oct 30 04:17:00.062042 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:00.061944 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"4ef93eb8-40a1-4388-873f-e363c354abee","Type":"ContainerDied","Data":"0a65e7fadf28c3aacff6c0b1181d436fc836ca3a40f692b429e59605fbdf4f77"} Oct 30 04:17:00.105840 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:00.105783 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=9.105764714 podStartE2EDuration="9.105764714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:00.104929507 +0000 UTC m=+5732.219244482" watchObservedRunningTime="2026-10-30 04:17:00.105764714 +0000 UTC m=+5732.220079690" Oct 30 04:17:01.069789 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:01.069750 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"4ef93eb8-40a1-4388-873f-e363c354abee","Type":"ContainerStarted","Data":"8a5854175ea2517a00d169b5409651a286660dd6bd7ab2b4d52a69e559a5c4ed"} Oct 30 04:17:01.070272 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:01.069824 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"4ef93eb8-40a1-4388-873f-e363c354abee","Type":"ContainerStarted","Data":"852b203ab59f57407551343c61f931115b8548a67054b27574fe9cfdd2f3f7ce"} Oct 30 04:17:01.070272 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:01.070060 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:17:01.071467 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:01.071440 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6","Type":"ContainerStarted","Data":"c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122"} Oct 30 04:17:01.121493 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:01.121435 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=2.933173203 podStartE2EDuration="6.121419797s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.188246594s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:01.120278763 +0000 UTC m=+5733.234593762" watchObservedRunningTime="2026-10-30 04:17:01.121419797 +0000 UTC m=+5733.235734768" Oct 30 04:17:05.091108 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:05.091070 5882 generic.go:356] "Generic (PLEG): container finished" podID="ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" containerID="c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122" exitCode=0 Oct 30 04:17:05.091476 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:05.091130 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6","Type":"ContainerDied","Data":"c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122"} Oct 30 04:17:06.098395 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:06.098360 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6","Type":"ContainerStarted","Data":"fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c"} Oct 30 04:17:06.098779 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:06.098431 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6","Type":"ContainerStarted","Data":"02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f"} Oct 30 04:17:06.098779 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:06.098458 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6","Type":"ContainerStarted","Data":"bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc"} Oct 30 04:17:06.174488 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:06.174437 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=7.48627996 podStartE2EDuration="11.174424951s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.688144991s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:06.173804787 +0000 UTC m=+5738.288119773" watchObservedRunningTime="2026-10-30 04:17:06.174424951 +0000 UTC m=+5738.288739917" Oct 30 04:17:07.077594 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:07.077560 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:17:09.584514 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:09.584475 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:11.074858 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:11.074826 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:17:14.583556 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:14.583509 5882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:14.588875 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:14.588855 5882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:15.142452 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:15.142424 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:23.942822 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:17:23.942783 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:17:26.550018 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.549979 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:17:26.550425 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.550253 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" containerName="prometheus" containerID="cri-o://bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc" gracePeriod=600 Oct 30 04:17:26.550425 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.550328 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" containerName="config-reloader" containerID="cri-o://02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f" gracePeriod=600 Oct 30 04:17:26.550547 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.550308 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" containerName="thanos-sidecar" containerID="cri-o://fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c" gracePeriod=600 Oct 30 04:17:26.793021 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.792994 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:26.911675 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.911583 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " Oct 30 04:17:26.911827 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.911726 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " Oct 30 04:17:26.911827 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.911760 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config\") pod \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " Oct 30 04:17:26.911827 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.911795 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-config\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-config\") pod \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " Oct 30 04:17:26.912009 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.911837 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " Oct 30 04:17:26.912009 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.911882 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " Oct 30 04:17:26.912009 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.911911 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " Oct 30 04:17:26.912009 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.911939 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-tls-assets\" (UniqueName: \"kubernetes.io/projected/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-tls-assets\") pod \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " Oct 30 04:17:26.912009 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.911970 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-config-out\") pod \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " Oct 30 04:17:26.912274 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.912014 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " Oct 30 04:17:26.912274 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.912039 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-kube-api-access-r6c6l\" (UniqueName: \"kubernetes.io/projected/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-kube-api-access-r6c6l\") pod \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " Oct 30 04:17:26.912274 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.912037 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-0" pod "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" (UID: "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:26.912274 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.912075 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-secret-prometheus-web-tls-ca\") pod \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " Oct 30 04:17:26.912274 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.912114 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-thanos-prometheus-http-client-file\") pod \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\" (UID: \"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6\") " Oct 30 04:17:26.912274 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.912261 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-1" pod "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" (UID: "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:26.912590 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.912436 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:26.912590 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.912440 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-2" pod "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" (UID: "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:26.912590 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.912460 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:26.914453 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.914413 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-config" pod "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" (UID: "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:17:26.914594 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.914468 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-config-out" pod "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" (UID: "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:17:26.915409 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.915375 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" (UID: "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:17:26.915570 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.915549 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-kube-api-access-r6c6l" pod "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" (UID: "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6"). InnerVolumeSpecName "kube-api-access-r6c6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:26.915835 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.915814 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-thanos-prometheus-http-client-file" pod "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" (UID: "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:17:26.915917 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.915814 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-tls-assets" pod "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" (UID: "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:26.916161 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.916143 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-secret-prometheus-web-tls-ca" pod "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" (UID: "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:17:26.916222 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.916166 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" (UID: "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:17:26.917657 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.917605 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4" pod "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" (UID: "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6"). InnerVolumeSpecName "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:17:26.926946 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:26.926898 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config" pod "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" (UID: "ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:17:27.013712 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.013677 5882 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:27.013712 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.013707 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:27.013712 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.013718 5882 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-tls-assets\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:27.013946 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.013726 5882 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-config-out\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:27.013946 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.013736 5882 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:27.013946 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.013746 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6c6l\" (UniqueName: \"kubernetes.io/projected/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-kube-api-access-r6c6l\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:27.013946 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.013755 5882 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-secret-prometheus-web-tls-ca\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:27.013946 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.013764 5882 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:27.013946 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.013791 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:17:27.013946 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.013800 5882 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-web-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:27.013946 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.013808 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:27.024961 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.024942 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:17:27.029258 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.029234 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:17:27.029436 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.029418 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="fc860910-960e-4e65-9d52-529517d1cb95" containerName="otc-container" containerID="cri-o://f679b9e9b7668632171f2840f1c506ab8ada39e6a8a0aceaa0d71a278494588b" gracePeriod=30 Oct 30 04:17:27.114704 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.114672 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-927d18ef2eb99137c68fbe5e0bb8ea629bb5219ed56cdede72da7c0f045be95e\"" Oct 30 04:17:27.191678 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.191577 5882 generic.go:356] "Generic (PLEG): container finished" podID="ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" containerID="fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c" exitCode=0 Oct 30 04:17:27.191678 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.191609 5882 generic.go:356] "Generic (PLEG): container finished" podID="ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" containerID="02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f" exitCode=0 Oct 30 04:17:27.191678 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.191637 5882 generic.go:356] "Generic (PLEG): container finished" podID="ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" containerID="bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc" exitCode=0 Oct 30 04:17:27.191678 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.191665 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6","Type":"ContainerDied","Data":"fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c"} Oct 30 04:17:27.191918 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.191677 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:27.191918 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.191733 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6","Type":"ContainerDied","Data":"02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f"} Oct 30 04:17:27.191918 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.191761 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6","Type":"ContainerDied","Data":"bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc"} Oct 30 04:17:27.191918 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.191784 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6","Type":"ContainerDied","Data":"6769dd53fcd3758493f69b382be998d7ac188816be1d40e868cb7fd074a32a59"} Oct 30 04:17:27.191918 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.191812 5882 scope.go:122] "RemoveContainer" containerID="fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c" Oct 30 04:17:27.200478 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.200458 5882 scope.go:122] "RemoveContainer" containerID="02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f" Oct 30 04:17:27.208237 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.208212 5882 scope.go:122] "RemoveContainer" containerID="bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc" Oct 30 04:17:27.216051 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.216033 5882 scope.go:122] "RemoveContainer" containerID="c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122" Oct 30 04:17:27.223586 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.223569 5882 scope.go:122] "RemoveContainer" containerID="fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c" Oct 30 04:17:27.223893 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:17:27.223875 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c\": container with ID starting with fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c not found: ID does not exist" containerID="fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c" Oct 30 04:17:27.223957 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.223901 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c"} err="failed to get container status \"fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c\": rpc error: code = NotFound desc = could not find container \"fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c\": container with ID starting with fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c not found: ID does not exist" Oct 30 04:17:27.223957 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.223918 5882 scope.go:122] "RemoveContainer" containerID="02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f" Oct 30 04:17:27.224144 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:17:27.224124 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f\": container with ID starting with 02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f not found: ID does not exist" containerID="02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f" Oct 30 04:17:27.224185 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.224150 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f"} err="failed to get container status \"02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f\": rpc error: code = NotFound desc = could not find container \"02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f\": container with ID starting with 02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f not found: ID does not exist" Oct 30 04:17:27.224185 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.224166 5882 scope.go:122] "RemoveContainer" containerID="bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc" Oct 30 04:17:27.224400 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:17:27.224383 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc\": container with ID starting with bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc not found: ID does not exist" containerID="bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc" Oct 30 04:17:27.224437 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.224404 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc"} err="failed to get container status \"bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc\": rpc error: code = NotFound desc = could not find container \"bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc\": container with ID starting with bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc not found: ID does not exist" Oct 30 04:17:27.224437 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.224418 5882 scope.go:122] "RemoveContainer" containerID="c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122" Oct 30 04:17:27.224635 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:17:27.224609 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122\": container with ID starting with c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122 not found: ID does not exist" containerID="c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122" Oct 30 04:17:27.224682 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.224643 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122"} err="failed to get container status \"c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122\": rpc error: code = NotFound desc = could not find container \"c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122\": container with ID starting with c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122 not found: ID does not exist" Oct 30 04:17:27.224682 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.224660 5882 scope.go:122] "RemoveContainer" containerID="fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c" Oct 30 04:17:27.224857 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.224841 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c"} err="failed to get container status \"fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c\": rpc error: code = NotFound desc = could not find container \"fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c\": container with ID starting with fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c not found: ID does not exist" Oct 30 04:17:27.224899 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.224857 5882 scope.go:122] "RemoveContainer" containerID="02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f" Oct 30 04:17:27.225040 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.225023 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f"} err="failed to get container status \"02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f\": rpc error: code = NotFound desc = could not find container \"02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f\": container with ID starting with 02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f not found: ID does not exist" Oct 30 04:17:27.225085 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.225040 5882 scope.go:122] "RemoveContainer" containerID="bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc" Oct 30 04:17:27.225224 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.225206 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc"} err="failed to get container status \"bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc\": rpc error: code = NotFound desc = could not find container \"bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc\": container with ID starting with bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc not found: ID does not exist" Oct 30 04:17:27.225267 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.225225 5882 scope.go:122] "RemoveContainer" containerID="c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122" Oct 30 04:17:27.225405 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.225389 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122"} err="failed to get container status \"c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122\": rpc error: code = NotFound desc = could not find container \"c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122\": container with ID starting with c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122 not found: ID does not exist" Oct 30 04:17:27.225444 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.225405 5882 scope.go:122] "RemoveContainer" containerID="fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c" Oct 30 04:17:27.225587 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.225572 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c"} err="failed to get container status \"fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c\": rpc error: code = NotFound desc = could not find container \"fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c\": container with ID starting with fb025e0fe8729f6a60ae7c2cc70326129f702bdc55ff74def9127879f80a123c not found: ID does not exist" Oct 30 04:17:27.225649 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.225587 5882 scope.go:122] "RemoveContainer" containerID="02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f" Oct 30 04:17:27.225787 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.225772 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f"} err="failed to get container status \"02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f\": rpc error: code = NotFound desc = could not find container \"02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f\": container with ID starting with 02f56bac9d910de7fc52a20ccec3fbb78ec9bfe9fb9ce3cea636771a9113906f not found: ID does not exist" Oct 30 04:17:27.225826 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.225787 5882 scope.go:122] "RemoveContainer" containerID="bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc" Oct 30 04:17:27.225964 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.225949 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc"} err="failed to get container status \"bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc\": rpc error: code = NotFound desc = could not find container \"bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc\": container with ID starting with bd0d2112a0a761b9b6efc3ba6a32b24064dd3c6e61fe3eb0bf6f0afbe5e794dc not found: ID does not exist" Oct 30 04:17:27.226009 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.225964 5882 scope.go:122] "RemoveContainer" containerID="c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122" Oct 30 04:17:27.226133 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.226118 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122"} err="failed to get container status \"c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122\": rpc error: code = NotFound desc = could not find container \"c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122\": container with ID starting with c00ca712d280f09db0ddb2fbaf9216a00b269edbdff04dc20aaa1d7cf1b07122 not found: ID does not exist" Oct 30 04:17:27.253493 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.253468 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:17:27.260586 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.260565 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:17:27.279174 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.279152 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4"] Oct 30 04:17:27.279540 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.279526 5882 cpu_manager.go:414] "containerMap: removing container" podUID="13fb3a4f-721a-477f-bcce-6279c2b5e3fe" containerName="thanos-querier" Oct 30 04:17:27.279584 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.279542 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13fb3a4f-721a-477f-bcce-6279c2b5e3fe" containerName="thanos-querier" Oct 30 04:17:27.279584 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.279557 5882 cpu_manager.go:414] "containerMap: removing container" podUID="ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" containerName="prometheus" Oct 30 04:17:27.279584 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.279564 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" containerName="prometheus" Oct 30 04:17:27.279584 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.279573 5882 cpu_manager.go:414] "containerMap: removing container" podUID="ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" containerName="config-reloader" Oct 30 04:17:27.279584 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.279578 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" containerName="config-reloader" Oct 30 04:17:27.279756 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.279594 5882 cpu_manager.go:414] "containerMap: removing container" podUID="ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" containerName="thanos-sidecar" Oct 30 04:17:27.279756 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.279599 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" containerName="thanos-sidecar" Oct 30 04:17:27.279756 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.279607 5882 cpu_manager.go:414] "containerMap: removing container" podUID="ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" containerName="init-config-reloader" Oct 30 04:17:27.279756 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.279611 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" containerName="init-config-reloader" Oct 30 04:17:27.279756 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.279680 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" containerName="thanos-sidecar" Oct 30 04:17:27.279756 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.279693 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="13fb3a4f-721a-477f-bcce-6279c2b5e3fe" containerName="thanos-querier" Oct 30 04:17:27.279756 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.279700 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" containerName="prometheus" Oct 30 04:17:27.279756 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.279707 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" containerName="config-reloader" Oct 30 04:17:27.284261 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.284244 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" Oct 30 04:17:27.303156 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.303133 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4"] Oct 30 04:17:27.417173 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.417142 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/247b0ad6-83ad-447c-bab6-3c4e1aa32956-ta-internal\") pod \"data-science-collector-targetallocator-74687f45b8-nb2v4\" (UID: \"247b0ad6-83ad-447c-bab6-3c4e1aa32956\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" Oct 30 04:17:27.417371 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.417207 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/247b0ad6-83ad-447c-bab6-3c4e1aa32956-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-74687f45b8-nb2v4\" (UID: \"247b0ad6-83ad-447c-bab6-3c4e1aa32956\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" Oct 30 04:17:27.417371 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.417304 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zzgx\" (UniqueName: \"kubernetes.io/projected/247b0ad6-83ad-447c-bab6-3c4e1aa32956-kube-api-access-5zzgx\") pod \"data-science-collector-targetallocator-74687f45b8-nb2v4\" (UID: \"247b0ad6-83ad-447c-bab6-3c4e1aa32956\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" Oct 30 04:17:27.518485 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.518454 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/247b0ad6-83ad-447c-bab6-3c4e1aa32956-ta-internal\") pod \"data-science-collector-targetallocator-74687f45b8-nb2v4\" (UID: \"247b0ad6-83ad-447c-bab6-3c4e1aa32956\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" Oct 30 04:17:27.518691 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.518521 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/247b0ad6-83ad-447c-bab6-3c4e1aa32956-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-74687f45b8-nb2v4\" (UID: \"247b0ad6-83ad-447c-bab6-3c4e1aa32956\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" Oct 30 04:17:27.518691 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.518544 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zzgx\" (UniqueName: \"kubernetes.io/projected/247b0ad6-83ad-447c-bab6-3c4e1aa32956-kube-api-access-5zzgx\") pod \"data-science-collector-targetallocator-74687f45b8-nb2v4\" (UID: \"247b0ad6-83ad-447c-bab6-3c4e1aa32956\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" Oct 30 04:17:27.519222 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.519194 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/247b0ad6-83ad-447c-bab6-3c4e1aa32956-ta-internal\") pod \"data-science-collector-targetallocator-74687f45b8-nb2v4\" (UID: \"247b0ad6-83ad-447c-bab6-3c4e1aa32956\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" Oct 30 04:17:27.520937 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.520916 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/247b0ad6-83ad-447c-bab6-3c4e1aa32956-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-74687f45b8-nb2v4\" (UID: \"247b0ad6-83ad-447c-bab6-3c4e1aa32956\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" Oct 30 04:17:27.543811 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.543783 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zzgx\" (UniqueName: \"kubernetes.io/projected/247b0ad6-83ad-447c-bab6-3c4e1aa32956-kube-api-access-5zzgx\") pod \"data-science-collector-targetallocator-74687f45b8-nb2v4\" (UID: \"247b0ad6-83ad-447c-bab6-3c4e1aa32956\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" Oct 30 04:17:27.595391 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.595360 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" Oct 30 04:17:27.886538 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.886499 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4"] Oct 30 04:17:27.887120 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:17:27.887086 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod247b0ad6_83ad_447c_bab6_3c4e1aa32956.slice/crio-f20a18b2c9c1a1a4172aed5e9db84a3939e08348a0b33e9ca2711c6e2981a4c3 WatchSource:0}: Error finding container f20a18b2c9c1a1a4172aed5e9db84a3939e08348a0b33e9ca2711c6e2981a4c3: Status 404 returned error can't find the container with id f20a18b2c9c1a1a4172aed5e9db84a3939e08348a0b33e9ca2711c6e2981a4c3 Oct 30 04:17:27.889321 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.889279 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:17:27.889391 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:27.889344 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:17:28.010376 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:28.010335 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6" path="/var/lib/kubelet/pods/ae29c8fd-02d3-40c0-9520-3c2a0a2f88e6/volumes" Oct 30 04:17:28.198713 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:28.198610 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" event={"ID":"247b0ad6-83ad-447c-bab6-3c4e1aa32956","Type":"ContainerStarted","Data":"5ea1b52a4f7e356d56c54629deea49d6b87bd6c8b2feb11e88310be6168394d5"} Oct 30 04:17:28.198713 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:28.198667 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" event={"ID":"247b0ad6-83ad-447c-bab6-3c4e1aa32956","Type":"ContainerStarted","Data":"f20a18b2c9c1a1a4172aed5e9db84a3939e08348a0b33e9ca2711c6e2981a4c3"} Oct 30 04:17:28.199075 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:28.198774 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" Oct 30 04:17:28.199851 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:28.199822 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" podUID="247b0ad6-83ad-447c-bab6-3c4e1aa32956" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.156:8080/readyz\": dial tcp 10.134.0.156:8080: connect: connection refused" Oct 30 04:17:28.243126 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:28.243069 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" podStartSLOduration=1.2430556130000001 podStartE2EDuration="1.243055613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:28.242703779 +0000 UTC m=+5760.357018753" watchObservedRunningTime="2026-10-30 04:17:28.243055613 +0000 UTC m=+5760.357370584" Oct 30 04:17:29.203752 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.203729 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" Oct 30 04:17:29.296059 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.296026 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z"] Oct 30 04:17:29.296297 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.296272 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" podUID="93ab81ec-b834-429b-b4f8-047abf18fa22" containerName="ta-container" containerID="cri-o://25333cbe79691dca0fcbbaeaf38f1220624ac69355712fce43f73bec966db0db" gracePeriod=30 Oct 30 04:17:29.428276 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.428225 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:29.530980 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.530955 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" Oct 30 04:17:29.538278 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.538258 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc860910-960e-4e65-9d52-529517d1cb95-kube-api-access-s9nfb\" (UniqueName: \"kubernetes.io/projected/fc860910-960e-4e65-9d52-529517d1cb95-kube-api-access-s9nfb\") pod \"fc860910-960e-4e65-9d52-529517d1cb95\" (UID: \"fc860910-960e-4e65-9d52-529517d1cb95\") " Oct 30 04:17:29.538366 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.538302 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-tls-certs\") pod \"fc860910-960e-4e65-9d52-529517d1cb95\" (UID: \"fc860910-960e-4e65-9d52-529517d1cb95\") " Oct 30 04:17:29.538366 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.538335 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-data-science-collector-ta-client-cert\") pod \"fc860910-960e-4e65-9d52-529517d1cb95\" (UID: \"fc860910-960e-4e65-9d52-529517d1cb95\") " Oct 30 04:17:29.538445 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.538372 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-monitoring-tls-certs\") pod \"fc860910-960e-4e65-9d52-529517d1cb95\" (UID: \"fc860910-960e-4e65-9d52-529517d1cb95\") " Oct 30 04:17:29.538445 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.538420 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc860910-960e-4e65-9d52-529517d1cb95-otc-internal\" (UniqueName: \"kubernetes.io/configmap/fc860910-960e-4e65-9d52-529517d1cb95-otc-internal\") pod \"fc860910-960e-4e65-9d52-529517d1cb95\" (UID: \"fc860910-960e-4e65-9d52-529517d1cb95\") " Oct 30 04:17:29.538939 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.538899 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc860910-960e-4e65-9d52-529517d1cb95-otc-internal" pod "fc860910-960e-4e65-9d52-529517d1cb95" (UID: "fc860910-960e-4e65-9d52-529517d1cb95"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:29.540438 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.540410 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-monitoring-tls-certs" pod "fc860910-960e-4e65-9d52-529517d1cb95" (UID: "fc860910-960e-4e65-9d52-529517d1cb95"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:17:29.540534 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.540469 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-tls-certs" pod "fc860910-960e-4e65-9d52-529517d1cb95" (UID: "fc860910-960e-4e65-9d52-529517d1cb95"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:17:29.540534 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.540489 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-data-science-collector-ta-client-cert" pod "fc860910-960e-4e65-9d52-529517d1cb95" (UID: "fc860910-960e-4e65-9d52-529517d1cb95"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:17:29.540534 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.540470 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc860910-960e-4e65-9d52-529517d1cb95-kube-api-access-s9nfb" pod "fc860910-960e-4e65-9d52-529517d1cb95" (UID: "fc860910-960e-4e65-9d52-529517d1cb95"). InnerVolumeSpecName "kube-api-access-s9nfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:29.639525 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.639493 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93ab81ec-b834-429b-b4f8-047abf18fa22-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/93ab81ec-b834-429b-b4f8-047abf18fa22-data-science-collector-ta-server-cert\") pod \"93ab81ec-b834-429b-b4f8-047abf18fa22\" (UID: \"93ab81ec-b834-429b-b4f8-047abf18fa22\") " Oct 30 04:17:29.639698 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.639545 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ab81ec-b834-429b-b4f8-047abf18fa22-kube-api-access-m8b5j\" (UniqueName: \"kubernetes.io/projected/93ab81ec-b834-429b-b4f8-047abf18fa22-kube-api-access-m8b5j\") pod \"93ab81ec-b834-429b-b4f8-047abf18fa22\" (UID: \"93ab81ec-b834-429b-b4f8-047abf18fa22\") " Oct 30 04:17:29.639698 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.639582 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93ab81ec-b834-429b-b4f8-047abf18fa22-ta-internal\" (UniqueName: \"kubernetes.io/configmap/93ab81ec-b834-429b-b4f8-047abf18fa22-ta-internal\") pod \"93ab81ec-b834-429b-b4f8-047abf18fa22\" (UID: \"93ab81ec-b834-429b-b4f8-047abf18fa22\") " Oct 30 04:17:29.639790 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.639759 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9nfb\" (UniqueName: \"kubernetes.io/projected/fc860910-960e-4e65-9d52-529517d1cb95-kube-api-access-s9nfb\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:29.639790 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.639771 5882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:29.639790 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.639782 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:29.639901 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.639791 5882 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc860910-960e-4e65-9d52-529517d1cb95-monitoring-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:29.639901 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.639799 5882 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fc860910-960e-4e65-9d52-529517d1cb95-otc-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:29.640008 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.639987 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ab81ec-b834-429b-b4f8-047abf18fa22-ta-internal" pod "93ab81ec-b834-429b-b4f8-047abf18fa22" (UID: "93ab81ec-b834-429b-b4f8-047abf18fa22"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:29.641608 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.641588 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ab81ec-b834-429b-b4f8-047abf18fa22-kube-api-access-m8b5j" pod "93ab81ec-b834-429b-b4f8-047abf18fa22" (UID: "93ab81ec-b834-429b-b4f8-047abf18fa22"). InnerVolumeSpecName "kube-api-access-m8b5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:29.641608 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.641599 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93ab81ec-b834-429b-b4f8-047abf18fa22-data-science-collector-ta-server-cert" pod "93ab81ec-b834-429b-b4f8-047abf18fa22" (UID: "93ab81ec-b834-429b-b4f8-047abf18fa22"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:17:29.740516 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.740430 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/93ab81ec-b834-429b-b4f8-047abf18fa22-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:29.740516 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.740472 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/93ab81ec-b834-429b-b4f8-047abf18fa22-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:29.740516 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:29.740483 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8b5j\" (UniqueName: \"kubernetes.io/projected/93ab81ec-b834-429b-b4f8-047abf18fa22-kube-api-access-m8b5j\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:30.207984 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:30.207953 5882 generic.go:356] "Generic (PLEG): container finished" podID="fc860910-960e-4e65-9d52-529517d1cb95" containerID="f679b9e9b7668632171f2840f1c506ab8ada39e6a8a0aceaa0d71a278494588b" exitCode=0 Oct 30 04:17:30.208429 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:30.207998 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:30.208429 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:30.208040 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fc860910-960e-4e65-9d52-529517d1cb95","Type":"ContainerDied","Data":"f679b9e9b7668632171f2840f1c506ab8ada39e6a8a0aceaa0d71a278494588b"} Oct 30 04:17:30.208429 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:30.208093 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fc860910-960e-4e65-9d52-529517d1cb95","Type":"ContainerDied","Data":"84f0ec802f4bba468f3374b0c660485ec059b443ccdf18932b3bcec02d1fbf6e"} Oct 30 04:17:30.208429 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:30.208120 5882 scope.go:122] "RemoveContainer" containerID="f679b9e9b7668632171f2840f1c506ab8ada39e6a8a0aceaa0d71a278494588b" Oct 30 04:17:30.209438 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:30.209417 5882 generic.go:356] "Generic (PLEG): container finished" podID="93ab81ec-b834-429b-b4f8-047abf18fa22" containerID="25333cbe79691dca0fcbbaeaf38f1220624ac69355712fce43f73bec966db0db" exitCode=0 Oct 30 04:17:30.209549 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:30.209453 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" event={"ID":"93ab81ec-b834-429b-b4f8-047abf18fa22","Type":"ContainerDied","Data":"25333cbe79691dca0fcbbaeaf38f1220624ac69355712fce43f73bec966db0db"} Oct 30 04:17:30.209549 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:30.209474 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" Oct 30 04:17:30.209549 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:30.209488 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z" event={"ID":"93ab81ec-b834-429b-b4f8-047abf18fa22","Type":"ContainerDied","Data":"796a512750e1c58ef3e116a6a2716817f0bf86eb5949122ebae7db1348b54d8f"} Oct 30 04:17:30.216957 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:30.216943 5882 scope.go:122] "RemoveContainer" containerID="f679b9e9b7668632171f2840f1c506ab8ada39e6a8a0aceaa0d71a278494588b" Oct 30 04:17:30.217222 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:17:30.217204 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f679b9e9b7668632171f2840f1c506ab8ada39e6a8a0aceaa0d71a278494588b\": container with ID starting with f679b9e9b7668632171f2840f1c506ab8ada39e6a8a0aceaa0d71a278494588b not found: ID does not exist" containerID="f679b9e9b7668632171f2840f1c506ab8ada39e6a8a0aceaa0d71a278494588b" Oct 30 04:17:30.217274 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:30.217237 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f679b9e9b7668632171f2840f1c506ab8ada39e6a8a0aceaa0d71a278494588b"} err="failed to get container status \"f679b9e9b7668632171f2840f1c506ab8ada39e6a8a0aceaa0d71a278494588b\": rpc error: code = NotFound desc = could not find container \"f679b9e9b7668632171f2840f1c506ab8ada39e6a8a0aceaa0d71a278494588b\": container with ID starting with f679b9e9b7668632171f2840f1c506ab8ada39e6a8a0aceaa0d71a278494588b not found: ID does not exist" Oct 30 04:17:30.217274 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:30.217253 5882 scope.go:122] "RemoveContainer" containerID="25333cbe79691dca0fcbbaeaf38f1220624ac69355712fce43f73bec966db0db" Oct 30 04:17:30.224446 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:30.224430 5882 scope.go:122] "RemoveContainer" containerID="25333cbe79691dca0fcbbaeaf38f1220624ac69355712fce43f73bec966db0db" Oct 30 04:17:30.224729 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:17:30.224685 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25333cbe79691dca0fcbbaeaf38f1220624ac69355712fce43f73bec966db0db\": container with ID starting with 25333cbe79691dca0fcbbaeaf38f1220624ac69355712fce43f73bec966db0db not found: ID does not exist" containerID="25333cbe79691dca0fcbbaeaf38f1220624ac69355712fce43f73bec966db0db" Oct 30 04:17:30.224729 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:30.224713 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25333cbe79691dca0fcbbaeaf38f1220624ac69355712fce43f73bec966db0db"} err="failed to get container status \"25333cbe79691dca0fcbbaeaf38f1220624ac69355712fce43f73bec966db0db\": rpc error: code = NotFound desc = could not find container \"25333cbe79691dca0fcbbaeaf38f1220624ac69355712fce43f73bec966db0db\": container with ID starting with 25333cbe79691dca0fcbbaeaf38f1220624ac69355712fce43f73bec966db0db not found: ID does not exist" Oct 30 04:17:30.263608 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:30.263584 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z"] Oct 30 04:17:30.265841 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:30.265819 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-866546d5d5-8ft4z"] Oct 30 04:17:30.303812 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:30.303743 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:17:30.308040 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:30.308014 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:17:32.009394 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:32.009362 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93ab81ec-b834-429b-b4f8-047abf18fa22" path="/var/lib/kubelet/pods/93ab81ec-b834-429b-b4f8-047abf18fa22/volumes" Oct 30 04:17:32.009788 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:32.009764 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc860910-960e-4e65-9d52-529517d1cb95" path="/var/lib/kubelet/pods/fc860910-960e-4e65-9d52-529517d1cb95/volumes" Oct 30 04:17:35.050541 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.050510 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:17:35.050984 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.050908 5882 cpu_manager.go:414] "containerMap: removing container" podUID="fc860910-960e-4e65-9d52-529517d1cb95" containerName="otc-container" Oct 30 04:17:35.050984 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.050929 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc860910-960e-4e65-9d52-529517d1cb95" containerName="otc-container" Oct 30 04:17:35.050984 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.050945 5882 cpu_manager.go:414] "containerMap: removing container" podUID="93ab81ec-b834-429b-b4f8-047abf18fa22" containerName="ta-container" Oct 30 04:17:35.050984 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.050950 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ab81ec-b834-429b-b4f8-047abf18fa22" containerName="ta-container" Oct 30 04:17:35.051106 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.051015 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ab81ec-b834-429b-b4f8-047abf18fa22" containerName="ta-container" Oct 30 04:17:35.051106 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.051030 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc860910-960e-4e65-9d52-529517d1cb95" containerName="otc-container" Oct 30 04:17:35.056254 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.056235 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.069084 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.068969 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:17:35.072874 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.072851 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 04:17:35.081672 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.081643 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 04:17:35.081790 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.081667 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 04:17:35.081790 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.081667 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 04:17:35.081790 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.081676 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:17:35.081790 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.081701 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-vt8ws\"" Oct 30 04:17:35.081790 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.081699 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 04:17:35.081790 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.081726 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 04:17:35.082009 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.081821 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 04:17:35.082078 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.081644 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 04:17:35.192122 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.192085 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.192122 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.192120 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb20d78e-ce5b-415e-9510-d08fab8563a3-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.192351 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.192143 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.192351 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.192252 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb20d78e-ce5b-415e-9510-d08fab8563a3-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.192351 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.192279 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.192351 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.192306 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.192492 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.192416 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.192492 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.192440 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.192492 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.192469 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.192607 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.192521 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.192692 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.192634 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.192692 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.192668 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj7jz\" (UniqueName: \"kubernetes.io/projected/eb20d78e-ce5b-415e-9510-d08fab8563a3-kube-api-access-zj7jz\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.293285 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.293251 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb20d78e-ce5b-415e-9510-d08fab8563a3-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.293285 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.293290 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.293523 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.293313 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.293523 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.293352 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.293523 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.293381 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.293523 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.293422 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.293523 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.293454 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.293523 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.293483 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.293523 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.293514 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj7jz\" (UniqueName: \"kubernetes.io/projected/eb20d78e-ce5b-415e-9510-d08fab8563a3-kube-api-access-zj7jz\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.293878 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.293572 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.293878 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.293607 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb20d78e-ce5b-415e-9510-d08fab8563a3-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.293878 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.293659 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.294088 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.294062 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.294415 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.294367 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.294415 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.294384 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.296493 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.296463 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.296728 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.296704 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb20d78e-ce5b-415e-9510-d08fab8563a3-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.296905 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.296883 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb20d78e-ce5b-415e-9510-d08fab8563a3-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.297106 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.297090 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.297171 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.297101 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.297171 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.297108 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.297275 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.297228 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.297476 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.297457 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.318561 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.318499 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj7jz\" (UniqueName: \"kubernetes.io/projected/eb20d78e-ce5b-415e-9510-d08fab8563a3-kube-api-access-zj7jz\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:35.662255 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.662170 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6"] Oct 30 04:17:35.667045 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.667027 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" Oct 30 04:17:35.686556 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.686529 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6"] Oct 30 04:17:35.797237 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.797199 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dnkl\" (UniqueName: \"kubernetes.io/projected/12128362-02d6-4179-aeeb-f328f8d7037d-kube-api-access-4dnkl\") pod \"data-science-collector-targetallocator-56d6b5865f-rd7q6\" (UID: \"12128362-02d6-4179-aeeb-f328f8d7037d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" Oct 30 04:17:35.797380 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.797250 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/12128362-02d6-4179-aeeb-f328f8d7037d-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-56d6b5865f-rd7q6\" (UID: \"12128362-02d6-4179-aeeb-f328f8d7037d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" Oct 30 04:17:35.797433 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.797377 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/12128362-02d6-4179-aeeb-f328f8d7037d-ta-internal\") pod \"data-science-collector-targetallocator-56d6b5865f-rd7q6\" (UID: \"12128362-02d6-4179-aeeb-f328f8d7037d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" Oct 30 04:17:35.898300 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.898264 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dnkl\" (UniqueName: \"kubernetes.io/projected/12128362-02d6-4179-aeeb-f328f8d7037d-kube-api-access-4dnkl\") pod \"data-science-collector-targetallocator-56d6b5865f-rd7q6\" (UID: \"12128362-02d6-4179-aeeb-f328f8d7037d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" Oct 30 04:17:35.898506 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.898311 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/12128362-02d6-4179-aeeb-f328f8d7037d-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-56d6b5865f-rd7q6\" (UID: \"12128362-02d6-4179-aeeb-f328f8d7037d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" Oct 30 04:17:35.898506 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.898354 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/12128362-02d6-4179-aeeb-f328f8d7037d-ta-internal\") pod \"data-science-collector-targetallocator-56d6b5865f-rd7q6\" (UID: \"12128362-02d6-4179-aeeb-f328f8d7037d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" Oct 30 04:17:35.899015 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.898984 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/12128362-02d6-4179-aeeb-f328f8d7037d-ta-internal\") pod \"data-science-collector-targetallocator-56d6b5865f-rd7q6\" (UID: \"12128362-02d6-4179-aeeb-f328f8d7037d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" Oct 30 04:17:35.900779 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.900759 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/12128362-02d6-4179-aeeb-f328f8d7037d-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-56d6b5865f-rd7q6\" (UID: \"12128362-02d6-4179-aeeb-f328f8d7037d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" Oct 30 04:17:35.923524 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.923459 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dnkl\" (UniqueName: \"kubernetes.io/projected/12128362-02d6-4179-aeeb-f328f8d7037d-kube-api-access-4dnkl\") pod \"data-science-collector-targetallocator-56d6b5865f-rd7q6\" (UID: \"12128362-02d6-4179-aeeb-f328f8d7037d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" Oct 30 04:17:35.978563 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:35.978535 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" Oct 30 04:17:36.273561 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:36.273532 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6"] Oct 30 04:17:36.274397 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:17:36.274369 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12128362_02d6_4179_aeeb_f328f8d7037d.slice/crio-725761edf740537fc90002d936861de5e5ae5bd906cfb830d380bc85cd0a467c WatchSource:0}: Error finding container 725761edf740537fc90002d936861de5e5ae5bd906cfb830d380bc85cd0a467c: Status 404 returned error can't find the container with id 725761edf740537fc90002d936861de5e5ae5bd906cfb830d380bc85cd0a467c Oct 30 04:17:36.276403 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:36.276373 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:17:36.276461 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:36.276427 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:17:37.244946 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:37.244911 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" event={"ID":"12128362-02d6-4179-aeeb-f328f8d7037d","Type":"ContainerStarted","Data":"b69a89df8a80f19f84dffed4cdbb89ff20f8632217479bab03ce45179d88477a"} Oct 30 04:17:37.245127 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:37.244960 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" Oct 30 04:17:37.245127 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:37.244978 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" event={"ID":"12128362-02d6-4179-aeeb-f328f8d7037d","Type":"ContainerStarted","Data":"725761edf740537fc90002d936861de5e5ae5bd906cfb830d380bc85cd0a467c"} Oct 30 04:17:37.246507 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:37.246489 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" Oct 30 04:17:37.289784 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:37.289734 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" podStartSLOduration=2.289721015 podStartE2EDuration="2.289721015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:37.288708178 +0000 UTC m=+5769.403023155" watchObservedRunningTime="2026-10-30 04:17:37.289721015 +0000 UTC m=+5769.404036045" Oct 30 04:17:37.362515 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:37.362477 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4"] Oct 30 04:17:37.362713 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:37.362662 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" podUID="247b0ad6-83ad-447c-bab6-3c4e1aa32956" containerName="ta-container" containerID="cri-o://5ea1b52a4f7e356d56c54629deea49d6b87bd6c8b2feb11e88310be6168394d5" gracePeriod=30 Oct 30 04:17:37.601582 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:37.601556 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" Oct 30 04:17:37.716949 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:37.716914 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247b0ad6-83ad-447c-bab6-3c4e1aa32956-kube-api-access-5zzgx\" (UniqueName: \"kubernetes.io/projected/247b0ad6-83ad-447c-bab6-3c4e1aa32956-kube-api-access-5zzgx\") pod \"247b0ad6-83ad-447c-bab6-3c4e1aa32956\" (UID: \"247b0ad6-83ad-447c-bab6-3c4e1aa32956\") " Oct 30 04:17:37.717139 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:37.716955 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/247b0ad6-83ad-447c-bab6-3c4e1aa32956-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/247b0ad6-83ad-447c-bab6-3c4e1aa32956-data-science-collector-ta-server-cert\") pod \"247b0ad6-83ad-447c-bab6-3c4e1aa32956\" (UID: \"247b0ad6-83ad-447c-bab6-3c4e1aa32956\") " Oct 30 04:17:37.717139 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:37.717020 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/247b0ad6-83ad-447c-bab6-3c4e1aa32956-ta-internal\" (UniqueName: \"kubernetes.io/configmap/247b0ad6-83ad-447c-bab6-3c4e1aa32956-ta-internal\") pod \"247b0ad6-83ad-447c-bab6-3c4e1aa32956\" (UID: \"247b0ad6-83ad-447c-bab6-3c4e1aa32956\") " Oct 30 04:17:37.717414 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:37.717389 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247b0ad6-83ad-447c-bab6-3c4e1aa32956-ta-internal" pod "247b0ad6-83ad-447c-bab6-3c4e1aa32956" (UID: "247b0ad6-83ad-447c-bab6-3c4e1aa32956"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:37.719032 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:37.719005 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247b0ad6-83ad-447c-bab6-3c4e1aa32956-kube-api-access-5zzgx" pod "247b0ad6-83ad-447c-bab6-3c4e1aa32956" (UID: "247b0ad6-83ad-447c-bab6-3c4e1aa32956"). InnerVolumeSpecName "kube-api-access-5zzgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:37.719117 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:37.719092 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/247b0ad6-83ad-447c-bab6-3c4e1aa32956-data-science-collector-ta-server-cert" pod "247b0ad6-83ad-447c-bab6-3c4e1aa32956" (UID: "247b0ad6-83ad-447c-bab6-3c4e1aa32956"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:17:37.818584 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:37.818500 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zzgx\" (UniqueName: \"kubernetes.io/projected/247b0ad6-83ad-447c-bab6-3c4e1aa32956-kube-api-access-5zzgx\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:37.818584 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:37.818529 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/247b0ad6-83ad-447c-bab6-3c4e1aa32956-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:37.818584 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:37.818539 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/247b0ad6-83ad-447c-bab6-3c4e1aa32956-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:17:38.250215 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:38.250178 5882 generic.go:356] "Generic (PLEG): container finished" podID="247b0ad6-83ad-447c-bab6-3c4e1aa32956" containerID="5ea1b52a4f7e356d56c54629deea49d6b87bd6c8b2feb11e88310be6168394d5" exitCode=0 Oct 30 04:17:38.250215 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:38.250216 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" Oct 30 04:17:38.250417 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:38.250266 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" event={"ID":"247b0ad6-83ad-447c-bab6-3c4e1aa32956","Type":"ContainerDied","Data":"5ea1b52a4f7e356d56c54629deea49d6b87bd6c8b2feb11e88310be6168394d5"} Oct 30 04:17:38.250417 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:38.250300 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4" event={"ID":"247b0ad6-83ad-447c-bab6-3c4e1aa32956","Type":"ContainerDied","Data":"f20a18b2c9c1a1a4172aed5e9db84a3939e08348a0b33e9ca2711c6e2981a4c3"} Oct 30 04:17:38.250417 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:38.250323 5882 scope.go:122] "RemoveContainer" containerID="5ea1b52a4f7e356d56c54629deea49d6b87bd6c8b2feb11e88310be6168394d5" Oct 30 04:17:38.258258 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:38.258242 5882 scope.go:122] "RemoveContainer" containerID="5ea1b52a4f7e356d56c54629deea49d6b87bd6c8b2feb11e88310be6168394d5" Oct 30 04:17:38.258491 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:17:38.258466 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ea1b52a4f7e356d56c54629deea49d6b87bd6c8b2feb11e88310be6168394d5\": container with ID starting with 5ea1b52a4f7e356d56c54629deea49d6b87bd6c8b2feb11e88310be6168394d5 not found: ID does not exist" containerID="5ea1b52a4f7e356d56c54629deea49d6b87bd6c8b2feb11e88310be6168394d5" Oct 30 04:17:38.258568 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:38.258495 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ea1b52a4f7e356d56c54629deea49d6b87bd6c8b2feb11e88310be6168394d5"} err="failed to get container status \"5ea1b52a4f7e356d56c54629deea49d6b87bd6c8b2feb11e88310be6168394d5\": rpc error: code = NotFound desc = could not find container \"5ea1b52a4f7e356d56c54629deea49d6b87bd6c8b2feb11e88310be6168394d5\": container with ID starting with 5ea1b52a4f7e356d56c54629deea49d6b87bd6c8b2feb11e88310be6168394d5 not found: ID does not exist" Oct 30 04:17:38.305571 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:38.305548 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4"] Oct 30 04:17:38.319776 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:38.319753 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-74687f45b8-nb2v4"] Oct 30 04:17:39.534866 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:39.534792 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:39.567798 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:39.567767 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:39.635553 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:39.635521 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:39.635709 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:39.635668 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:39.652111 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:39.652091 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") DevicePath \"csi-927d18ef2eb99137c68fbe5e0bb8ea629bb5219ed56cdede72da7c0f045be95e\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:39.755282 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:39.755247 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a535375d22e374ba333968465471c7604a06f9d95a513910fe4bb083b629e986/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:39.785324 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:39.785269 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:39.876379 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:39.876349 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:40.010251 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:40.010218 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="247b0ad6-83ad-447c-bab6-3c4e1aa32956" path="/var/lib/kubelet/pods/247b0ad6-83ad-447c-bab6-3c4e1aa32956/volumes" Oct 30 04:17:40.196991 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:40.196963 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:17:40.197380 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:17:40.197358 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb20d78e_ce5b_415e_9510_d08fab8563a3.slice/crio-b024233e1964d2783e954e1fa781bccd82e00e60704cafda972bec7ba70af3bf WatchSource:0}: Error finding container b024233e1964d2783e954e1fa781bccd82e00e60704cafda972bec7ba70af3bf: Status 404 returned error can't find the container with id b024233e1964d2783e954e1fa781bccd82e00e60704cafda972bec7ba70af3bf Oct 30 04:17:40.260882 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:40.260853 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"eb20d78e-ce5b-415e-9510-d08fab8563a3","Type":"ContainerStarted","Data":"b024233e1964d2783e954e1fa781bccd82e00e60704cafda972bec7ba70af3bf"} Oct 30 04:17:41.267685 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:41.267652 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"eb20d78e-ce5b-415e-9510-d08fab8563a3","Type":"ContainerStarted","Data":"858fe7b6c13ee6817896fd179b3842d042daca22621b03b626dc7544c87fb6ae"} Oct 30 04:17:44.286289 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:44.286255 5882 generic.go:356] "Generic (PLEG): container finished" podID="eb20d78e-ce5b-415e-9510-d08fab8563a3" containerID="858fe7b6c13ee6817896fd179b3842d042daca22621b03b626dc7544c87fb6ae" exitCode=0 Oct 30 04:17:44.286675 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:44.286327 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"eb20d78e-ce5b-415e-9510-d08fab8563a3","Type":"ContainerDied","Data":"858fe7b6c13ee6817896fd179b3842d042daca22621b03b626dc7544c87fb6ae"} Oct 30 04:17:45.292931 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:45.292896 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"eb20d78e-ce5b-415e-9510-d08fab8563a3","Type":"ContainerStarted","Data":"6040d9a3ef3539e8a5e2b89b40a629a9de2a1a1acab7f4d5aa6e0ea8be6edb2e"} Oct 30 04:17:45.293292 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:45.292961 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"eb20d78e-ce5b-415e-9510-d08fab8563a3","Type":"ContainerStarted","Data":"c6cb6ba6050d5b3e13fa5ff5672a26893abbe0fda369e41e3ff24920f8f63761"} Oct 30 04:17:45.293292 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:45.292984 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"eb20d78e-ce5b-415e-9510-d08fab8563a3","Type":"ContainerStarted","Data":"ff83613c2d2a0d121513e40126f532f679e2da31b2359b3ec1dbacf5a4eefc7d"} Oct 30 04:17:45.357305 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:45.357250 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=6.872981804 podStartE2EDuration="10.357237166s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.484255362s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:45.348899604 +0000 UTC m=+5777.463214579" watchObservedRunningTime="2026-10-30 04:17:45.357237166 +0000 UTC m=+5777.471552168" Oct 30 04:17:49.877306 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:49.877271 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:53.942152 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:17:53.942116 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:17:54.876570 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:54.876534 5882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:54.881920 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:54.881898 5882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:17:55.341946 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:17:55.341919 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:01.828670 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:01.828633 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx"] Oct 30 04:18:01.829061 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:01.828997 5882 cpu_manager.go:414] "containerMap: removing container" podUID="247b0ad6-83ad-447c-bab6-3c4e1aa32956" containerName="ta-container" Oct 30 04:18:01.829061 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:01.829008 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247b0ad6-83ad-447c-bab6-3c4e1aa32956" containerName="ta-container" Oct 30 04:18:01.829133 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:01.829096 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="247b0ad6-83ad-447c-bab6-3c4e1aa32956" containerName="ta-container" Oct 30 04:18:01.832441 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:01.832422 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" Oct 30 04:18:01.852688 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:01.852648 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx"] Oct 30 04:18:01.939010 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:01.938980 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/55b27cda-7dfb-4a05-a70d-ec903c203fb5-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6999ff9b94-2t2qx\" (UID: \"55b27cda-7dfb-4a05-a70d-ec903c203fb5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" Oct 30 04:18:01.939179 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:01.939038 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/55b27cda-7dfb-4a05-a70d-ec903c203fb5-ta-internal\") pod \"data-science-collector-targetallocator-6999ff9b94-2t2qx\" (UID: \"55b27cda-7dfb-4a05-a70d-ec903c203fb5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" Oct 30 04:18:01.939179 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:01.939145 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk5ls\" (UniqueName: \"kubernetes.io/projected/55b27cda-7dfb-4a05-a70d-ec903c203fb5-kube-api-access-kk5ls\") pod \"data-science-collector-targetallocator-6999ff9b94-2t2qx\" (UID: \"55b27cda-7dfb-4a05-a70d-ec903c203fb5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" Oct 30 04:18:02.039991 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:02.039951 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk5ls\" (UniqueName: \"kubernetes.io/projected/55b27cda-7dfb-4a05-a70d-ec903c203fb5-kube-api-access-kk5ls\") pod \"data-science-collector-targetallocator-6999ff9b94-2t2qx\" (UID: \"55b27cda-7dfb-4a05-a70d-ec903c203fb5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" Oct 30 04:18:02.040171 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:02.040025 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/55b27cda-7dfb-4a05-a70d-ec903c203fb5-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6999ff9b94-2t2qx\" (UID: \"55b27cda-7dfb-4a05-a70d-ec903c203fb5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" Oct 30 04:18:02.040171 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:02.040072 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/55b27cda-7dfb-4a05-a70d-ec903c203fb5-ta-internal\") pod \"data-science-collector-targetallocator-6999ff9b94-2t2qx\" (UID: \"55b27cda-7dfb-4a05-a70d-ec903c203fb5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" Oct 30 04:18:02.040842 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:02.040822 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/55b27cda-7dfb-4a05-a70d-ec903c203fb5-ta-internal\") pod \"data-science-collector-targetallocator-6999ff9b94-2t2qx\" (UID: \"55b27cda-7dfb-4a05-a70d-ec903c203fb5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" Oct 30 04:18:02.042717 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:02.042693 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/55b27cda-7dfb-4a05-a70d-ec903c203fb5-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6999ff9b94-2t2qx\" (UID: \"55b27cda-7dfb-4a05-a70d-ec903c203fb5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" Oct 30 04:18:02.065288 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:02.065252 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk5ls\" (UniqueName: \"kubernetes.io/projected/55b27cda-7dfb-4a05-a70d-ec903c203fb5-kube-api-access-kk5ls\") pod \"data-science-collector-targetallocator-6999ff9b94-2t2qx\" (UID: \"55b27cda-7dfb-4a05-a70d-ec903c203fb5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" Oct 30 04:18:02.144013 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:02.143925 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" Oct 30 04:18:02.470200 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:02.470165 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx"] Oct 30 04:18:02.470864 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:18:02.470834 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55b27cda_7dfb_4a05_a70d_ec903c203fb5.slice/crio-3eea64cdcce5bed423e460b2e885c8743d130328ea7ec0d7280312208ba9841b WatchSource:0}: Error finding container 3eea64cdcce5bed423e460b2e885c8743d130328ea7ec0d7280312208ba9841b: Status 404 returned error can't find the container with id 3eea64cdcce5bed423e460b2e885c8743d130328ea7ec0d7280312208ba9841b Oct 30 04:18:02.473197 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:02.473163 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:18:02.473277 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:02.473227 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:18:03.377020 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:03.376982 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" event={"ID":"55b27cda-7dfb-4a05-a70d-ec903c203fb5","Type":"ContainerStarted","Data":"d1bfe7ea09cd5bb2b062280f5d486b29d82c7048a727123d33143f31495eb9ce"} Oct 30 04:18:03.377020 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:03.377024 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" event={"ID":"55b27cda-7dfb-4a05-a70d-ec903c203fb5","Type":"ContainerStarted","Data":"3eea64cdcce5bed423e460b2e885c8743d130328ea7ec0d7280312208ba9841b"} Oct 30 04:18:03.377515 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:03.377169 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" Oct 30 04:18:03.379013 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:03.378989 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" Oct 30 04:18:03.423289 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:03.423235 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" podStartSLOduration=2.423221867 podStartE2EDuration="2.423221867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:03.422294058 +0000 UTC m=+5795.536609033" watchObservedRunningTime="2026-10-30 04:18:03.423221867 +0000 UTC m=+5795.537536842" Oct 30 04:18:03.497193 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:03.497153 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6"] Oct 30 04:18:03.497431 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:03.497335 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" podUID="12128362-02d6-4179-aeeb-f328f8d7037d" containerName="ta-container" containerID="cri-o://b69a89df8a80f19f84dffed4cdbb89ff20f8632217479bab03ce45179d88477a" gracePeriod=30 Oct 30 04:18:03.738994 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:03.738967 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" Oct 30 04:18:03.856472 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:03.856436 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12128362-02d6-4179-aeeb-f328f8d7037d-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/12128362-02d6-4179-aeeb-f328f8d7037d-data-science-collector-ta-server-cert\") pod \"12128362-02d6-4179-aeeb-f328f8d7037d\" (UID: \"12128362-02d6-4179-aeeb-f328f8d7037d\") " Oct 30 04:18:03.856656 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:03.856480 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12128362-02d6-4179-aeeb-f328f8d7037d-kube-api-access-4dnkl\" (UniqueName: \"kubernetes.io/projected/12128362-02d6-4179-aeeb-f328f8d7037d-kube-api-access-4dnkl\") pod \"12128362-02d6-4179-aeeb-f328f8d7037d\" (UID: \"12128362-02d6-4179-aeeb-f328f8d7037d\") " Oct 30 04:18:03.856656 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:03.856531 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12128362-02d6-4179-aeeb-f328f8d7037d-ta-internal\" (UniqueName: \"kubernetes.io/configmap/12128362-02d6-4179-aeeb-f328f8d7037d-ta-internal\") pod \"12128362-02d6-4179-aeeb-f328f8d7037d\" (UID: \"12128362-02d6-4179-aeeb-f328f8d7037d\") " Oct 30 04:18:03.856956 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:03.856923 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12128362-02d6-4179-aeeb-f328f8d7037d-ta-internal" pod "12128362-02d6-4179-aeeb-f328f8d7037d" (UID: "12128362-02d6-4179-aeeb-f328f8d7037d"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:18:03.858523 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:03.858499 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12128362-02d6-4179-aeeb-f328f8d7037d-data-science-collector-ta-server-cert" pod "12128362-02d6-4179-aeeb-f328f8d7037d" (UID: "12128362-02d6-4179-aeeb-f328f8d7037d"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:03.858607 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:03.858549 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12128362-02d6-4179-aeeb-f328f8d7037d-kube-api-access-4dnkl" pod "12128362-02d6-4179-aeeb-f328f8d7037d" (UID: "12128362-02d6-4179-aeeb-f328f8d7037d"). InnerVolumeSpecName "kube-api-access-4dnkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:03.957458 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:03.957365 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/12128362-02d6-4179-aeeb-f328f8d7037d-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:03.957458 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:03.957410 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/12128362-02d6-4179-aeeb-f328f8d7037d-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:03.957458 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:03.957422 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dnkl\" (UniqueName: \"kubernetes.io/projected/12128362-02d6-4179-aeeb-f328f8d7037d-kube-api-access-4dnkl\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:04.383051 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:04.383012 5882 generic.go:356] "Generic (PLEG): container finished" podID="12128362-02d6-4179-aeeb-f328f8d7037d" containerID="b69a89df8a80f19f84dffed4cdbb89ff20f8632217479bab03ce45179d88477a" exitCode=0 Oct 30 04:18:04.383486 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:04.383062 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" Oct 30 04:18:04.383486 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:04.383090 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" event={"ID":"12128362-02d6-4179-aeeb-f328f8d7037d","Type":"ContainerDied","Data":"b69a89df8a80f19f84dffed4cdbb89ff20f8632217479bab03ce45179d88477a"} Oct 30 04:18:04.383486 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:04.383133 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6" event={"ID":"12128362-02d6-4179-aeeb-f328f8d7037d","Type":"ContainerDied","Data":"725761edf740537fc90002d936861de5e5ae5bd906cfb830d380bc85cd0a467c"} Oct 30 04:18:04.383486 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:04.383150 5882 scope.go:122] "RemoveContainer" containerID="b69a89df8a80f19f84dffed4cdbb89ff20f8632217479bab03ce45179d88477a" Oct 30 04:18:04.391394 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:04.391376 5882 scope.go:122] "RemoveContainer" containerID="b69a89df8a80f19f84dffed4cdbb89ff20f8632217479bab03ce45179d88477a" Oct 30 04:18:04.391761 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:18:04.391742 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b69a89df8a80f19f84dffed4cdbb89ff20f8632217479bab03ce45179d88477a\": container with ID starting with b69a89df8a80f19f84dffed4cdbb89ff20f8632217479bab03ce45179d88477a not found: ID does not exist" containerID="b69a89df8a80f19f84dffed4cdbb89ff20f8632217479bab03ce45179d88477a" Oct 30 04:18:04.391842 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:04.391769 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b69a89df8a80f19f84dffed4cdbb89ff20f8632217479bab03ce45179d88477a"} err="failed to get container status \"b69a89df8a80f19f84dffed4cdbb89ff20f8632217479bab03ce45179d88477a\": rpc error: code = NotFound desc = could not find container \"b69a89df8a80f19f84dffed4cdbb89ff20f8632217479bab03ce45179d88477a\": container with ID starting with b69a89df8a80f19f84dffed4cdbb89ff20f8632217479bab03ce45179d88477a not found: ID does not exist" Oct 30 04:18:04.437862 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:04.437832 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6"] Oct 30 04:18:04.451738 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:04.451713 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-56d6b5865f-rd7q6"] Oct 30 04:18:06.010223 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:06.010195 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12128362-02d6-4179-aeeb-f328f8d7037d" path="/var/lib/kubelet/pods/12128362-02d6-4179-aeeb-f328f8d7037d/volumes" Oct 30 04:18:06.716804 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:06.716767 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:18:06.717133 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:06.717093 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="4ef93eb8-40a1-4388-873f-e363c354abee" containerName="config-reloader" containerID="cri-o://8a5854175ea2517a00d169b5409651a286660dd6bd7ab2b4d52a69e559a5c4ed" gracePeriod=120 Oct 30 04:18:06.717222 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:06.717078 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="4ef93eb8-40a1-4388-873f-e363c354abee" containerName="alertmanager" containerID="cri-o://852b203ab59f57407551343c61f931115b8548a67054b27574fe9cfdd2f3f7ce" gracePeriod=120 Oct 30 04:18:06.730510 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:06.730478 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:18:06.730778 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:06.730750 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="eb20d78e-ce5b-415e-9510-d08fab8563a3" containerName="prometheus" containerID="cri-o://ff83613c2d2a0d121513e40126f532f679e2da31b2359b3ec1dbacf5a4eefc7d" gracePeriod=600 Oct 30 04:18:06.730910 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:06.730818 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="eb20d78e-ce5b-415e-9510-d08fab8563a3" containerName="thanos-sidecar" containerID="cri-o://6040d9a3ef3539e8a5e2b89b40a629a9de2a1a1acab7f4d5aa6e0ea8be6edb2e" gracePeriod=600 Oct 30 04:18:06.730910 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:06.730832 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="eb20d78e-ce5b-415e-9510-d08fab8563a3" containerName="config-reloader" containerID="cri-o://c6cb6ba6050d5b3e13fa5ff5672a26893abbe0fda369e41e3ff24920f8f63761" gracePeriod=600 Oct 30 04:18:07.076551 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.076507 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="4ef93eb8-40a1-4388-873f-e363c354abee" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.154:9093/-/ready\": dial tcp 10.134.0.154:9093: connect: connection refused" Oct 30 04:18:07.376544 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.376495 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx"] Oct 30 04:18:07.376778 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.376740 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" podUID="55b27cda-7dfb-4a05-a70d-ec903c203fb5" containerName="ta-container" containerID="cri-o://d1bfe7ea09cd5bb2b062280f5d486b29d82c7048a727123d33143f31495eb9ce" gracePeriod=30 Oct 30 04:18:07.404548 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.404515 5882 generic.go:356] "Generic (PLEG): container finished" podID="eb20d78e-ce5b-415e-9510-d08fab8563a3" containerID="6040d9a3ef3539e8a5e2b89b40a629a9de2a1a1acab7f4d5aa6e0ea8be6edb2e" exitCode=0 Oct 30 04:18:07.404548 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.404547 5882 generic.go:356] "Generic (PLEG): container finished" podID="eb20d78e-ce5b-415e-9510-d08fab8563a3" containerID="c6cb6ba6050d5b3e13fa5ff5672a26893abbe0fda369e41e3ff24920f8f63761" exitCode=0 Oct 30 04:18:07.404727 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.404557 5882 generic.go:356] "Generic (PLEG): container finished" podID="eb20d78e-ce5b-415e-9510-d08fab8563a3" containerID="ff83613c2d2a0d121513e40126f532f679e2da31b2359b3ec1dbacf5a4eefc7d" exitCode=0 Oct 30 04:18:07.404727 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.404678 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"eb20d78e-ce5b-415e-9510-d08fab8563a3","Type":"ContainerDied","Data":"6040d9a3ef3539e8a5e2b89b40a629a9de2a1a1acab7f4d5aa6e0ea8be6edb2e"} Oct 30 04:18:07.404813 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.404747 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"eb20d78e-ce5b-415e-9510-d08fab8563a3","Type":"ContainerDied","Data":"c6cb6ba6050d5b3e13fa5ff5672a26893abbe0fda369e41e3ff24920f8f63761"} Oct 30 04:18:07.404813 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.404789 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"eb20d78e-ce5b-415e-9510-d08fab8563a3","Type":"ContainerDied","Data":"ff83613c2d2a0d121513e40126f532f679e2da31b2359b3ec1dbacf5a4eefc7d"} Oct 30 04:18:07.407762 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.407735 5882 generic.go:356] "Generic (PLEG): container finished" podID="4ef93eb8-40a1-4388-873f-e363c354abee" containerID="8a5854175ea2517a00d169b5409651a286660dd6bd7ab2b4d52a69e559a5c4ed" exitCode=0 Oct 30 04:18:07.407878 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.407807 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"4ef93eb8-40a1-4388-873f-e363c354abee","Type":"ContainerDied","Data":"8a5854175ea2517a00d169b5409651a286660dd6bd7ab2b4d52a69e559a5c4ed"} Oct 30 04:18:07.493341 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.493310 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:07.499437 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.499398 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:18:07.499675 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.499645 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="07adcc73-ec51-42e3-bf3d-115b9cbc34fd" containerName="perses" containerID="cri-o://3552a0ee9d2c25d5ec1d317a30fd3ccc2aca5f34520ddf1dc914c76e97dbe08f" gracePeriod=30 Oct 30 04:18:07.592254 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.592219 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"eb20d78e-ce5b-415e-9510-d08fab8563a3\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " Oct 30 04:18:07.592437 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.592268 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb20d78e-ce5b-415e-9510-d08fab8563a3-config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb20d78e-ce5b-415e-9510-d08fab8563a3-config-out\") pod \"eb20d78e-ce5b-415e-9510-d08fab8563a3\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " Oct 30 04:18:07.592437 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.592296 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-secret-prometheus-web-tls-ca\") pod \"eb20d78e-ce5b-415e-9510-d08fab8563a3\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " Oct 30 04:18:07.592437 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.592393 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"eb20d78e-ce5b-415e-9510-d08fab8563a3\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " Oct 30 04:18:07.592631 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.592516 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"eb20d78e-ce5b-415e-9510-d08fab8563a3\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " Oct 30 04:18:07.592631 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.592552 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-config\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-config\") pod \"eb20d78e-ce5b-415e-9510-d08fab8563a3\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " Oct 30 04:18:07.592631 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.592591 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config\") pod \"eb20d78e-ce5b-415e-9510-d08fab8563a3\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " Oct 30 04:18:07.592797 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.592722 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"eb20d78e-ce5b-415e-9510-d08fab8563a3\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " Oct 30 04:18:07.592797 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.592754 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb20d78e-ce5b-415e-9510-d08fab8563a3-kube-api-access-zj7jz\" (UniqueName: \"kubernetes.io/projected/eb20d78e-ce5b-415e-9510-d08fab8563a3-kube-api-access-zj7jz\") pod \"eb20d78e-ce5b-415e-9510-d08fab8563a3\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " Oct 30 04:18:07.592797 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.592783 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-thanos-prometheus-http-client-file\") pod \"eb20d78e-ce5b-415e-9510-d08fab8563a3\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " Oct 30 04:18:07.592967 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.592816 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb20d78e-ce5b-415e-9510-d08fab8563a3-tls-assets\" (UniqueName: \"kubernetes.io/projected/eb20d78e-ce5b-415e-9510-d08fab8563a3-tls-assets\") pod \"eb20d78e-ce5b-415e-9510-d08fab8563a3\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " Oct 30 04:18:07.592967 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.592868 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"eb20d78e-ce5b-415e-9510-d08fab8563a3\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " Oct 30 04:18:07.592967 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.592912 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"eb20d78e-ce5b-415e-9510-d08fab8563a3\" (UID: \"eb20d78e-ce5b-415e-9510-d08fab8563a3\") " Oct 30 04:18:07.593426 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.593383 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-2" pod "eb20d78e-ce5b-415e-9510-d08fab8563a3" (UID: "eb20d78e-ce5b-415e-9510-d08fab8563a3"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:18:07.597201 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.594596 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-0" pod "eb20d78e-ce5b-415e-9510-d08fab8563a3" (UID: "eb20d78e-ce5b-415e-9510-d08fab8563a3"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:18:07.597784 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.597726 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-1" pod "eb20d78e-ce5b-415e-9510-d08fab8563a3" (UID: "eb20d78e-ce5b-415e-9510-d08fab8563a3"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:18:07.598797 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.598765 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "eb20d78e-ce5b-415e-9510-d08fab8563a3" (UID: "eb20d78e-ce5b-415e-9510-d08fab8563a3"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:07.598905 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.598854 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "eb20d78e-ce5b-415e-9510-d08fab8563a3" (UID: "eb20d78e-ce5b-415e-9510-d08fab8563a3"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:07.599454 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.599419 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-config" pod "eb20d78e-ce5b-415e-9510-d08fab8563a3" (UID: "eb20d78e-ce5b-415e-9510-d08fab8563a3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:07.599769 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.599717 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb20d78e-ce5b-415e-9510-d08fab8563a3-tls-assets" pod "eb20d78e-ce5b-415e-9510-d08fab8563a3" (UID: "eb20d78e-ce5b-415e-9510-d08fab8563a3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:07.600272 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.600220 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-thanos-prometheus-http-client-file" pod "eb20d78e-ce5b-415e-9510-d08fab8563a3" (UID: "eb20d78e-ce5b-415e-9510-d08fab8563a3"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:07.600763 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.600731 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb20d78e-ce5b-415e-9510-d08fab8563a3-config-out" pod "eb20d78e-ce5b-415e-9510-d08fab8563a3" (UID: "eb20d78e-ce5b-415e-9510-d08fab8563a3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:18:07.601211 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.601170 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb20d78e-ce5b-415e-9510-d08fab8563a3-kube-api-access-zj7jz" pod "eb20d78e-ce5b-415e-9510-d08fab8563a3" (UID: "eb20d78e-ce5b-415e-9510-d08fab8563a3"). InnerVolumeSpecName "kube-api-access-zj7jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:07.601334 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.601274 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-secret-prometheus-web-tls-ca" pod "eb20d78e-ce5b-415e-9510-d08fab8563a3" (UID: "eb20d78e-ce5b-415e-9510-d08fab8563a3"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:07.601983 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.601963 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4" pod "eb20d78e-ce5b-415e-9510-d08fab8563a3" (UID: "eb20d78e-ce5b-415e-9510-d08fab8563a3"). InnerVolumeSpecName "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:18:07.615668 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.615643 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config" pod "eb20d78e-ce5b-415e-9510-d08fab8563a3" (UID: "eb20d78e-ce5b-415e-9510-d08fab8563a3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:07.635045 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.635018 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" Oct 30 04:18:07.694081 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.694047 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55b27cda-7dfb-4a05-a70d-ec903c203fb5-ta-internal\" (UniqueName: \"kubernetes.io/configmap/55b27cda-7dfb-4a05-a70d-ec903c203fb5-ta-internal\") pod \"55b27cda-7dfb-4a05-a70d-ec903c203fb5\" (UID: \"55b27cda-7dfb-4a05-a70d-ec903c203fb5\") " Oct 30 04:18:07.694245 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.694137 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55b27cda-7dfb-4a05-a70d-ec903c203fb5-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/55b27cda-7dfb-4a05-a70d-ec903c203fb5-data-science-collector-ta-server-cert\") pod \"55b27cda-7dfb-4a05-a70d-ec903c203fb5\" (UID: \"55b27cda-7dfb-4a05-a70d-ec903c203fb5\") " Oct 30 04:18:07.694312 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.694248 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b27cda-7dfb-4a05-a70d-ec903c203fb5-kube-api-access-kk5ls\" (UniqueName: \"kubernetes.io/projected/55b27cda-7dfb-4a05-a70d-ec903c203fb5-kube-api-access-kk5ls\") pod \"55b27cda-7dfb-4a05-a70d-ec903c203fb5\" (UID: \"55b27cda-7dfb-4a05-a70d-ec903c203fb5\") " Oct 30 04:18:07.694400 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.694376 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b27cda-7dfb-4a05-a70d-ec903c203fb5-ta-internal" pod "55b27cda-7dfb-4a05-a70d-ec903c203fb5" (UID: "55b27cda-7dfb-4a05-a70d-ec903c203fb5"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:18:07.694525 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.694510 5882 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.694700 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.694531 5882 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb20d78e-ce5b-415e-9510-d08fab8563a3-config-out\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.694700 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.694544 5882 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-secret-prometheus-web-tls-ca\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.694700 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.694563 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:18:07.694700 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.694575 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.694700 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.694588 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.694700 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.694603 5882 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.694700 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.694630 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/55b27cda-7dfb-4a05-a70d-ec903c203fb5-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.694700 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.694646 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.694700 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.694665 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj7jz\" (UniqueName: \"kubernetes.io/projected/eb20d78e-ce5b-415e-9510-d08fab8563a3-kube-api-access-zj7jz\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.694700 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.694682 5882 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.694700 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.694695 5882 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb20d78e-ce5b-415e-9510-d08fab8563a3-tls-assets\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.695375 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.694709 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb20d78e-ce5b-415e-9510-d08fab8563a3-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.695375 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.694723 5882 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/eb20d78e-ce5b-415e-9510-d08fab8563a3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.696525 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.696495 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55b27cda-7dfb-4a05-a70d-ec903c203fb5-data-science-collector-ta-server-cert" pod "55b27cda-7dfb-4a05-a70d-ec903c203fb5" (UID: "55b27cda-7dfb-4a05-a70d-ec903c203fb5"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:07.696630 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.696526 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b27cda-7dfb-4a05-a70d-ec903c203fb5-kube-api-access-kk5ls" pod "55b27cda-7dfb-4a05-a70d-ec903c203fb5" (UID: "55b27cda-7dfb-4a05-a70d-ec903c203fb5"). InnerVolumeSpecName "kube-api-access-kk5ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:07.708474 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.708454 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:18:07.737060 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.737034 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:07.795322 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.795287 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-kube-api-access-wn7zk\" (UniqueName: \"kubernetes.io/projected/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-kube-api-access-wn7zk\") pod \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\" (UID: \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\") " Oct 30 04:18:07.795519 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.795432 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\" (UID: \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\") " Oct 30 04:18:07.795596 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.795579 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-config\" (UniqueName: \"kubernetes.io/configmap/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-config\") pod \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\" (UID: \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\") " Oct 30 04:18:07.795691 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.795642 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-plugins\" (UniqueName: \"kubernetes.io/empty-dir/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-plugins\") pod \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\" (UID: \"07adcc73-ec51-42e3-bf3d-115b9cbc34fd\") " Oct 30 04:18:07.796048 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.795985 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-config" pod "07adcc73-ec51-42e3-bf3d-115b9cbc34fd" (UID: "07adcc73-ec51-42e3-bf3d-115b9cbc34fd"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:18:07.796048 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.795970 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk5ls\" (UniqueName: \"kubernetes.io/projected/55b27cda-7dfb-4a05-a70d-ec903c203fb5-kube-api-access-kk5ls\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.796048 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.796048 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/55b27cda-7dfb-4a05-a70d-ec903c203fb5-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.796308 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.796079 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-927d18ef2eb99137c68fbe5e0bb8ea629bb5219ed56cdede72da7c0f045be95e\"" Oct 30 04:18:07.797703 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.797675 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-kube-api-access-wn7zk" pod "07adcc73-ec51-42e3-bf3d-115b9cbc34fd" (UID: "07adcc73-ec51-42e3-bf3d-115b9cbc34fd"). InnerVolumeSpecName "kube-api-access-wn7zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:07.799455 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.799430 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e" pod "07adcc73-ec51-42e3-bf3d-115b9cbc34fd" (UID: "07adcc73-ec51-42e3-bf3d-115b9cbc34fd"). InnerVolumeSpecName "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:18:07.897228 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.897154 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn7zk\" (UniqueName: \"kubernetes.io/projected/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-kube-api-access-wn7zk\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.897228 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.897205 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:18:07.897228 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.897218 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.910479 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.910458 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:18:07.946065 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.946023 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-plugins" pod "07adcc73-ec51-42e3-bf3d-115b9cbc34fd" (UID: "07adcc73-ec51-42e3-bf3d-115b9cbc34fd"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:18:07.998417 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.998386 5882 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/07adcc73-ec51-42e3-bf3d-115b9cbc34fd-plugins\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.998417 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:07.998416 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" Oct 30 04:18:08.415900 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.415859 5882 generic.go:356] "Generic (PLEG): container finished" podID="07adcc73-ec51-42e3-bf3d-115b9cbc34fd" containerID="3552a0ee9d2c25d5ec1d317a30fd3ccc2aca5f34520ddf1dc914c76e97dbe08f" exitCode=0 Oct 30 04:18:08.416314 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.416038 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:08.416648 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.416563 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"07adcc73-ec51-42e3-bf3d-115b9cbc34fd","Type":"ContainerDied","Data":"3552a0ee9d2c25d5ec1d317a30fd3ccc2aca5f34520ddf1dc914c76e97dbe08f"} Oct 30 04:18:08.417369 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.416607 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"07adcc73-ec51-42e3-bf3d-115b9cbc34fd","Type":"ContainerDied","Data":"8c7d1d81834bc65a09f1bc07b5f02c3e15fbef824ed2f1b1448d78b9b7f1506a"} Oct 30 04:18:08.417747 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.416809 5882 scope.go:122] "RemoveContainer" containerID="3552a0ee9d2c25d5ec1d317a30fd3ccc2aca5f34520ddf1dc914c76e97dbe08f" Oct 30 04:18:08.420369 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.420335 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"eb20d78e-ce5b-415e-9510-d08fab8563a3","Type":"ContainerDied","Data":"b024233e1964d2783e954e1fa781bccd82e00e60704cafda972bec7ba70af3bf"} Oct 30 04:18:08.420460 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.420342 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:08.422592 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.422568 5882 generic.go:356] "Generic (PLEG): container finished" podID="55b27cda-7dfb-4a05-a70d-ec903c203fb5" containerID="d1bfe7ea09cd5bb2b062280f5d486b29d82c7048a727123d33143f31495eb9ce" exitCode=0 Oct 30 04:18:08.422730 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.422606 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" Oct 30 04:18:08.422730 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.422626 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" event={"ID":"55b27cda-7dfb-4a05-a70d-ec903c203fb5","Type":"ContainerDied","Data":"d1bfe7ea09cd5bb2b062280f5d486b29d82c7048a727123d33143f31495eb9ce"} Oct 30 04:18:08.422730 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.422652 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx" event={"ID":"55b27cda-7dfb-4a05-a70d-ec903c203fb5","Type":"ContainerDied","Data":"3eea64cdcce5bed423e460b2e885c8743d130328ea7ec0d7280312208ba9841b"} Oct 30 04:18:08.429311 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.429286 5882 scope.go:122] "RemoveContainer" containerID="3552a0ee9d2c25d5ec1d317a30fd3ccc2aca5f34520ddf1dc914c76e97dbe08f" Oct 30 04:18:08.429641 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:18:08.429597 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3552a0ee9d2c25d5ec1d317a30fd3ccc2aca5f34520ddf1dc914c76e97dbe08f\": container with ID starting with 3552a0ee9d2c25d5ec1d317a30fd3ccc2aca5f34520ddf1dc914c76e97dbe08f not found: ID does not exist" containerID="3552a0ee9d2c25d5ec1d317a30fd3ccc2aca5f34520ddf1dc914c76e97dbe08f" Oct 30 04:18:08.429723 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.429650 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3552a0ee9d2c25d5ec1d317a30fd3ccc2aca5f34520ddf1dc914c76e97dbe08f"} err="failed to get container status \"3552a0ee9d2c25d5ec1d317a30fd3ccc2aca5f34520ddf1dc914c76e97dbe08f\": rpc error: code = NotFound desc = could not find container \"3552a0ee9d2c25d5ec1d317a30fd3ccc2aca5f34520ddf1dc914c76e97dbe08f\": container with ID starting with 3552a0ee9d2c25d5ec1d317a30fd3ccc2aca5f34520ddf1dc914c76e97dbe08f not found: ID does not exist" Oct 30 04:18:08.429723 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.429674 5882 scope.go:122] "RemoveContainer" containerID="6040d9a3ef3539e8a5e2b89b40a629a9de2a1a1acab7f4d5aa6e0ea8be6edb2e" Oct 30 04:18:08.438389 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.438368 5882 scope.go:122] "RemoveContainer" containerID="c6cb6ba6050d5b3e13fa5ff5672a26893abbe0fda369e41e3ff24920f8f63761" Oct 30 04:18:08.448178 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.448153 5882 scope.go:122] "RemoveContainer" containerID="ff83613c2d2a0d121513e40126f532f679e2da31b2359b3ec1dbacf5a4eefc7d" Oct 30 04:18:08.456201 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.456183 5882 scope.go:122] "RemoveContainer" containerID="858fe7b6c13ee6817896fd179b3842d042daca22621b03b626dc7544c87fb6ae" Oct 30 04:18:08.466804 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.466785 5882 scope.go:122] "RemoveContainer" containerID="d1bfe7ea09cd5bb2b062280f5d486b29d82c7048a727123d33143f31495eb9ce" Oct 30 04:18:08.475737 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.475712 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:18:08.476564 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.476545 5882 scope.go:122] "RemoveContainer" containerID="d1bfe7ea09cd5bb2b062280f5d486b29d82c7048a727123d33143f31495eb9ce" Oct 30 04:18:08.476993 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:18:08.476974 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1bfe7ea09cd5bb2b062280f5d486b29d82c7048a727123d33143f31495eb9ce\": container with ID starting with d1bfe7ea09cd5bb2b062280f5d486b29d82c7048a727123d33143f31495eb9ce not found: ID does not exist" containerID="d1bfe7ea09cd5bb2b062280f5d486b29d82c7048a727123d33143f31495eb9ce" Oct 30 04:18:08.477072 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.477002 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1bfe7ea09cd5bb2b062280f5d486b29d82c7048a727123d33143f31495eb9ce"} err="failed to get container status \"d1bfe7ea09cd5bb2b062280f5d486b29d82c7048a727123d33143f31495eb9ce\": rpc error: code = NotFound desc = could not find container \"d1bfe7ea09cd5bb2b062280f5d486b29d82c7048a727123d33143f31495eb9ce\": container with ID starting with d1bfe7ea09cd5bb2b062280f5d486b29d82c7048a727123d33143f31495eb9ce not found: ID does not exist" Oct 30 04:18:08.490063 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.490035 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:18:08.519907 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.519876 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:18:08.524019 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.523993 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:18:08.561934 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.561910 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx"] Oct 30 04:18:08.567853 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:08.567829 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6999ff9b94-2t2qx"] Oct 30 04:18:10.008982 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:10.008949 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07adcc73-ec51-42e3-bf3d-115b9cbc34fd" path="/var/lib/kubelet/pods/07adcc73-ec51-42e3-bf3d-115b9cbc34fd/volumes" Oct 30 04:18:10.009344 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:10.009327 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55b27cda-7dfb-4a05-a70d-ec903c203fb5" path="/var/lib/kubelet/pods/55b27cda-7dfb-4a05-a70d-ec903c203fb5/volumes" Oct 30 04:18:10.009763 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:10.009748 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb20d78e-ce5b-415e-9510-d08fab8563a3" path="/var/lib/kubelet/pods/eb20d78e-ce5b-415e-9510-d08fab8563a3/volumes" Oct 30 04:18:12.076476 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:12.076441 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="4ef93eb8-40a1-4388-873f-e363c354abee" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.154:9093/-/ready\": dial tcp 10.134.0.154:9093: connect: connection refused" Oct 30 04:18:16.961627 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:16.961591 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:17.077064 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.077029 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ef93eb8-40a1-4388-873f-e363c354abee-tls-assets\" (UniqueName: \"kubernetes.io/projected/4ef93eb8-40a1-4388-873f-e363c354abee-tls-assets\") pod \"4ef93eb8-40a1-4388-873f-e363c354abee\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " Oct 30 04:18:17.077251 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.077074 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef93eb8-40a1-4388-873f-e363c354abee-config-out\" (UniqueName: \"kubernetes.io/empty-dir/4ef93eb8-40a1-4388-873f-e363c354abee-config-out\") pod \"4ef93eb8-40a1-4388-873f-e363c354abee\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " Oct 30 04:18:17.077251 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.077120 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef93eb8-40a1-4388-873f-e363c354abee-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4ef93eb8-40a1-4388-873f-e363c354abee-alertmanager-data-science-monitoringstack-db\") pod \"4ef93eb8-40a1-4388-873f-e363c354abee\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " Oct 30 04:18:17.077251 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.077148 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-cluster-tls-config\") pod \"4ef93eb8-40a1-4388-873f-e363c354abee\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " Oct 30 04:18:17.077251 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.077179 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-config-volume\" (UniqueName: \"kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-config-volume\") pod \"4ef93eb8-40a1-4388-873f-e363c354abee\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " Oct 30 04:18:17.077251 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.077198 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ef93eb8-40a1-4388-873f-e363c354abee-kube-api-access-6kx72\" (UniqueName: \"kubernetes.io/projected/4ef93eb8-40a1-4388-873f-e363c354abee-kube-api-access-6kx72\") pod \"4ef93eb8-40a1-4388-873f-e363c354abee\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " Oct 30 04:18:17.077251 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.077249 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-web-config\" (UniqueName: \"kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-web-config\") pod \"4ef93eb8-40a1-4388-873f-e363c354abee\" (UID: \"4ef93eb8-40a1-4388-873f-e363c354abee\") " Oct 30 04:18:17.077568 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.077509 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef93eb8-40a1-4388-873f-e363c354abee-alertmanager-data-science-monitoringstack-db" pod "4ef93eb8-40a1-4388-873f-e363c354abee" (UID: "4ef93eb8-40a1-4388-873f-e363c354abee"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:18:17.077667 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.077654 5882 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4ef93eb8-40a1-4388-873f-e363c354abee-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:17.079669 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.079593 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef93eb8-40a1-4388-873f-e363c354abee-tls-assets" pod "4ef93eb8-40a1-4388-873f-e363c354abee" (UID: "4ef93eb8-40a1-4388-873f-e363c354abee"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:17.079960 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.079936 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef93eb8-40a1-4388-873f-e363c354abee-kube-api-access-6kx72" pod "4ef93eb8-40a1-4388-873f-e363c354abee" (UID: "4ef93eb8-40a1-4388-873f-e363c354abee"). InnerVolumeSpecName "kube-api-access-6kx72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:17.080052 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.079958 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-config-volume" pod "4ef93eb8-40a1-4388-873f-e363c354abee" (UID: "4ef93eb8-40a1-4388-873f-e363c354abee"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:17.080052 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.079939 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef93eb8-40a1-4388-873f-e363c354abee-config-out" pod "4ef93eb8-40a1-4388-873f-e363c354abee" (UID: "4ef93eb8-40a1-4388-873f-e363c354abee"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:18:17.083798 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.083773 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-cluster-tls-config" pod "4ef93eb8-40a1-4388-873f-e363c354abee" (UID: "4ef93eb8-40a1-4388-873f-e363c354abee"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:17.090493 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.090470 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-web-config" pod "4ef93eb8-40a1-4388-873f-e363c354abee" (UID: "4ef93eb8-40a1-4388-873f-e363c354abee"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:17.178560 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.178530 5882 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-web-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:17.178560 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.178553 5882 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4ef93eb8-40a1-4388-873f-e363c354abee-tls-assets\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:17.178560 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.178561 5882 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4ef93eb8-40a1-4388-873f-e363c354abee-config-out\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:17.178775 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.178569 5882 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-cluster-tls-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:17.178775 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.178578 5882 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4ef93eb8-40a1-4388-873f-e363c354abee-config-volume\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:17.178775 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.178588 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kx72\" (UniqueName: \"kubernetes.io/projected/4ef93eb8-40a1-4388-873f-e363c354abee-kube-api-access-6kx72\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:17.465937 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.465845 5882 generic.go:356] "Generic (PLEG): container finished" podID="4ef93eb8-40a1-4388-873f-e363c354abee" containerID="852b203ab59f57407551343c61f931115b8548a67054b27574fe9cfdd2f3f7ce" exitCode=0 Oct 30 04:18:17.465937 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.465900 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:17.466156 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.465929 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"4ef93eb8-40a1-4388-873f-e363c354abee","Type":"ContainerDied","Data":"852b203ab59f57407551343c61f931115b8548a67054b27574fe9cfdd2f3f7ce"} Oct 30 04:18:17.466156 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.465993 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"4ef93eb8-40a1-4388-873f-e363c354abee","Type":"ContainerDied","Data":"7bb089bc15ef7db93a4332c22d8dc65a48a4484d01d3988e9d8cac2d7100f3bd"} Oct 30 04:18:17.466156 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.466021 5882 scope.go:122] "RemoveContainer" containerID="8a5854175ea2517a00d169b5409651a286660dd6bd7ab2b4d52a69e559a5c4ed" Oct 30 04:18:17.474834 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.474818 5882 scope.go:122] "RemoveContainer" containerID="852b203ab59f57407551343c61f931115b8548a67054b27574fe9cfdd2f3f7ce" Oct 30 04:18:17.482828 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.482807 5882 scope.go:122] "RemoveContainer" containerID="0a65e7fadf28c3aacff6c0b1181d436fc836ca3a40f692b429e59605fbdf4f77" Oct 30 04:18:17.490262 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.490237 5882 scope.go:122] "RemoveContainer" containerID="8a5854175ea2517a00d169b5409651a286660dd6bd7ab2b4d52a69e559a5c4ed" Oct 30 04:18:17.490559 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:18:17.490540 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a5854175ea2517a00d169b5409651a286660dd6bd7ab2b4d52a69e559a5c4ed\": container with ID starting with 8a5854175ea2517a00d169b5409651a286660dd6bd7ab2b4d52a69e559a5c4ed not found: ID does not exist" containerID="8a5854175ea2517a00d169b5409651a286660dd6bd7ab2b4d52a69e559a5c4ed" Oct 30 04:18:17.490664 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.490573 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a5854175ea2517a00d169b5409651a286660dd6bd7ab2b4d52a69e559a5c4ed"} err="failed to get container status \"8a5854175ea2517a00d169b5409651a286660dd6bd7ab2b4d52a69e559a5c4ed\": rpc error: code = NotFound desc = could not find container \"8a5854175ea2517a00d169b5409651a286660dd6bd7ab2b4d52a69e559a5c4ed\": container with ID starting with 8a5854175ea2517a00d169b5409651a286660dd6bd7ab2b4d52a69e559a5c4ed not found: ID does not exist" Oct 30 04:18:17.490664 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.490599 5882 scope.go:122] "RemoveContainer" containerID="852b203ab59f57407551343c61f931115b8548a67054b27574fe9cfdd2f3f7ce" Oct 30 04:18:17.490987 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:18:17.490968 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"852b203ab59f57407551343c61f931115b8548a67054b27574fe9cfdd2f3f7ce\": container with ID starting with 852b203ab59f57407551343c61f931115b8548a67054b27574fe9cfdd2f3f7ce not found: ID does not exist" containerID="852b203ab59f57407551343c61f931115b8548a67054b27574fe9cfdd2f3f7ce" Oct 30 04:18:17.491029 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.490993 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"852b203ab59f57407551343c61f931115b8548a67054b27574fe9cfdd2f3f7ce"} err="failed to get container status \"852b203ab59f57407551343c61f931115b8548a67054b27574fe9cfdd2f3f7ce\": rpc error: code = NotFound desc = could not find container \"852b203ab59f57407551343c61f931115b8548a67054b27574fe9cfdd2f3f7ce\": container with ID starting with 852b203ab59f57407551343c61f931115b8548a67054b27574fe9cfdd2f3f7ce not found: ID does not exist" Oct 30 04:18:17.491029 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.491008 5882 scope.go:122] "RemoveContainer" containerID="0a65e7fadf28c3aacff6c0b1181d436fc836ca3a40f692b429e59605fbdf4f77" Oct 30 04:18:17.491240 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:18:17.491219 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a65e7fadf28c3aacff6c0b1181d436fc836ca3a40f692b429e59605fbdf4f77\": container with ID starting with 0a65e7fadf28c3aacff6c0b1181d436fc836ca3a40f692b429e59605fbdf4f77 not found: ID does not exist" containerID="0a65e7fadf28c3aacff6c0b1181d436fc836ca3a40f692b429e59605fbdf4f77" Oct 30 04:18:17.491294 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.491248 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a65e7fadf28c3aacff6c0b1181d436fc836ca3a40f692b429e59605fbdf4f77"} err="failed to get container status \"0a65e7fadf28c3aacff6c0b1181d436fc836ca3a40f692b429e59605fbdf4f77\": rpc error: code = NotFound desc = could not find container \"0a65e7fadf28c3aacff6c0b1181d436fc836ca3a40f692b429e59605fbdf4f77\": container with ID starting with 0a65e7fadf28c3aacff6c0b1181d436fc836ca3a40f692b429e59605fbdf4f77 not found: ID does not exist" Oct 30 04:18:17.523074 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.523036 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:18:17.537118 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:17.537087 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:18:18.010353 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:18.010322 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ef93eb8-40a1-4388-873f-e363c354abee" path="/var/lib/kubelet/pods/4ef93eb8-40a1-4388-873f-e363c354abee/volumes" Oct 30 04:18:23.942349 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:18:23.942312 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:18:28.373909 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.373874 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb"] Oct 30 04:18:28.374373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374222 5882 cpu_manager.go:414] "containerMap: removing container" podUID="4ef93eb8-40a1-4388-873f-e363c354abee" containerName="config-reloader" Oct 30 04:18:28.374373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374233 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ef93eb8-40a1-4388-873f-e363c354abee" containerName="config-reloader" Oct 30 04:18:28.374373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374248 5882 cpu_manager.go:414] "containerMap: removing container" podUID="12128362-02d6-4179-aeeb-f328f8d7037d" containerName="ta-container" Oct 30 04:18:28.374373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374253 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12128362-02d6-4179-aeeb-f328f8d7037d" containerName="ta-container" Oct 30 04:18:28.374373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374269 5882 cpu_manager.go:414] "containerMap: removing container" podUID="eb20d78e-ce5b-415e-9510-d08fab8563a3" containerName="config-reloader" Oct 30 04:18:28.374373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374275 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb20d78e-ce5b-415e-9510-d08fab8563a3" containerName="config-reloader" Oct 30 04:18:28.374373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374282 5882 cpu_manager.go:414] "containerMap: removing container" podUID="eb20d78e-ce5b-415e-9510-d08fab8563a3" containerName="thanos-sidecar" Oct 30 04:18:28.374373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374286 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb20d78e-ce5b-415e-9510-d08fab8563a3" containerName="thanos-sidecar" Oct 30 04:18:28.374373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374297 5882 cpu_manager.go:414] "containerMap: removing container" podUID="4ef93eb8-40a1-4388-873f-e363c354abee" containerName="init-config-reloader" Oct 30 04:18:28.374373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374301 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ef93eb8-40a1-4388-873f-e363c354abee" containerName="init-config-reloader" Oct 30 04:18:28.374373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374310 5882 cpu_manager.go:414] "containerMap: removing container" podUID="eb20d78e-ce5b-415e-9510-d08fab8563a3" containerName="init-config-reloader" Oct 30 04:18:28.374373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374314 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb20d78e-ce5b-415e-9510-d08fab8563a3" containerName="init-config-reloader" Oct 30 04:18:28.374373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374327 5882 cpu_manager.go:414] "containerMap: removing container" podUID="4ef93eb8-40a1-4388-873f-e363c354abee" containerName="alertmanager" Oct 30 04:18:28.374373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374332 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ef93eb8-40a1-4388-873f-e363c354abee" containerName="alertmanager" Oct 30 04:18:28.374373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374338 5882 cpu_manager.go:414] "containerMap: removing container" podUID="eb20d78e-ce5b-415e-9510-d08fab8563a3" containerName="prometheus" Oct 30 04:18:28.374373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374343 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb20d78e-ce5b-415e-9510-d08fab8563a3" containerName="prometheus" Oct 30 04:18:28.374373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374350 5882 cpu_manager.go:414] "containerMap: removing container" podUID="55b27cda-7dfb-4a05-a70d-ec903c203fb5" containerName="ta-container" Oct 30 04:18:28.374373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374354 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55b27cda-7dfb-4a05-a70d-ec903c203fb5" containerName="ta-container" Oct 30 04:18:28.374373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374363 5882 cpu_manager.go:414] "containerMap: removing container" podUID="07adcc73-ec51-42e3-bf3d-115b9cbc34fd" containerName="perses" Oct 30 04:18:28.374373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374367 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07adcc73-ec51-42e3-bf3d-115b9cbc34fd" containerName="perses" Oct 30 04:18:28.375034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374413 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ef93eb8-40a1-4388-873f-e363c354abee" containerName="alertmanager" Oct 30 04:18:28.375034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374420 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb20d78e-ce5b-415e-9510-d08fab8563a3" containerName="prometheus" Oct 30 04:18:28.375034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374425 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="55b27cda-7dfb-4a05-a70d-ec903c203fb5" containerName="ta-container" Oct 30 04:18:28.375034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374434 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="07adcc73-ec51-42e3-bf3d-115b9cbc34fd" containerName="perses" Oct 30 04:18:28.375034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374439 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ef93eb8-40a1-4388-873f-e363c354abee" containerName="config-reloader" Oct 30 04:18:28.375034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374446 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="12128362-02d6-4179-aeeb-f328f8d7037d" containerName="ta-container" Oct 30 04:18:28.375034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374453 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb20d78e-ce5b-415e-9510-d08fab8563a3" containerName="config-reloader" Oct 30 04:18:28.375034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.374458 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb20d78e-ce5b-415e-9510-d08fab8563a3" containerName="thanos-sidecar" Oct 30 04:18:28.379136 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.379112 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb" Oct 30 04:18:28.396608 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.396577 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb"] Oct 30 04:18:28.409422 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.409402 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 04:18:28.409536 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.409434 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 04:18:28.409577 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.409558 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 04:18:28.479034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.479005 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpw9c\" (UniqueName: \"kubernetes.io/projected/90a7fc39-32e0-4216-8d81-73e822f2ba10-kube-api-access-vpw9c\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb\" (UID: \"90a7fc39-32e0-4216-8d81-73e822f2ba10\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb" Oct 30 04:18:28.580407 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.580366 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpw9c\" (UniqueName: \"kubernetes.io/projected/90a7fc39-32e0-4216-8d81-73e822f2ba10-kube-api-access-vpw9c\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb\" (UID: \"90a7fc39-32e0-4216-8d81-73e822f2ba10\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb" Oct 30 04:18:28.605477 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.605449 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpw9c\" (UniqueName: \"kubernetes.io/projected/90a7fc39-32e0-4216-8d81-73e822f2ba10-kube-api-access-vpw9c\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb\" (UID: \"90a7fc39-32e0-4216-8d81-73e822f2ba10\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb" Oct 30 04:18:28.705908 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.705836 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb" Oct 30 04:18:28.994390 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:28.994364 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb"] Oct 30 04:18:28.996350 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:18:28.996325 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90a7fc39_32e0_4216_8d81_73e822f2ba10.slice/crio-c37acfe2721881eb4daad5e1cbdcfae3e28fab6e17981ac765a677859e745556 WatchSource:0}: Error finding container c37acfe2721881eb4daad5e1cbdcfae3e28fab6e17981ac765a677859e745556: Status 404 returned error can't find the container with id c37acfe2721881eb4daad5e1cbdcfae3e28fab6e17981ac765a677859e745556 Oct 30 04:18:29.384090 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.384057 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:18:29.387630 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.387599 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:29.404523 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.404498 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:18:29.407730 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.407710 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:29.408760 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.408740 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:18:29.417827 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.417808 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:18:29.429160 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.428968 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:18:29.476810 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.476782 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:18:29.476962 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.476780 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 30 04:18:29.477031 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.476989 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8q6hm\"" Oct 30 04:18:29.477031 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.476991 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:18:29.477119 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.477066 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:18:29.477807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.477788 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl"] Oct 30 04:18:29.482839 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.482822 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" Oct 30 04:18:29.488340 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.488315 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hr5f\" (UniqueName: \"kubernetes.io/projected/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-kube-api-access-2hr5f\") pod \"data-science-collector-collector-1\" (UID: \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:29.488444 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.488426 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:29.488500 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.488486 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:29.488553 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.488513 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:29.488602 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.488567 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:29.503999 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.503969 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl"] Oct 30 04:18:29.505846 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.505827 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 30 04:18:29.505846 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.505822 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-cmvz6\"" Oct 30 04:18:29.519588 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.519560 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb" event={"ID":"90a7fc39-32e0-4216-8d81-73e822f2ba10","Type":"ContainerStarted","Data":"46874a11d2cdc4d4bf4a2c46d91d82a7c063a9dfc3c75a10110ae79dae82a969"} Oct 30 04:18:29.519707 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.519600 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb" event={"ID":"90a7fc39-32e0-4216-8d81-73e822f2ba10","Type":"ContainerStarted","Data":"c37acfe2721881eb4daad5e1cbdcfae3e28fab6e17981ac765a677859e745556"} Oct 30 04:18:29.570924 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.570872 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb" podStartSLOduration=1.570859451 podStartE2EDuration="1.570859451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:29.570217253 +0000 UTC m=+5821.684532227" watchObservedRunningTime="2026-10-30 04:18:29.570859451 +0000 UTC m=+5821.685174419" Oct 30 04:18:29.576866 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.576842 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 30 04:18:29.589232 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.589208 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:29.589367 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.589240 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:29.589367 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.589263 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/3c40df95-7863-4ebe-88e2-957f5786b42f-ta-internal\") pod \"data-science-collector-targetallocator-578bdfb4dd-gmmfl\" (UID: \"3c40df95-7863-4ebe-88e2-957f5786b42f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" Oct 30 04:18:29.589367 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.589295 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:29.589526 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.589368 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1cb2478f-259b-4eae-9fee-4d527b0c0143-config\") pod \"data-science-perses-0\" (UID: \"1cb2478f-259b-4eae-9fee-4d527b0c0143\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:29.589526 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.589470 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvq5s\" (UniqueName: \"kubernetes.io/projected/1cb2478f-259b-4eae-9fee-4d527b0c0143-kube-api-access-zvq5s\") pod \"data-science-perses-0\" (UID: \"1cb2478f-259b-4eae-9fee-4d527b0c0143\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:29.589526 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.589504 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hr5f\" (UniqueName: \"kubernetes.io/projected/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-kube-api-access-2hr5f\") pod \"data-science-collector-collector-1\" (UID: \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:29.589772 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.589581 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64hqt\" (UniqueName: \"kubernetes.io/projected/3c40df95-7863-4ebe-88e2-957f5786b42f-kube-api-access-64hqt\") pod \"data-science-collector-targetallocator-578bdfb4dd-gmmfl\" (UID: \"3c40df95-7863-4ebe-88e2-957f5786b42f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" Oct 30 04:18:29.589772 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.589611 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:29.589772 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.589695 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3c40df95-7863-4ebe-88e2-957f5786b42f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-578bdfb4dd-gmmfl\" (UID: \"3c40df95-7863-4ebe-88e2-957f5786b42f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" Oct 30 04:18:29.590007 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.589985 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/1cb2478f-259b-4eae-9fee-4d527b0c0143-plugins\") pod \"data-science-perses-0\" (UID: \"1cb2478f-259b-4eae-9fee-4d527b0c0143\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:29.590159 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.590129 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:29.592545 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.592520 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:29.592658 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.592543 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:29.592658 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.592565 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:29.615171 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.615125 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hr5f\" (UniqueName: \"kubernetes.io/projected/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-kube-api-access-2hr5f\") pod \"data-science-collector-collector-1\" (UID: \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:29.690865 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.690788 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1cb2478f-259b-4eae-9fee-4d527b0c0143-config\") pod \"data-science-perses-0\" (UID: \"1cb2478f-259b-4eae-9fee-4d527b0c0143\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:29.690865 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.690829 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvq5s\" (UniqueName: \"kubernetes.io/projected/1cb2478f-259b-4eae-9fee-4d527b0c0143-kube-api-access-zvq5s\") pod \"data-science-perses-0\" (UID: \"1cb2478f-259b-4eae-9fee-4d527b0c0143\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:29.691095 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.690879 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64hqt\" (UniqueName: \"kubernetes.io/projected/3c40df95-7863-4ebe-88e2-957f5786b42f-kube-api-access-64hqt\") pod \"data-science-collector-targetallocator-578bdfb4dd-gmmfl\" (UID: \"3c40df95-7863-4ebe-88e2-957f5786b42f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" Oct 30 04:18:29.691095 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.690912 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3c40df95-7863-4ebe-88e2-957f5786b42f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-578bdfb4dd-gmmfl\" (UID: \"3c40df95-7863-4ebe-88e2-957f5786b42f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" Oct 30 04:18:29.691095 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.690944 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/1cb2478f-259b-4eae-9fee-4d527b0c0143-plugins\") pod \"data-science-perses-0\" (UID: \"1cb2478f-259b-4eae-9fee-4d527b0c0143\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:29.691095 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.691006 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/3c40df95-7863-4ebe-88e2-957f5786b42f-ta-internal\") pod \"data-science-collector-targetallocator-578bdfb4dd-gmmfl\" (UID: \"3c40df95-7863-4ebe-88e2-957f5786b42f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" Oct 30 04:18:29.691408 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.691379 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/1cb2478f-259b-4eae-9fee-4d527b0c0143-plugins\") pod \"data-science-perses-0\" (UID: \"1cb2478f-259b-4eae-9fee-4d527b0c0143\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:29.691635 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.691595 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1cb2478f-259b-4eae-9fee-4d527b0c0143-config\") pod \"data-science-perses-0\" (UID: \"1cb2478f-259b-4eae-9fee-4d527b0c0143\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:29.691743 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.691725 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/3c40df95-7863-4ebe-88e2-957f5786b42f-ta-internal\") pod \"data-science-collector-targetallocator-578bdfb4dd-gmmfl\" (UID: \"3c40df95-7863-4ebe-88e2-957f5786b42f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" Oct 30 04:18:29.693410 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.693386 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3c40df95-7863-4ebe-88e2-957f5786b42f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-578bdfb4dd-gmmfl\" (UID: \"3c40df95-7863-4ebe-88e2-957f5786b42f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" Oct 30 04:18:29.700228 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.700207 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:29.716656 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.716595 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64hqt\" (UniqueName: \"kubernetes.io/projected/3c40df95-7863-4ebe-88e2-957f5786b42f-kube-api-access-64hqt\") pod \"data-science-collector-targetallocator-578bdfb4dd-gmmfl\" (UID: \"3c40df95-7863-4ebe-88e2-957f5786b42f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" Oct 30 04:18:29.716801 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.716772 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvq5s\" (UniqueName: \"kubernetes.io/projected/1cb2478f-259b-4eae-9fee-4d527b0c0143-kube-api-access-zvq5s\") pod \"data-science-perses-0\" (UID: \"1cb2478f-259b-4eae-9fee-4d527b0c0143\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:29.796150 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.796119 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" Oct 30 04:18:29.999221 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:29.999190 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:18:30.000110 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:18:30.000080 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd52c18_cf38_40b7_a37c_f0e1ef1e69d6.slice/crio-f550e4a9f7ba9bcf42b83b06845b380cdaac1d867e038050b8e2f42021d35113 WatchSource:0}: Error finding container f550e4a9f7ba9bcf42b83b06845b380cdaac1d867e038050b8e2f42021d35113: Status 404 returned error can't find the container with id f550e4a9f7ba9bcf42b83b06845b380cdaac1d867e038050b8e2f42021d35113 Oct 30 04:18:30.002282 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:30.002248 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:18:30.002397 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:30.002320 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:18:30.088477 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:30.088449 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl"] Oct 30 04:18:30.090080 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:18:30.090054 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c40df95_7863_4ebe_88e2_957f5786b42f.slice/crio-cc964ee1f91a2826709521feefbab75e4b914a7f4b37c9deb610a6405431da51 WatchSource:0}: Error finding container cc964ee1f91a2826709521feefbab75e4b914a7f4b37c9deb610a6405431da51: Status 404 returned error can't find the container with id cc964ee1f91a2826709521feefbab75e4b914a7f4b37c9deb610a6405431da51 Oct 30 04:18:30.092512 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:30.092479 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:18:30.092594 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:30.092539 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:18:30.525512 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:30.525480 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" event={"ID":"3c40df95-7863-4ebe-88e2-957f5786b42f","Type":"ContainerStarted","Data":"ac66aeaaaf80ee7b39bd1b31d11252096527bba13c3edf520aafa490ff9c37e1"} Oct 30 04:18:30.525978 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:30.525519 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" event={"ID":"3c40df95-7863-4ebe-88e2-957f5786b42f","Type":"ContainerStarted","Data":"cc964ee1f91a2826709521feefbab75e4b914a7f4b37c9deb610a6405431da51"} Oct 30 04:18:30.525978 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:30.525678 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" Oct 30 04:18:30.526991 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:30.526956 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" podUID="3c40df95-7863-4ebe-88e2-957f5786b42f" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.163:8080/readyz\": dial tcp 10.134.0.163:8080: connect: connection refused" Oct 30 04:18:30.527463 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:30.527434 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6","Type":"ContainerStarted","Data":"c6163facf9e72b28df062089f8a38071da5ffcbd24c420eaf443a4583aa9d2d8"} Oct 30 04:18:30.527568 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:30.527480 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6","Type":"ContainerStarted","Data":"f550e4a9f7ba9bcf42b83b06845b380cdaac1d867e038050b8e2f42021d35113"} Oct 30 04:18:30.571001 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:30.570952 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" podStartSLOduration=1.5709391190000002 podStartE2EDuration="1.570939119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:30.57079646 +0000 UTC m=+5822.685111436" watchObservedRunningTime="2026-10-30 04:18:30.570939119 +0000 UTC m=+5822.685254093" Oct 30 04:18:30.615664 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:30.615597 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.6155846139999999 podStartE2EDuration="1.615584614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:30.614891945 +0000 UTC m=+5822.729206923" watchObservedRunningTime="2026-10-30 04:18:30.615584614 +0000 UTC m=+5822.729899585" Oct 30 04:18:31.534008 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:31.533975 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" Oct 30 04:18:31.912891 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:31.912815 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"1cb2478f-259b-4eae-9fee-4d527b0c0143\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:31.946152 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:31.946120 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"1cb2478f-259b-4eae-9fee-4d527b0c0143\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:32.014036 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:32.014006 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"1cb2478f-259b-4eae-9fee-4d527b0c0143\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:32.014176 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:32.014140 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"1cb2478f-259b-4eae-9fee-4d527b0c0143\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:32.030032 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:32.030009 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"1cb2478f-259b-4eae-9fee-4d527b0c0143\") DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:32.133667 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:32.133632 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"1cb2478f-259b-4eae-9fee-4d527b0c0143\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2c422b71c1e0b8fdf34659f2cef7846e06b4353ed88027dc99ade5d7fc29e9d6/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:32.166714 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:32.166654 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"1cb2478f-259b-4eae-9fee-4d527b0c0143\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:32.421497 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:32.421414 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:32.714954 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:32.714920 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:18:32.715771 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:18:32.715743 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cb2478f_259b_4eae_9fee_4d527b0c0143.slice/crio-6e2a5459a2687b0506bed23575dc41c84ab5c4850e93cc3f4787b1677d37c632 WatchSource:0}: Error finding container 6e2a5459a2687b0506bed23575dc41c84ab5c4850e93cc3f4787b1677d37c632: Status 404 returned error can't find the container with id 6e2a5459a2687b0506bed23575dc41c84ab5c4850e93cc3f4787b1677d37c632 Oct 30 04:18:32.843046 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:32.843011 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb"] Oct 30 04:18:32.843204 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:32.843185 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb" podUID="90a7fc39-32e0-4216-8d81-73e822f2ba10" containerName="thanos-querier" containerID="cri-o://46874a11d2cdc4d4bf4a2c46d91d82a7c063a9dfc3c75a10110ae79dae82a969" gracePeriod=30 Oct 30 04:18:33.078675 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.078078 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:18:33.085967 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.085939 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.091219 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.091143 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb" Oct 30 04:18:33.115395 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.115304 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 04:18:33.115395 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.115332 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:18:33.123403 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.123375 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:18:33.123403 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.123377 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 04:18:33.123566 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.123378 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 04:18:33.123566 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.123432 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-47r6j\"" Oct 30 04:18:33.128779 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.128746 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:18:33.129180 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.129160 5882 cpu_manager.go:414] "containerMap: removing container" podUID="90a7fc39-32e0-4216-8d81-73e822f2ba10" containerName="thanos-querier" Oct 30 04:18:33.129325 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.129309 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a7fc39-32e0-4216-8d81-73e822f2ba10" containerName="thanos-querier" Oct 30 04:18:33.129550 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.129528 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a7fc39-32e0-4216-8d81-73e822f2ba10" containerName="thanos-querier" Oct 30 04:18:33.133291 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.133270 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.165244 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.165212 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:18:33.209836 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.209744 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 04:18:33.209836 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.209749 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 04:18:33.209836 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.209797 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 04:18:33.210141 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.209860 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 04:18:33.210141 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.209914 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-7vtv5\"" Oct 30 04:18:33.210141 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.209921 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 04:18:33.210141 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.209931 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:18:33.215334 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.215271 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 04:18:33.215334 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.215310 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 04:18:33.215545 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.215529 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 04:18:33.228355 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.228327 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a7fc39-32e0-4216-8d81-73e822f2ba10-kube-api-access-vpw9c\" (UniqueName: \"kubernetes.io/projected/90a7fc39-32e0-4216-8d81-73e822f2ba10-kube-api-access-vpw9c\") pod \"90a7fc39-32e0-4216-8d81-73e822f2ba10\" (UID: \"90a7fc39-32e0-4216-8d81-73e822f2ba10\") " Oct 30 04:18:33.228470 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.228460 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.228569 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.228484 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrk55\" (UniqueName: \"kubernetes.io/projected/0cd1858d-e47d-4405-ad1d-33b83b43846d-kube-api-access-jrk55\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.228569 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.228505 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0cd1858d-e47d-4405-ad1d-33b83b43846d-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.228730 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.228563 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.228730 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.228640 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/78072d40-0ad6-4a50-a475-652f1cdd980c-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.228874 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.228778 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78072d40-0ad6-4a50-a475-652f1cdd980c-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.228874 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.228812 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.228874 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.228853 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.229027 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.228883 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78072d40-0ad6-4a50-a475-652f1cdd980c-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.229027 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.228912 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.229027 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.228942 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0cd1858d-e47d-4405-ad1d-33b83b43846d-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.229027 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.228973 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.229027 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.229012 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpzj5\" (UniqueName: \"kubernetes.io/projected/78072d40-0ad6-4a50-a475-652f1cdd980c-kube-api-access-lpzj5\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.229346 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.229051 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.229346 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.229078 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.229346 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.229110 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.229346 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.229149 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.229346 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.229200 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.229346 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.229235 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.230746 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.230719 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a7fc39-32e0-4216-8d81-73e822f2ba10-kube-api-access-vpw9c" pod "90a7fc39-32e0-4216-8d81-73e822f2ba10" (UID: "90a7fc39-32e0-4216-8d81-73e822f2ba10"). InnerVolumeSpecName "kube-api-access-vpw9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:33.330219 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.330182 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78072d40-0ad6-4a50-a475-652f1cdd980c-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.330413 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.330232 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.330413 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.330254 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0cd1858d-e47d-4405-ad1d-33b83b43846d-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.330413 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.330274 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.330592 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.330434 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpzj5\" (UniqueName: \"kubernetes.io/projected/78072d40-0ad6-4a50-a475-652f1cdd980c-kube-api-access-lpzj5\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.330592 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.330511 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.330592 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.330544 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.330592 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.330578 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.330824 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.330632 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.330824 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.330681 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.330824 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.330719 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.330824 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.330757 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.330824 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.330784 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrk55\" (UniqueName: \"kubernetes.io/projected/0cd1858d-e47d-4405-ad1d-33b83b43846d-kube-api-access-jrk55\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.330824 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.330811 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0cd1858d-e47d-4405-ad1d-33b83b43846d-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.331141 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.330838 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.331141 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.330865 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/78072d40-0ad6-4a50-a475-652f1cdd980c-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.331141 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.330957 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78072d40-0ad6-4a50-a475-652f1cdd980c-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.331141 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.330985 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.331141 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.331033 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.331141 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.331095 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpw9c\" (UniqueName: \"kubernetes.io/projected/90a7fc39-32e0-4216-8d81-73e822f2ba10-kube-api-access-vpw9c\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:33.331466 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.331288 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.331466 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.331397 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.332753 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.331710 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/78072d40-0ad6-4a50-a475-652f1cdd980c-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.332753 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.332281 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.335148 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.334429 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78072d40-0ad6-4a50-a475-652f1cdd980c-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.335148 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.335090 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.335688 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.335665 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.336832 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.336801 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78072d40-0ad6-4a50-a475-652f1cdd980c-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.337035 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.337008 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.337126 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.337026 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.337186 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.337161 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.337452 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.337429 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.337519 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.337475 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.337963 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.337937 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0cd1858d-e47d-4405-ad1d-33b83b43846d-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.338060 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.337982 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.338060 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.338043 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0cd1858d-e47d-4405-ad1d-33b83b43846d-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.338316 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.338295 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.356240 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.356212 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpzj5\" (UniqueName: \"kubernetes.io/projected/78072d40-0ad6-4a50-a475-652f1cdd980c-kube-api-access-lpzj5\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.356946 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.356922 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrk55\" (UniqueName: \"kubernetes.io/projected/0cd1858d-e47d-4405-ad1d-33b83b43846d-kube-api-access-jrk55\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:33.415597 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.415565 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:33.543260 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.543218 5882 generic.go:356] "Generic (PLEG): container finished" podID="90a7fc39-32e0-4216-8d81-73e822f2ba10" containerID="46874a11d2cdc4d4bf4a2c46d91d82a7c063a9dfc3c75a10110ae79dae82a969" exitCode=0 Oct 30 04:18:33.543430 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.543266 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb" Oct 30 04:18:33.543430 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.543313 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb" event={"ID":"90a7fc39-32e0-4216-8d81-73e822f2ba10","Type":"ContainerDied","Data":"46874a11d2cdc4d4bf4a2c46d91d82a7c063a9dfc3c75a10110ae79dae82a969"} Oct 30 04:18:33.543430 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.543352 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb" event={"ID":"90a7fc39-32e0-4216-8d81-73e822f2ba10","Type":"ContainerDied","Data":"c37acfe2721881eb4daad5e1cbdcfae3e28fab6e17981ac765a677859e745556"} Oct 30 04:18:33.543430 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.543369 5882 scope.go:122] "RemoveContainer" containerID="46874a11d2cdc4d4bf4a2c46d91d82a7c063a9dfc3c75a10110ae79dae82a969" Oct 30 04:18:33.545084 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.545064 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"1cb2478f-259b-4eae-9fee-4d527b0c0143","Type":"ContainerStarted","Data":"bfcf280dbf506612a9062e5fc017fe9724a4bb036b3b498daf2cdcb38971a63a"} Oct 30 04:18:33.545201 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.545094 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"1cb2478f-259b-4eae-9fee-4d527b0c0143","Type":"ContainerStarted","Data":"6e2a5459a2687b0506bed23575dc41c84ab5c4850e93cc3f4787b1677d37c632"} Oct 30 04:18:33.545265 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.545221 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:33.553265 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.553239 5882 scope.go:122] "RemoveContainer" containerID="46874a11d2cdc4d4bf4a2c46d91d82a7c063a9dfc3c75a10110ae79dae82a969" Oct 30 04:18:33.553576 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:18:33.553551 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46874a11d2cdc4d4bf4a2c46d91d82a7c063a9dfc3c75a10110ae79dae82a969\": container with ID starting with 46874a11d2cdc4d4bf4a2c46d91d82a7c063a9dfc3c75a10110ae79dae82a969 not found: ID does not exist" containerID="46874a11d2cdc4d4bf4a2c46d91d82a7c063a9dfc3c75a10110ae79dae82a969" Oct 30 04:18:33.553683 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.553587 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46874a11d2cdc4d4bf4a2c46d91d82a7c063a9dfc3c75a10110ae79dae82a969"} err="failed to get container status \"46874a11d2cdc4d4bf4a2c46d91d82a7c063a9dfc3c75a10110ae79dae82a969\": rpc error: code = NotFound desc = could not find container \"46874a11d2cdc4d4bf4a2c46d91d82a7c063a9dfc3c75a10110ae79dae82a969\": container with ID starting with 46874a11d2cdc4d4bf4a2c46d91d82a7c063a9dfc3c75a10110ae79dae82a969 not found: ID does not exist" Oct 30 04:18:33.599338 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.599307 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb"] Oct 30 04:18:33.605309 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.605281 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-95sxb"] Oct 30 04:18:33.649467 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.649404 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=4.649390777 podStartE2EDuration="4.649390777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:33.648463609 +0000 UTC m=+5825.762778584" watchObservedRunningTime="2026-10-30 04:18:33.649390777 +0000 UTC m=+5825.763705742" Oct 30 04:18:33.727892 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:33.727855 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:18:33.728346 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:18:33.728324 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78072d40_0ad6_4a50_a475_652f1cdd980c.slice/crio-70b99802ab48ba5198612a9475c9c29b9fcadc03146d8d8872313a51285f4dac WatchSource:0}: Error finding container 70b99802ab48ba5198612a9475c9c29b9fcadc03146d8d8872313a51285f4dac: Status 404 returned error can't find the container with id 70b99802ab48ba5198612a9475c9c29b9fcadc03146d8d8872313a51285f4dac Oct 30 04:18:34.010412 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:34.010377 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a7fc39-32e0-4216-8d81-73e822f2ba10" path="/var/lib/kubelet/pods/90a7fc39-32e0-4216-8d81-73e822f2ba10/volumes" Oct 30 04:18:34.552844 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:34.552804 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"78072d40-0ad6-4a50-a475-652f1cdd980c","Type":"ContainerStarted","Data":"a8ffe0b26f33fc7efdbc82e449b77900c376b8f576ee55e5f60c3d03f8221666"} Oct 30 04:18:34.553041 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:34.552880 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"78072d40-0ad6-4a50-a475-652f1cdd980c","Type":"ContainerStarted","Data":"70b99802ab48ba5198612a9475c9c29b9fcadc03146d8d8872313a51285f4dac"} Oct 30 04:18:37.567993 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:37.567957 5882 generic.go:356] "Generic (PLEG): container finished" podID="78072d40-0ad6-4a50-a475-652f1cdd980c" containerID="a8ffe0b26f33fc7efdbc82e449b77900c376b8f576ee55e5f60c3d03f8221666" exitCode=0 Oct 30 04:18:37.568418 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:37.568034 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"78072d40-0ad6-4a50-a475-652f1cdd980c","Type":"ContainerDied","Data":"a8ffe0b26f33fc7efdbc82e449b77900c376b8f576ee55e5f60c3d03f8221666"} Oct 30 04:18:38.573818 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:38.573786 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"78072d40-0ad6-4a50-a475-652f1cdd980c","Type":"ContainerStarted","Data":"40e1f66f8ea505c961be3e318f880a878065f6ecc44272606856155b49e76036"} Oct 30 04:18:38.574228 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:38.573845 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"78072d40-0ad6-4a50-a475-652f1cdd980c","Type":"ContainerStarted","Data":"f9b902aa5911d56fda79ee288d89f26b017953d51dc7ad7866a9851521a6e0b7"} Oct 30 04:18:38.574228 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:38.574016 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:38.652776 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:38.652724 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=2.366124306 podStartE2EDuration="5.652711224s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.286586918s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:38.638061196 +0000 UTC m=+5830.752376174" watchObservedRunningTime="2026-10-30 04:18:38.652711224 +0000 UTC m=+5830.767026193" Oct 30 04:18:40.804826 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:40.804787 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:40.837310 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:40.837282 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:40.906257 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:40.906225 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:40.906384 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:40.906354 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:40.922339 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:40.922309 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") DevicePath \"csi-927d18ef2eb99137c68fbe5e0bb8ea629bb5219ed56cdede72da7c0f045be95e\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:41.027248 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:41.027207 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a535375d22e374ba333968465471c7604a06f9d95a513910fe4bb083b629e986/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:41.055883 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:41.055827 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:41.258043 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:41.258008 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:41.581764 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:41.581733 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:18:41.582416 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:18:41.582385 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd1858d_e47d_4405_ad1d_33b83b43846d.slice/crio-9b40d3e7d97c218125156632477bf8eefbf45002e115fc0affd58b8888baf078 WatchSource:0}: Error finding container 9b40d3e7d97c218125156632477bf8eefbf45002e115fc0affd58b8888baf078: Status 404 returned error can't find the container with id 9b40d3e7d97c218125156632477bf8eefbf45002e115fc0affd58b8888baf078 Oct 30 04:18:41.589477 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:41.589434 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0cd1858d-e47d-4405-ad1d-33b83b43846d","Type":"ContainerStarted","Data":"9b40d3e7d97c218125156632477bf8eefbf45002e115fc0affd58b8888baf078"} Oct 30 04:18:42.595771 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:42.595732 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0cd1858d-e47d-4405-ad1d-33b83b43846d","Type":"ContainerStarted","Data":"c68f4fd6143ccce635e4e812c6c4d556df3c2e10a791599b1041ca3dcac1035c"} Oct 30 04:18:44.556195 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:44.556165 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:18:44.579835 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:44.579797 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:18:45.611130 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:45.611032 5882 generic.go:356] "Generic (PLEG): container finished" podID="0cd1858d-e47d-4405-ad1d-33b83b43846d" containerID="c68f4fd6143ccce635e4e812c6c4d556df3c2e10a791599b1041ca3dcac1035c" exitCode=0 Oct 30 04:18:45.611130 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:45.611107 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0cd1858d-e47d-4405-ad1d-33b83b43846d","Type":"ContainerDied","Data":"c68f4fd6143ccce635e4e812c6c4d556df3c2e10a791599b1041ca3dcac1035c"} Oct 30 04:18:46.617595 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:46.617555 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0cd1858d-e47d-4405-ad1d-33b83b43846d","Type":"ContainerStarted","Data":"12ae11190d08e587d760a511d365d63a23dcdc4ef34628d784a3816eaeafbdda"} Oct 30 04:18:46.617996 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:46.617682 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0cd1858d-e47d-4405-ad1d-33b83b43846d","Type":"ContainerStarted","Data":"0cac6c3f5f51df269857622a1bbc8d13606d4e9a66a685edf78212f17e558188"} Oct 30 04:18:46.617996 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:46.617709 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0cd1858d-e47d-4405-ad1d-33b83b43846d","Type":"ContainerStarted","Data":"e2fe712d1b3582d040888a7389a8a7c1db2ea319bb6bdf53226531ff9b6620bc"} Oct 30 04:18:46.700525 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:46.700473 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=10.401136372 podStartE2EDuration="13.700460124s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.299323752s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:46.686202978 +0000 UTC m=+5838.800517953" watchObservedRunningTime="2026-10-30 04:18:46.700460124 +0000 UTC m=+5838.814775096" Oct 30 04:18:50.497251 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.497209 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:18:50.497704 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.497417 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6" containerName="otc-container" containerID="cri-o://c6163facf9e72b28df062089f8a38071da5ffcbd24c420eaf443a4583aa9d2d8" gracePeriod=30 Oct 30 04:18:50.637025 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.636994 5882 generic.go:356] "Generic (PLEG): container finished" podID="bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6" containerID="c6163facf9e72b28df062089f8a38071da5ffcbd24c420eaf443a4583aa9d2d8" exitCode=0 Oct 30 04:18:50.637163 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.637041 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6","Type":"ContainerDied","Data":"c6163facf9e72b28df062089f8a38071da5ffcbd24c420eaf443a4583aa9d2d8"} Oct 30 04:18:50.728469 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.728440 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv"] Oct 30 04:18:50.732134 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.732114 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" Oct 30 04:18:50.747728 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.747667 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:50.753869 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.753833 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv"] Oct 30 04:18:50.798357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.798320 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b795fd60-5d7d-43aa-96ec-4b130206e6f1-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-74795d8b78-glmjv\" (UID: \"b795fd60-5d7d-43aa-96ec-4b130206e6f1\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" Oct 30 04:18:50.798515 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.798427 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r229\" (UniqueName: \"kubernetes.io/projected/b795fd60-5d7d-43aa-96ec-4b130206e6f1-kube-api-access-8r229\") pod \"data-science-collector-targetallocator-74795d8b78-glmjv\" (UID: \"b795fd60-5d7d-43aa-96ec-4b130206e6f1\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" Oct 30 04:18:50.798560 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.798517 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b795fd60-5d7d-43aa-96ec-4b130206e6f1-ta-internal\") pod \"data-science-collector-targetallocator-74795d8b78-glmjv\" (UID: \"b795fd60-5d7d-43aa-96ec-4b130206e6f1\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" Oct 30 04:18:50.899818 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.899787 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-otc-internal\" (UniqueName: \"kubernetes.io/configmap/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-otc-internal\") pod \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\" (UID: \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\") " Oct 30 04:18:50.899987 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.899841 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-data-science-collector-ta-client-cert\") pod \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\" (UID: \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\") " Oct 30 04:18:50.899987 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.899918 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-tls-certs\") pod \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\" (UID: \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\") " Oct 30 04:18:50.899987 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.899962 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-kube-api-access-2hr5f\" (UniqueName: \"kubernetes.io/projected/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-kube-api-access-2hr5f\") pod \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\" (UID: \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\") " Oct 30 04:18:50.899987 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.899985 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-monitoring-tls-certs\") pod \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\" (UID: \"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6\") " Oct 30 04:18:50.900289 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.900117 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r229\" (UniqueName: \"kubernetes.io/projected/b795fd60-5d7d-43aa-96ec-4b130206e6f1-kube-api-access-8r229\") pod \"data-science-collector-targetallocator-74795d8b78-glmjv\" (UID: \"b795fd60-5d7d-43aa-96ec-4b130206e6f1\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" Oct 30 04:18:50.900289 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.900193 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-otc-internal" pod "bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6" (UID: "bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:18:50.900289 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.900235 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b795fd60-5d7d-43aa-96ec-4b130206e6f1-ta-internal\") pod \"data-science-collector-targetallocator-74795d8b78-glmjv\" (UID: \"b795fd60-5d7d-43aa-96ec-4b130206e6f1\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" Oct 30 04:18:50.900289 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.900262 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b795fd60-5d7d-43aa-96ec-4b130206e6f1-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-74795d8b78-glmjv\" (UID: \"b795fd60-5d7d-43aa-96ec-4b130206e6f1\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" Oct 30 04:18:50.900522 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.900346 5882 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-otc-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:50.901166 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.901136 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b795fd60-5d7d-43aa-96ec-4b130206e6f1-ta-internal\") pod \"data-science-collector-targetallocator-74795d8b78-glmjv\" (UID: \"b795fd60-5d7d-43aa-96ec-4b130206e6f1\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" Oct 30 04:18:50.902157 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.902134 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-data-science-collector-ta-client-cert" pod "bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6" (UID: "bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:50.902294 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.902263 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-kube-api-access-2hr5f" pod "bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6" (UID: "bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6"). InnerVolumeSpecName "kube-api-access-2hr5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:50.902602 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.902584 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-tls-certs" pod "bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6" (UID: "bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:50.902704 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.902605 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-monitoring-tls-certs" pod "bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6" (UID: "bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:50.902840 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.902823 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b795fd60-5d7d-43aa-96ec-4b130206e6f1-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-74795d8b78-glmjv\" (UID: \"b795fd60-5d7d-43aa-96ec-4b130206e6f1\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" Oct 30 04:18:50.927129 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:50.927097 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r229\" (UniqueName: \"kubernetes.io/projected/b795fd60-5d7d-43aa-96ec-4b130206e6f1-kube-api-access-8r229\") pod \"data-science-collector-targetallocator-74795d8b78-glmjv\" (UID: \"b795fd60-5d7d-43aa-96ec-4b130206e6f1\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" Oct 30 04:18:51.001772 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:51.001689 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:51.001772 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:51.001718 5882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:51.001772 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:51.001731 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hr5f\" (UniqueName: \"kubernetes.io/projected/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-kube-api-access-2hr5f\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:51.001772 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:51.001744 5882 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6-monitoring-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:51.055572 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:51.055539 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" Oct 30 04:18:51.259106 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:51.259032 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:51.349407 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:51.349380 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv"] Oct 30 04:18:51.350403 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:18:51.350372 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb795fd60_5d7d_43aa_96ec_4b130206e6f1.slice/crio-8db95ef7d80a5a6922b631255db0379f65362ed935aa871f77e6971652d5ecec WatchSource:0}: Error finding container 8db95ef7d80a5a6922b631255db0379f65362ed935aa871f77e6971652d5ecec: Status 404 returned error can't find the container with id 8db95ef7d80a5a6922b631255db0379f65362ed935aa871f77e6971652d5ecec Oct 30 04:18:51.352538 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:51.352507 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:18:51.352633 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:51.352560 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:18:51.642719 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:51.642642 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:51.643186 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:51.642639 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6","Type":"ContainerDied","Data":"f550e4a9f7ba9bcf42b83b06845b380cdaac1d867e038050b8e2f42021d35113"} Oct 30 04:18:51.643186 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:51.642783 5882 scope.go:122] "RemoveContainer" containerID="c6163facf9e72b28df062089f8a38071da5ffcbd24c420eaf443a4583aa9d2d8" Oct 30 04:18:51.644385 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:51.644363 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" event={"ID":"b795fd60-5d7d-43aa-96ec-4b130206e6f1","Type":"ContainerStarted","Data":"8c98779c2f950e8691c1a86140ddd101cdc0b7ef0a8403abe69724552e9ee2d1"} Oct 30 04:18:51.644487 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:51.644398 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" event={"ID":"b795fd60-5d7d-43aa-96ec-4b130206e6f1","Type":"ContainerStarted","Data":"8db95ef7d80a5a6922b631255db0379f65362ed935aa871f77e6971652d5ecec"} Oct 30 04:18:51.644595 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:51.644578 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" Oct 30 04:18:51.645611 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:51.645587 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" podUID="b795fd60-5d7d-43aa-96ec-4b130206e6f1" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.166:8080/readyz\": dial tcp 10.134.0.166:8080: connect: connection refused" Oct 30 04:18:51.704337 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:51.704291 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" podStartSLOduration=1.7042789630000001 podStartE2EDuration="1.704278963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:51.703170696 +0000 UTC m=+5843.817485682" watchObservedRunningTime="2026-10-30 04:18:51.704278963 +0000 UTC m=+5843.818594018" Oct 30 04:18:51.742014 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:51.741987 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:18:51.748189 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:51.748157 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:18:52.009105 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:52.009074 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6" path="/var/lib/kubelet/pods/bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6/volumes" Oct 30 04:18:52.651497 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:52.651422 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" Oct 30 04:18:52.728171 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:52.728142 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl"] Oct 30 04:18:52.728397 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:52.728372 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" podUID="3c40df95-7863-4ebe-88e2-957f5786b42f" containerName="ta-container" containerID="cri-o://ac66aeaaaf80ee7b39bd1b31d11252096527bba13c3edf520aafa490ff9c37e1" gracePeriod=30 Oct 30 04:18:52.972698 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:52.972673 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" Oct 30 04:18:53.019665 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:53.019609 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c40df95-7863-4ebe-88e2-957f5786b42f-ta-internal\" (UniqueName: \"kubernetes.io/configmap/3c40df95-7863-4ebe-88e2-957f5786b42f-ta-internal\") pod \"3c40df95-7863-4ebe-88e2-957f5786b42f\" (UID: \"3c40df95-7863-4ebe-88e2-957f5786b42f\") " Oct 30 04:18:53.019832 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:53.019704 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c40df95-7863-4ebe-88e2-957f5786b42f-kube-api-access-64hqt\" (UniqueName: \"kubernetes.io/projected/3c40df95-7863-4ebe-88e2-957f5786b42f-kube-api-access-64hqt\") pod \"3c40df95-7863-4ebe-88e2-957f5786b42f\" (UID: \"3c40df95-7863-4ebe-88e2-957f5786b42f\") " Oct 30 04:18:53.019832 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:53.019738 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c40df95-7863-4ebe-88e2-957f5786b42f-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3c40df95-7863-4ebe-88e2-957f5786b42f-data-science-collector-ta-server-cert\") pod \"3c40df95-7863-4ebe-88e2-957f5786b42f\" (UID: \"3c40df95-7863-4ebe-88e2-957f5786b42f\") " Oct 30 04:18:53.020023 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:53.020001 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c40df95-7863-4ebe-88e2-957f5786b42f-ta-internal" pod "3c40df95-7863-4ebe-88e2-957f5786b42f" (UID: "3c40df95-7863-4ebe-88e2-957f5786b42f"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:18:53.021753 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:53.021726 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c40df95-7863-4ebe-88e2-957f5786b42f-kube-api-access-64hqt" pod "3c40df95-7863-4ebe-88e2-957f5786b42f" (UID: "3c40df95-7863-4ebe-88e2-957f5786b42f"). InnerVolumeSpecName "kube-api-access-64hqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:53.021961 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:53.021937 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c40df95-7863-4ebe-88e2-957f5786b42f-data-science-collector-ta-server-cert" pod "3c40df95-7863-4ebe-88e2-957f5786b42f" (UID: "3c40df95-7863-4ebe-88e2-957f5786b42f"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:53.120946 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:53.120910 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/3c40df95-7863-4ebe-88e2-957f5786b42f-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:53.120946 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:53.120942 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64hqt\" (UniqueName: \"kubernetes.io/projected/3c40df95-7863-4ebe-88e2-957f5786b42f-kube-api-access-64hqt\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:53.120946 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:53.120952 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3c40df95-7863-4ebe-88e2-957f5786b42f-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:18:53.655574 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:53.655534 5882 generic.go:356] "Generic (PLEG): container finished" podID="3c40df95-7863-4ebe-88e2-957f5786b42f" containerID="ac66aeaaaf80ee7b39bd1b31d11252096527bba13c3edf520aafa490ff9c37e1" exitCode=0 Oct 30 04:18:53.656034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:53.655584 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" Oct 30 04:18:53.656034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:53.655638 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" event={"ID":"3c40df95-7863-4ebe-88e2-957f5786b42f","Type":"ContainerDied","Data":"ac66aeaaaf80ee7b39bd1b31d11252096527bba13c3edf520aafa490ff9c37e1"} Oct 30 04:18:53.656034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:53.655684 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl" event={"ID":"3c40df95-7863-4ebe-88e2-957f5786b42f","Type":"ContainerDied","Data":"cc964ee1f91a2826709521feefbab75e4b914a7f4b37c9deb610a6405431da51"} Oct 30 04:18:53.656034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:53.655702 5882 scope.go:122] "RemoveContainer" containerID="ac66aeaaaf80ee7b39bd1b31d11252096527bba13c3edf520aafa490ff9c37e1" Oct 30 04:18:53.664607 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:53.664590 5882 scope.go:122] "RemoveContainer" containerID="ac66aeaaaf80ee7b39bd1b31d11252096527bba13c3edf520aafa490ff9c37e1" Oct 30 04:18:53.664897 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:18:53.664875 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac66aeaaaf80ee7b39bd1b31d11252096527bba13c3edf520aafa490ff9c37e1\": container with ID starting with ac66aeaaaf80ee7b39bd1b31d11252096527bba13c3edf520aafa490ff9c37e1 not found: ID does not exist" containerID="ac66aeaaaf80ee7b39bd1b31d11252096527bba13c3edf520aafa490ff9c37e1" Oct 30 04:18:53.664985 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:53.664905 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac66aeaaaf80ee7b39bd1b31d11252096527bba13c3edf520aafa490ff9c37e1"} err="failed to get container status \"ac66aeaaaf80ee7b39bd1b31d11252096527bba13c3edf520aafa490ff9c37e1\": rpc error: code = NotFound desc = could not find container \"ac66aeaaaf80ee7b39bd1b31d11252096527bba13c3edf520aafa490ff9c37e1\": container with ID starting with ac66aeaaaf80ee7b39bd1b31d11252096527bba13c3edf520aafa490ff9c37e1 not found: ID does not exist" Oct 30 04:18:53.710482 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:53.710449 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl"] Oct 30 04:18:53.724441 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:53.724416 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-578bdfb4dd-gmmfl"] Oct 30 04:18:53.943376 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:18:53.943296 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:18:54.009544 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:54.009512 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c40df95-7863-4ebe-88e2-957f5786b42f" path="/var/lib/kubelet/pods/3c40df95-7863-4ebe-88e2-957f5786b42f/volumes" Oct 30 04:18:56.259064 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:56.259032 5882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:56.264264 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:56.264241 5882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:18:56.674049 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:18:56.673972 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:06.271406 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.271367 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9"] Oct 30 04:19:06.271818 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.271774 5882 cpu_manager.go:414] "containerMap: removing container" podUID="3c40df95-7863-4ebe-88e2-957f5786b42f" containerName="ta-container" Oct 30 04:19:06.271818 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.271786 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c40df95-7863-4ebe-88e2-957f5786b42f" containerName="ta-container" Oct 30 04:19:06.271818 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.271808 5882 cpu_manager.go:414] "containerMap: removing container" podUID="bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6" containerName="otc-container" Oct 30 04:19:06.271818 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.271813 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6" containerName="otc-container" Oct 30 04:19:06.271950 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.271867 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c40df95-7863-4ebe-88e2-957f5786b42f" containerName="ta-container" Oct 30 04:19:06.271950 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.271878 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd52c18-cf38-40b7-a37c-f0e1ef1e69d6" containerName="otc-container" Oct 30 04:19:06.276356 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.276334 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" Oct 30 04:19:06.311115 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.311078 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9"] Oct 30 04:19:06.346586 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.346557 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/84697a88-0f07-445d-9284-b26d20b6d7f1-ta-internal\") pod \"data-science-collector-targetallocator-6c949f6bc9-zhpv9\" (UID: \"84697a88-0f07-445d-9284-b26d20b6d7f1\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" Oct 30 04:19:06.346752 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.346594 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl56h\" (UniqueName: \"kubernetes.io/projected/84697a88-0f07-445d-9284-b26d20b6d7f1-kube-api-access-cl56h\") pod \"data-science-collector-targetallocator-6c949f6bc9-zhpv9\" (UID: \"84697a88-0f07-445d-9284-b26d20b6d7f1\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" Oct 30 04:19:06.346807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.346771 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/84697a88-0f07-445d-9284-b26d20b6d7f1-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6c949f6bc9-zhpv9\" (UID: \"84697a88-0f07-445d-9284-b26d20b6d7f1\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" Oct 30 04:19:06.447666 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.447611 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/84697a88-0f07-445d-9284-b26d20b6d7f1-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6c949f6bc9-zhpv9\" (UID: \"84697a88-0f07-445d-9284-b26d20b6d7f1\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" Oct 30 04:19:06.447839 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.447700 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/84697a88-0f07-445d-9284-b26d20b6d7f1-ta-internal\") pod \"data-science-collector-targetallocator-6c949f6bc9-zhpv9\" (UID: \"84697a88-0f07-445d-9284-b26d20b6d7f1\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" Oct 30 04:19:06.447839 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.447731 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl56h\" (UniqueName: \"kubernetes.io/projected/84697a88-0f07-445d-9284-b26d20b6d7f1-kube-api-access-cl56h\") pod \"data-science-collector-targetallocator-6c949f6bc9-zhpv9\" (UID: \"84697a88-0f07-445d-9284-b26d20b6d7f1\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" Oct 30 04:19:06.448365 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.448343 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/84697a88-0f07-445d-9284-b26d20b6d7f1-ta-internal\") pod \"data-science-collector-targetallocator-6c949f6bc9-zhpv9\" (UID: \"84697a88-0f07-445d-9284-b26d20b6d7f1\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" Oct 30 04:19:06.450119 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.450099 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/84697a88-0f07-445d-9284-b26d20b6d7f1-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6c949f6bc9-zhpv9\" (UID: \"84697a88-0f07-445d-9284-b26d20b6d7f1\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" Oct 30 04:19:06.472761 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.472732 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl56h\" (UniqueName: \"kubernetes.io/projected/84697a88-0f07-445d-9284-b26d20b6d7f1-kube-api-access-cl56h\") pod \"data-science-collector-targetallocator-6c949f6bc9-zhpv9\" (UID: \"84697a88-0f07-445d-9284-b26d20b6d7f1\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" Oct 30 04:19:06.587926 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.587858 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" Oct 30 04:19:06.903301 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.903268 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9"] Oct 30 04:19:06.903946 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:19:06.903913 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84697a88_0f07_445d_9284_b26d20b6d7f1.slice/crio-1f2f2aba8720f9799a6f57cc22422c763fb34f2baaf52f23cf80712793ac2cb0 WatchSource:0}: Error finding container 1f2f2aba8720f9799a6f57cc22422c763fb34f2baaf52f23cf80712793ac2cb0: Status 404 returned error can't find the container with id 1f2f2aba8720f9799a6f57cc22422c763fb34f2baaf52f23cf80712793ac2cb0 Oct 30 04:19:06.906134 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.906104 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:19:06.906231 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:06.906158 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:19:07.719859 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:07.719820 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" event={"ID":"84697a88-0f07-445d-9284-b26d20b6d7f1","Type":"ContainerStarted","Data":"3d10f3404249719822a8c9ca7b3541191b29c0696e6cc6996b7b8391d94bc25c"} Oct 30 04:19:07.720316 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:07.719870 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" event={"ID":"84697a88-0f07-445d-9284-b26d20b6d7f1","Type":"ContainerStarted","Data":"1f2f2aba8720f9799a6f57cc22422c763fb34f2baaf52f23cf80712793ac2cb0"} Oct 30 04:19:07.720316 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:07.719891 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" Oct 30 04:19:07.721475 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:07.721450 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" Oct 30 04:19:07.809801 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:07.809748 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" podStartSLOduration=1.809734542 podStartE2EDuration="1.809734542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:07.765318472 +0000 UTC m=+5859.879633449" watchObservedRunningTime="2026-10-30 04:19:07.809734542 +0000 UTC m=+5859.924049516" Oct 30 04:19:07.838406 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:07.838369 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv"] Oct 30 04:19:07.838594 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:07.838569 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" podUID="b795fd60-5d7d-43aa-96ec-4b130206e6f1" containerName="ta-container" containerID="cri-o://8c98779c2f950e8691c1a86140ddd101cdc0b7ef0a8403abe69724552e9ee2d1" gracePeriod=30 Oct 30 04:19:08.081090 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:08.081066 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" Oct 30 04:19:08.163713 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:08.163687 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b795fd60-5d7d-43aa-96ec-4b130206e6f1-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b795fd60-5d7d-43aa-96ec-4b130206e6f1-data-science-collector-ta-server-cert\") pod \"b795fd60-5d7d-43aa-96ec-4b130206e6f1\" (UID: \"b795fd60-5d7d-43aa-96ec-4b130206e6f1\") " Oct 30 04:19:08.163882 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:08.163729 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b795fd60-5d7d-43aa-96ec-4b130206e6f1-ta-internal\" (UniqueName: \"kubernetes.io/configmap/b795fd60-5d7d-43aa-96ec-4b130206e6f1-ta-internal\") pod \"b795fd60-5d7d-43aa-96ec-4b130206e6f1\" (UID: \"b795fd60-5d7d-43aa-96ec-4b130206e6f1\") " Oct 30 04:19:08.163882 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:08.163786 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b795fd60-5d7d-43aa-96ec-4b130206e6f1-kube-api-access-8r229\" (UniqueName: \"kubernetes.io/projected/b795fd60-5d7d-43aa-96ec-4b130206e6f1-kube-api-access-8r229\") pod \"b795fd60-5d7d-43aa-96ec-4b130206e6f1\" (UID: \"b795fd60-5d7d-43aa-96ec-4b130206e6f1\") " Oct 30 04:19:08.164123 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:08.164101 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b795fd60-5d7d-43aa-96ec-4b130206e6f1-ta-internal" pod "b795fd60-5d7d-43aa-96ec-4b130206e6f1" (UID: "b795fd60-5d7d-43aa-96ec-4b130206e6f1"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:08.165825 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:08.165793 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b795fd60-5d7d-43aa-96ec-4b130206e6f1-kube-api-access-8r229" pod "b795fd60-5d7d-43aa-96ec-4b130206e6f1" (UID: "b795fd60-5d7d-43aa-96ec-4b130206e6f1"). InnerVolumeSpecName "kube-api-access-8r229". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:08.165929 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:08.165881 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b795fd60-5d7d-43aa-96ec-4b130206e6f1-data-science-collector-ta-server-cert" pod "b795fd60-5d7d-43aa-96ec-4b130206e6f1" (UID: "b795fd60-5d7d-43aa-96ec-4b130206e6f1"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:08.264783 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:08.264707 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b795fd60-5d7d-43aa-96ec-4b130206e6f1-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:08.264783 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:08.264734 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b795fd60-5d7d-43aa-96ec-4b130206e6f1-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:08.264783 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:08.264743 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r229\" (UniqueName: \"kubernetes.io/projected/b795fd60-5d7d-43aa-96ec-4b130206e6f1-kube-api-access-8r229\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:08.726004 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:08.725919 5882 generic.go:356] "Generic (PLEG): container finished" podID="b795fd60-5d7d-43aa-96ec-4b130206e6f1" containerID="8c98779c2f950e8691c1a86140ddd101cdc0b7ef0a8403abe69724552e9ee2d1" exitCode=0 Oct 30 04:19:08.726004 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:08.725961 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" Oct 30 04:19:08.726492 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:08.726012 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" event={"ID":"b795fd60-5d7d-43aa-96ec-4b130206e6f1","Type":"ContainerDied","Data":"8c98779c2f950e8691c1a86140ddd101cdc0b7ef0a8403abe69724552e9ee2d1"} Oct 30 04:19:08.726492 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:08.726046 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv" event={"ID":"b795fd60-5d7d-43aa-96ec-4b130206e6f1","Type":"ContainerDied","Data":"8db95ef7d80a5a6922b631255db0379f65362ed935aa871f77e6971652d5ecec"} Oct 30 04:19:08.726492 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:08.726065 5882 scope.go:122] "RemoveContainer" containerID="8c98779c2f950e8691c1a86140ddd101cdc0b7ef0a8403abe69724552e9ee2d1" Oct 30 04:19:08.734607 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:08.734589 5882 scope.go:122] "RemoveContainer" containerID="8c98779c2f950e8691c1a86140ddd101cdc0b7ef0a8403abe69724552e9ee2d1" Oct 30 04:19:08.734871 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:19:08.734850 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c98779c2f950e8691c1a86140ddd101cdc0b7ef0a8403abe69724552e9ee2d1\": container with ID starting with 8c98779c2f950e8691c1a86140ddd101cdc0b7ef0a8403abe69724552e9ee2d1 not found: ID does not exist" containerID="8c98779c2f950e8691c1a86140ddd101cdc0b7ef0a8403abe69724552e9ee2d1" Oct 30 04:19:08.734952 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:08.734884 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c98779c2f950e8691c1a86140ddd101cdc0b7ef0a8403abe69724552e9ee2d1"} err="failed to get container status \"8c98779c2f950e8691c1a86140ddd101cdc0b7ef0a8403abe69724552e9ee2d1\": rpc error: code = NotFound desc = could not find container \"8c98779c2f950e8691c1a86140ddd101cdc0b7ef0a8403abe69724552e9ee2d1\": container with ID starting with 8c98779c2f950e8691c1a86140ddd101cdc0b7ef0a8403abe69724552e9ee2d1 not found: ID does not exist" Oct 30 04:19:08.781199 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:08.781169 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv"] Oct 30 04:19:08.795323 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:08.795296 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-74795d8b78-glmjv"] Oct 30 04:19:10.009402 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:10.009370 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b795fd60-5d7d-43aa-96ec-4b130206e6f1" path="/var/lib/kubelet/pods/b795fd60-5d7d-43aa-96ec-4b130206e6f1/volumes" Oct 30 04:19:10.808979 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:10.808948 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:19:10.809265 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:10.809217 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="0cd1858d-e47d-4405-ad1d-33b83b43846d" containerName="prometheus" containerID="cri-o://e2fe712d1b3582d040888a7389a8a7c1db2ea319bb6bdf53226531ff9b6620bc" gracePeriod=600 Oct 30 04:19:10.809348 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:10.809250 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="0cd1858d-e47d-4405-ad1d-33b83b43846d" containerName="config-reloader" containerID="cri-o://0cac6c3f5f51df269857622a1bbc8d13606d4e9a66a685edf78212f17e558188" gracePeriod=600 Oct 30 04:19:10.809435 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:10.809247 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="0cd1858d-e47d-4405-ad1d-33b83b43846d" containerName="thanos-sidecar" containerID="cri-o://12ae11190d08e587d760a511d365d63a23dcdc4ef34628d784a3816eaeafbdda" gracePeriod=600 Oct 30 04:19:10.813884 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:10.813860 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:19:10.814080 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:10.814060 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="78072d40-0ad6-4a50-a475-652f1cdd980c" containerName="alertmanager" containerID="cri-o://f9b902aa5911d56fda79ee288d89f26b017953d51dc7ad7866a9851521a6e0b7" gracePeriod=120 Oct 30 04:19:10.814184 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:10.814146 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="78072d40-0ad6-4a50-a475-652f1cdd980c" containerName="config-reloader" containerID="cri-o://40e1f66f8ea505c961be3e318f880a878065f6ecc44272606856155b49e76036" gracePeriod=120 Oct 30 04:19:11.655529 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.655505 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:11.674819 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.674789 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="0cd1858d-e47d-4405-ad1d-33b83b43846d" containerName="prometheus" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:19:11.744759 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.744732 5882 generic.go:356] "Generic (PLEG): container finished" podID="0cd1858d-e47d-4405-ad1d-33b83b43846d" containerID="12ae11190d08e587d760a511d365d63a23dcdc4ef34628d784a3816eaeafbdda" exitCode=0 Oct 30 04:19:11.744759 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.744757 5882 generic.go:356] "Generic (PLEG): container finished" podID="0cd1858d-e47d-4405-ad1d-33b83b43846d" containerID="0cac6c3f5f51df269857622a1bbc8d13606d4e9a66a685edf78212f17e558188" exitCode=0 Oct 30 04:19:11.744939 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.744801 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0cd1858d-e47d-4405-ad1d-33b83b43846d","Type":"ContainerDied","Data":"12ae11190d08e587d760a511d365d63a23dcdc4ef34628d784a3816eaeafbdda"} Oct 30 04:19:11.744939 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.744865 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0cd1858d-e47d-4405-ad1d-33b83b43846d","Type":"ContainerDied","Data":"0cac6c3f5f51df269857622a1bbc8d13606d4e9a66a685edf78212f17e558188"} Oct 30 04:19:11.746862 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.746839 5882 generic.go:356] "Generic (PLEG): container finished" podID="78072d40-0ad6-4a50-a475-652f1cdd980c" containerID="40e1f66f8ea505c961be3e318f880a878065f6ecc44272606856155b49e76036" exitCode=0 Oct 30 04:19:11.746862 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.746860 5882 generic.go:356] "Generic (PLEG): container finished" podID="78072d40-0ad6-4a50-a475-652f1cdd980c" containerID="f9b902aa5911d56fda79ee288d89f26b017953d51dc7ad7866a9851521a6e0b7" exitCode=0 Oct 30 04:19:11.747011 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.746917 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:11.747011 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.746919 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"78072d40-0ad6-4a50-a475-652f1cdd980c","Type":"ContainerDied","Data":"40e1f66f8ea505c961be3e318f880a878065f6ecc44272606856155b49e76036"} Oct 30 04:19:11.747011 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.746975 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"78072d40-0ad6-4a50-a475-652f1cdd980c","Type":"ContainerDied","Data":"f9b902aa5911d56fda79ee288d89f26b017953d51dc7ad7866a9851521a6e0b7"} Oct 30 04:19:11.747011 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.746997 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"78072d40-0ad6-4a50-a475-652f1cdd980c","Type":"ContainerDied","Data":"70b99802ab48ba5198612a9475c9c29b9fcadc03146d8d8872313a51285f4dac"} Oct 30 04:19:11.747137 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.747021 5882 scope.go:122] "RemoveContainer" containerID="40e1f66f8ea505c961be3e318f880a878065f6ecc44272606856155b49e76036" Oct 30 04:19:11.756880 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.756858 5882 scope.go:122] "RemoveContainer" containerID="f9b902aa5911d56fda79ee288d89f26b017953d51dc7ad7866a9851521a6e0b7" Oct 30 04:19:11.764279 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.764248 5882 scope.go:122] "RemoveContainer" containerID="a8ffe0b26f33fc7efdbc82e449b77900c376b8f576ee55e5f60c3d03f8221666" Oct 30 04:19:11.771699 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.771684 5882 scope.go:122] "RemoveContainer" containerID="40e1f66f8ea505c961be3e318f880a878065f6ecc44272606856155b49e76036" Oct 30 04:19:11.771971 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:19:11.771949 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e1f66f8ea505c961be3e318f880a878065f6ecc44272606856155b49e76036\": container with ID starting with 40e1f66f8ea505c961be3e318f880a878065f6ecc44272606856155b49e76036 not found: ID does not exist" containerID="40e1f66f8ea505c961be3e318f880a878065f6ecc44272606856155b49e76036" Oct 30 04:19:11.772027 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.771979 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e1f66f8ea505c961be3e318f880a878065f6ecc44272606856155b49e76036"} err="failed to get container status \"40e1f66f8ea505c961be3e318f880a878065f6ecc44272606856155b49e76036\": rpc error: code = NotFound desc = could not find container \"40e1f66f8ea505c961be3e318f880a878065f6ecc44272606856155b49e76036\": container with ID starting with 40e1f66f8ea505c961be3e318f880a878065f6ecc44272606856155b49e76036 not found: ID does not exist" Oct 30 04:19:11.772027 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.771997 5882 scope.go:122] "RemoveContainer" containerID="f9b902aa5911d56fda79ee288d89f26b017953d51dc7ad7866a9851521a6e0b7" Oct 30 04:19:11.772278 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:19:11.772254 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b902aa5911d56fda79ee288d89f26b017953d51dc7ad7866a9851521a6e0b7\": container with ID starting with f9b902aa5911d56fda79ee288d89f26b017953d51dc7ad7866a9851521a6e0b7 not found: ID does not exist" containerID="f9b902aa5911d56fda79ee288d89f26b017953d51dc7ad7866a9851521a6e0b7" Oct 30 04:19:11.772334 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.772288 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b902aa5911d56fda79ee288d89f26b017953d51dc7ad7866a9851521a6e0b7"} err="failed to get container status \"f9b902aa5911d56fda79ee288d89f26b017953d51dc7ad7866a9851521a6e0b7\": rpc error: code = NotFound desc = could not find container \"f9b902aa5911d56fda79ee288d89f26b017953d51dc7ad7866a9851521a6e0b7\": container with ID starting with f9b902aa5911d56fda79ee288d89f26b017953d51dc7ad7866a9851521a6e0b7 not found: ID does not exist" Oct 30 04:19:11.772334 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.772311 5882 scope.go:122] "RemoveContainer" containerID="a8ffe0b26f33fc7efdbc82e449b77900c376b8f576ee55e5f60c3d03f8221666" Oct 30 04:19:11.772555 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:19:11.772533 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ffe0b26f33fc7efdbc82e449b77900c376b8f576ee55e5f60c3d03f8221666\": container with ID starting with a8ffe0b26f33fc7efdbc82e449b77900c376b8f576ee55e5f60c3d03f8221666 not found: ID does not exist" containerID="a8ffe0b26f33fc7efdbc82e449b77900c376b8f576ee55e5f60c3d03f8221666" Oct 30 04:19:11.772606 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.772560 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ffe0b26f33fc7efdbc82e449b77900c376b8f576ee55e5f60c3d03f8221666"} err="failed to get container status \"a8ffe0b26f33fc7efdbc82e449b77900c376b8f576ee55e5f60c3d03f8221666\": rpc error: code = NotFound desc = could not find container \"a8ffe0b26f33fc7efdbc82e449b77900c376b8f576ee55e5f60c3d03f8221666\": container with ID starting with a8ffe0b26f33fc7efdbc82e449b77900c376b8f576ee55e5f60c3d03f8221666 not found: ID does not exist" Oct 30 04:19:11.772606 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.772573 5882 scope.go:122] "RemoveContainer" containerID="40e1f66f8ea505c961be3e318f880a878065f6ecc44272606856155b49e76036" Oct 30 04:19:11.772856 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.772838 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e1f66f8ea505c961be3e318f880a878065f6ecc44272606856155b49e76036"} err="failed to get container status \"40e1f66f8ea505c961be3e318f880a878065f6ecc44272606856155b49e76036\": rpc error: code = NotFound desc = could not find container \"40e1f66f8ea505c961be3e318f880a878065f6ecc44272606856155b49e76036\": container with ID starting with 40e1f66f8ea505c961be3e318f880a878065f6ecc44272606856155b49e76036 not found: ID does not exist" Oct 30 04:19:11.772903 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.772865 5882 scope.go:122] "RemoveContainer" containerID="f9b902aa5911d56fda79ee288d89f26b017953d51dc7ad7866a9851521a6e0b7" Oct 30 04:19:11.773107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.773088 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b902aa5911d56fda79ee288d89f26b017953d51dc7ad7866a9851521a6e0b7"} err="failed to get container status \"f9b902aa5911d56fda79ee288d89f26b017953d51dc7ad7866a9851521a6e0b7\": rpc error: code = NotFound desc = could not find container \"f9b902aa5911d56fda79ee288d89f26b017953d51dc7ad7866a9851521a6e0b7\": container with ID starting with f9b902aa5911d56fda79ee288d89f26b017953d51dc7ad7866a9851521a6e0b7 not found: ID does not exist" Oct 30 04:19:11.773148 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.773108 5882 scope.go:122] "RemoveContainer" containerID="a8ffe0b26f33fc7efdbc82e449b77900c376b8f576ee55e5f60c3d03f8221666" Oct 30 04:19:11.773332 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.773313 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ffe0b26f33fc7efdbc82e449b77900c376b8f576ee55e5f60c3d03f8221666"} err="failed to get container status \"a8ffe0b26f33fc7efdbc82e449b77900c376b8f576ee55e5f60c3d03f8221666\": rpc error: code = NotFound desc = could not find container \"a8ffe0b26f33fc7efdbc82e449b77900c376b8f576ee55e5f60c3d03f8221666\": container with ID starting with a8ffe0b26f33fc7efdbc82e449b77900c376b8f576ee55e5f60c3d03f8221666 not found: ID does not exist" Oct 30 04:19:11.797663 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.797641 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78072d40-0ad6-4a50-a475-652f1cdd980c-kube-api-access-lpzj5\" (UniqueName: \"kubernetes.io/projected/78072d40-0ad6-4a50-a475-652f1cdd980c-kube-api-access-lpzj5\") pod \"78072d40-0ad6-4a50-a475-652f1cdd980c\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " Oct 30 04:19:11.797728 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.797682 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-cluster-tls-config\") pod \"78072d40-0ad6-4a50-a475-652f1cdd980c\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " Oct 30 04:19:11.797775 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.797736 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78072d40-0ad6-4a50-a475-652f1cdd980c-tls-assets\" (UniqueName: \"kubernetes.io/projected/78072d40-0ad6-4a50-a475-652f1cdd980c-tls-assets\") pod \"78072d40-0ad6-4a50-a475-652f1cdd980c\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " Oct 30 04:19:11.797809 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.797776 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78072d40-0ad6-4a50-a475-652f1cdd980c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/78072d40-0ad6-4a50-a475-652f1cdd980c-config-out\") pod \"78072d40-0ad6-4a50-a475-652f1cdd980c\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " Oct 30 04:19:11.797809 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.797797 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78072d40-0ad6-4a50-a475-652f1cdd980c-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/78072d40-0ad6-4a50-a475-652f1cdd980c-alertmanager-data-science-monitoringstack-db\") pod \"78072d40-0ad6-4a50-a475-652f1cdd980c\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " Oct 30 04:19:11.797906 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.797825 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-web-config\" (UniqueName: \"kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-web-config\") pod \"78072d40-0ad6-4a50-a475-652f1cdd980c\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " Oct 30 04:19:11.797968 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.797940 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-config-volume\" (UniqueName: \"kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-config-volume\") pod \"78072d40-0ad6-4a50-a475-652f1cdd980c\" (UID: \"78072d40-0ad6-4a50-a475-652f1cdd980c\") " Oct 30 04:19:11.798188 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.798140 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78072d40-0ad6-4a50-a475-652f1cdd980c-alertmanager-data-science-monitoringstack-db" pod "78072d40-0ad6-4a50-a475-652f1cdd980c" (UID: "78072d40-0ad6-4a50-a475-652f1cdd980c"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:19:11.798550 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.798519 5882 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/78072d40-0ad6-4a50-a475-652f1cdd980c-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:11.799897 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.799872 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-config-volume" pod "78072d40-0ad6-4a50-a475-652f1cdd980c" (UID: "78072d40-0ad6-4a50-a475-652f1cdd980c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:11.800009 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.799923 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78072d40-0ad6-4a50-a475-652f1cdd980c-tls-assets" pod "78072d40-0ad6-4a50-a475-652f1cdd980c" (UID: "78072d40-0ad6-4a50-a475-652f1cdd980c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:11.800199 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.800171 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78072d40-0ad6-4a50-a475-652f1cdd980c-kube-api-access-lpzj5" pod "78072d40-0ad6-4a50-a475-652f1cdd980c" (UID: "78072d40-0ad6-4a50-a475-652f1cdd980c"). InnerVolumeSpecName "kube-api-access-lpzj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:11.800283 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.800238 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78072d40-0ad6-4a50-a475-652f1cdd980c-config-out" pod "78072d40-0ad6-4a50-a475-652f1cdd980c" (UID: "78072d40-0ad6-4a50-a475-652f1cdd980c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:19:11.804201 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.804181 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-cluster-tls-config" pod "78072d40-0ad6-4a50-a475-652f1cdd980c" (UID: "78072d40-0ad6-4a50-a475-652f1cdd980c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:11.811081 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.811059 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-web-config" pod "78072d40-0ad6-4a50-a475-652f1cdd980c" (UID: "78072d40-0ad6-4a50-a475-652f1cdd980c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:11.899358 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.899327 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpzj5\" (UniqueName: \"kubernetes.io/projected/78072d40-0ad6-4a50-a475-652f1cdd980c-kube-api-access-lpzj5\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:11.899358 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.899357 5882 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-cluster-tls-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:11.899584 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.899373 5882 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/78072d40-0ad6-4a50-a475-652f1cdd980c-tls-assets\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:11.899584 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.899385 5882 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/78072d40-0ad6-4a50-a475-652f1cdd980c-config-out\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:11.899584 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.899396 5882 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-web-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:11.899584 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.899406 5882 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/78072d40-0ad6-4a50-a475-652f1cdd980c-config-volume\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:11.968117 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.968078 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9"] Oct 30 04:19:11.968367 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:11.968339 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" podUID="84697a88-0f07-445d-9284-b26d20b6d7f1" containerName="ta-container" containerID="cri-o://3d10f3404249719822a8c9ca7b3541191b29c0696e6cc6996b7b8391d94bc25c" gracePeriod=30 Oct 30 04:19:12.095000 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.094970 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:19:12.095255 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.095228 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="1cb2478f-259b-4eae-9fee-4d527b0c0143" containerName="perses" containerID="cri-o://bfcf280dbf506612a9062e5fc017fe9724a4bb036b3b498daf2cdcb38971a63a" gracePeriod=30 Oct 30 04:19:12.098649 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.098628 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:19:12.105905 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.105879 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:19:12.226544 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.226518 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" Oct 30 04:19:12.303116 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.303037 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84697a88-0f07-445d-9284-b26d20b6d7f1-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/84697a88-0f07-445d-9284-b26d20b6d7f1-data-science-collector-ta-server-cert\") pod \"84697a88-0f07-445d-9284-b26d20b6d7f1\" (UID: \"84697a88-0f07-445d-9284-b26d20b6d7f1\") " Oct 30 04:19:12.303116 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.303077 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84697a88-0f07-445d-9284-b26d20b6d7f1-ta-internal\" (UniqueName: \"kubernetes.io/configmap/84697a88-0f07-445d-9284-b26d20b6d7f1-ta-internal\") pod \"84697a88-0f07-445d-9284-b26d20b6d7f1\" (UID: \"84697a88-0f07-445d-9284-b26d20b6d7f1\") " Oct 30 04:19:12.303299 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.303151 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84697a88-0f07-445d-9284-b26d20b6d7f1-kube-api-access-cl56h\" (UniqueName: \"kubernetes.io/projected/84697a88-0f07-445d-9284-b26d20b6d7f1-kube-api-access-cl56h\") pod \"84697a88-0f07-445d-9284-b26d20b6d7f1\" (UID: \"84697a88-0f07-445d-9284-b26d20b6d7f1\") " Oct 30 04:19:12.303562 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.303528 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84697a88-0f07-445d-9284-b26d20b6d7f1-ta-internal" pod "84697a88-0f07-445d-9284-b26d20b6d7f1" (UID: "84697a88-0f07-445d-9284-b26d20b6d7f1"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:12.306301 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.306271 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84697a88-0f07-445d-9284-b26d20b6d7f1-data-science-collector-ta-server-cert" pod "84697a88-0f07-445d-9284-b26d20b6d7f1" (UID: "84697a88-0f07-445d-9284-b26d20b6d7f1"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:12.306399 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.306305 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84697a88-0f07-445d-9284-b26d20b6d7f1-kube-api-access-cl56h" pod "84697a88-0f07-445d-9284-b26d20b6d7f1" (UID: "84697a88-0f07-445d-9284-b26d20b6d7f1"). InnerVolumeSpecName "kube-api-access-cl56h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:12.324404 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.324385 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:12.404109 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.404076 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"1cb2478f-259b-4eae-9fee-4d527b0c0143\" (UID: \"1cb2478f-259b-4eae-9fee-4d527b0c0143\") " Oct 30 04:19:12.404279 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.404185 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb2478f-259b-4eae-9fee-4d527b0c0143-plugins\" (UniqueName: \"kubernetes.io/empty-dir/1cb2478f-259b-4eae-9fee-4d527b0c0143-plugins\") pod \"1cb2478f-259b-4eae-9fee-4d527b0c0143\" (UID: \"1cb2478f-259b-4eae-9fee-4d527b0c0143\") " Oct 30 04:19:12.404279 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.404219 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cb2478f-259b-4eae-9fee-4d527b0c0143-config\" (UniqueName: \"kubernetes.io/configmap/1cb2478f-259b-4eae-9fee-4d527b0c0143-config\") pod \"1cb2478f-259b-4eae-9fee-4d527b0c0143\" (UID: \"1cb2478f-259b-4eae-9fee-4d527b0c0143\") " Oct 30 04:19:12.404405 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.404280 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb2478f-259b-4eae-9fee-4d527b0c0143-kube-api-access-zvq5s\" (UniqueName: \"kubernetes.io/projected/1cb2478f-259b-4eae-9fee-4d527b0c0143-kube-api-access-zvq5s\") pod \"1cb2478f-259b-4eae-9fee-4d527b0c0143\" (UID: \"1cb2478f-259b-4eae-9fee-4d527b0c0143\") " Oct 30 04:19:12.404603 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.404574 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/84697a88-0f07-445d-9284-b26d20b6d7f1-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:12.404603 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.404602 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/84697a88-0f07-445d-9284-b26d20b6d7f1-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:12.404798 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.404607 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb2478f-259b-4eae-9fee-4d527b0c0143-config" pod "1cb2478f-259b-4eae-9fee-4d527b0c0143" (UID: "1cb2478f-259b-4eae-9fee-4d527b0c0143"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:12.404798 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.404647 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl56h\" (UniqueName: \"kubernetes.io/projected/84697a88-0f07-445d-9284-b26d20b6d7f1-kube-api-access-cl56h\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:12.406430 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.406408 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb2478f-259b-4eae-9fee-4d527b0c0143-kube-api-access-zvq5s" pod "1cb2478f-259b-4eae-9fee-4d527b0c0143" (UID: "1cb2478f-259b-4eae-9fee-4d527b0c0143"). InnerVolumeSpecName "kube-api-access-zvq5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:12.406812 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.406794 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e" pod "1cb2478f-259b-4eae-9fee-4d527b0c0143" (UID: "1cb2478f-259b-4eae-9fee-4d527b0c0143"). InnerVolumeSpecName "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:19:12.505050 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.505020 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1cb2478f-259b-4eae-9fee-4d527b0c0143-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:12.505050 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.505044 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvq5s\" (UniqueName: \"kubernetes.io/projected/1cb2478f-259b-4eae-9fee-4d527b0c0143-kube-api-access-zvq5s\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:12.505274 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.505073 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:19:12.518003 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.517981 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:19:12.555751 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.555687 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb2478f-259b-4eae-9fee-4d527b0c0143-plugins" pod "1cb2478f-259b-4eae-9fee-4d527b0c0143" (UID: "1cb2478f-259b-4eae-9fee-4d527b0c0143"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:19:12.606504 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.606474 5882 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/1cb2478f-259b-4eae-9fee-4d527b0c0143-plugins\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:12.606504 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.606501 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" Oct 30 04:19:12.753312 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.753273 5882 generic.go:356] "Generic (PLEG): container finished" podID="84697a88-0f07-445d-9284-b26d20b6d7f1" containerID="3d10f3404249719822a8c9ca7b3541191b29c0696e6cc6996b7b8391d94bc25c" exitCode=0 Oct 30 04:19:12.753312 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.753314 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" Oct 30 04:19:12.753826 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.753352 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" event={"ID":"84697a88-0f07-445d-9284-b26d20b6d7f1","Type":"ContainerDied","Data":"3d10f3404249719822a8c9ca7b3541191b29c0696e6cc6996b7b8391d94bc25c"} Oct 30 04:19:12.753826 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.753396 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9" event={"ID":"84697a88-0f07-445d-9284-b26d20b6d7f1","Type":"ContainerDied","Data":"1f2f2aba8720f9799a6f57cc22422c763fb34f2baaf52f23cf80712793ac2cb0"} Oct 30 04:19:12.753826 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.753413 5882 scope.go:122] "RemoveContainer" containerID="3d10f3404249719822a8c9ca7b3541191b29c0696e6cc6996b7b8391d94bc25c" Oct 30 04:19:12.754917 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.754885 5882 generic.go:356] "Generic (PLEG): container finished" podID="1cb2478f-259b-4eae-9fee-4d527b0c0143" containerID="bfcf280dbf506612a9062e5fc017fe9724a4bb036b3b498daf2cdcb38971a63a" exitCode=0 Oct 30 04:19:12.755029 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.754930 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:12.755029 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.754930 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"1cb2478f-259b-4eae-9fee-4d527b0c0143","Type":"ContainerDied","Data":"bfcf280dbf506612a9062e5fc017fe9724a4bb036b3b498daf2cdcb38971a63a"} Oct 30 04:19:12.755124 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.755057 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"1cb2478f-259b-4eae-9fee-4d527b0c0143","Type":"ContainerDied","Data":"6e2a5459a2687b0506bed23575dc41c84ab5c4850e93cc3f4787b1677d37c632"} Oct 30 04:19:12.762370 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.762352 5882 scope.go:122] "RemoveContainer" containerID="3d10f3404249719822a8c9ca7b3541191b29c0696e6cc6996b7b8391d94bc25c" Oct 30 04:19:12.762686 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:19:12.762665 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d10f3404249719822a8c9ca7b3541191b29c0696e6cc6996b7b8391d94bc25c\": container with ID starting with 3d10f3404249719822a8c9ca7b3541191b29c0696e6cc6996b7b8391d94bc25c not found: ID does not exist" containerID="3d10f3404249719822a8c9ca7b3541191b29c0696e6cc6996b7b8391d94bc25c" Oct 30 04:19:12.762760 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.762692 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d10f3404249719822a8c9ca7b3541191b29c0696e6cc6996b7b8391d94bc25c"} err="failed to get container status \"3d10f3404249719822a8c9ca7b3541191b29c0696e6cc6996b7b8391d94bc25c\": rpc error: code = NotFound desc = could not find container \"3d10f3404249719822a8c9ca7b3541191b29c0696e6cc6996b7b8391d94bc25c\": container with ID starting with 3d10f3404249719822a8c9ca7b3541191b29c0696e6cc6996b7b8391d94bc25c not found: ID does not exist" Oct 30 04:19:12.762760 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.762708 5882 scope.go:122] "RemoveContainer" containerID="bfcf280dbf506612a9062e5fc017fe9724a4bb036b3b498daf2cdcb38971a63a" Oct 30 04:19:12.770885 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.770867 5882 scope.go:122] "RemoveContainer" containerID="bfcf280dbf506612a9062e5fc017fe9724a4bb036b3b498daf2cdcb38971a63a" Oct 30 04:19:12.771133 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:19:12.771113 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfcf280dbf506612a9062e5fc017fe9724a4bb036b3b498daf2cdcb38971a63a\": container with ID starting with bfcf280dbf506612a9062e5fc017fe9724a4bb036b3b498daf2cdcb38971a63a not found: ID does not exist" containerID="bfcf280dbf506612a9062e5fc017fe9724a4bb036b3b498daf2cdcb38971a63a" Oct 30 04:19:12.771176 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.771140 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfcf280dbf506612a9062e5fc017fe9724a4bb036b3b498daf2cdcb38971a63a"} err="failed to get container status \"bfcf280dbf506612a9062e5fc017fe9724a4bb036b3b498daf2cdcb38971a63a\": rpc error: code = NotFound desc = could not find container \"bfcf280dbf506612a9062e5fc017fe9724a4bb036b3b498daf2cdcb38971a63a\": container with ID starting with bfcf280dbf506612a9062e5fc017fe9724a4bb036b3b498daf2cdcb38971a63a not found: ID does not exist" Oct 30 04:19:12.809796 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.809741 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9"] Oct 30 04:19:12.813492 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.813472 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c949f6bc9-zhpv9"] Oct 30 04:19:12.850202 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.850176 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:19:12.854605 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:12.854579 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:19:14.009609 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:14.009572 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cb2478f-259b-4eae-9fee-4d527b0c0143" path="/var/lib/kubelet/pods/1cb2478f-259b-4eae-9fee-4d527b0c0143/volumes" Oct 30 04:19:14.010004 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:14.009980 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78072d40-0ad6-4a50-a475-652f1cdd980c" path="/var/lib/kubelet/pods/78072d40-0ad6-4a50-a475-652f1cdd980c/volumes" Oct 30 04:19:14.010411 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:14.010398 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84697a88-0f07-445d-9284-b26d20b6d7f1" path="/var/lib/kubelet/pods/84697a88-0f07-445d-9284-b26d20b6d7f1/volumes" Oct 30 04:19:16.674398 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:16.674351 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="0cd1858d-e47d-4405-ad1d-33b83b43846d" containerName="prometheus" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:19:20.156532 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.156507 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:20.277565 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.277537 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd1858d-e47d-4405-ad1d-33b83b43846d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0cd1858d-e47d-4405-ad1d-33b83b43846d-config-out\") pod \"0cd1858d-e47d-4405-ad1d-33b83b43846d\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " Oct 30 04:19:20.277760 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.277573 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-config\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-config\") pod \"0cd1858d-e47d-4405-ad1d-33b83b43846d\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " Oct 30 04:19:20.277760 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.277606 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"0cd1858d-e47d-4405-ad1d-33b83b43846d\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " Oct 30 04:19:20.277760 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.277656 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"0cd1858d-e47d-4405-ad1d-33b83b43846d\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " Oct 30 04:19:20.277760 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.277726 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"0cd1858d-e47d-4405-ad1d-33b83b43846d\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " Oct 30 04:19:20.277760 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.277750 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd1858d-e47d-4405-ad1d-33b83b43846d-kube-api-access-jrk55\" (UniqueName: \"kubernetes.io/projected/0cd1858d-e47d-4405-ad1d-33b83b43846d-kube-api-access-jrk55\") pod \"0cd1858d-e47d-4405-ad1d-33b83b43846d\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " Oct 30 04:19:20.278004 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.277777 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"0cd1858d-e47d-4405-ad1d-33b83b43846d\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " Oct 30 04:19:20.278004 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.277819 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-thanos-prometheus-http-client-file\") pod \"0cd1858d-e47d-4405-ad1d-33b83b43846d\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " Oct 30 04:19:20.278004 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.277856 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config\") pod \"0cd1858d-e47d-4405-ad1d-33b83b43846d\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " Oct 30 04:19:20.278004 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.277945 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"0cd1858d-e47d-4405-ad1d-33b83b43846d\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " Oct 30 04:19:20.278004 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.277989 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-secret-prometheus-web-tls-ca\") pod \"0cd1858d-e47d-4405-ad1d-33b83b43846d\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " Oct 30 04:19:20.278285 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.278040 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd1858d-e47d-4405-ad1d-33b83b43846d-tls-assets\" (UniqueName: \"kubernetes.io/projected/0cd1858d-e47d-4405-ad1d-33b83b43846d-tls-assets\") pod \"0cd1858d-e47d-4405-ad1d-33b83b43846d\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " Oct 30 04:19:20.278285 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.278097 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"0cd1858d-e47d-4405-ad1d-33b83b43846d\" (UID: \"0cd1858d-e47d-4405-ad1d-33b83b43846d\") " Oct 30 04:19:20.278391 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.278287 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-0" pod "0cd1858d-e47d-4405-ad1d-33b83b43846d" (UID: "0cd1858d-e47d-4405-ad1d-33b83b43846d"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:20.278455 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.278441 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:20.278859 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.278576 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-1" pod "0cd1858d-e47d-4405-ad1d-33b83b43846d" (UID: "0cd1858d-e47d-4405-ad1d-33b83b43846d"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:20.278859 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.278654 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-2" pod "0cd1858d-e47d-4405-ad1d-33b83b43846d" (UID: "0cd1858d-e47d-4405-ad1d-33b83b43846d"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:20.280477 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.280444 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "0cd1858d-e47d-4405-ad1d-33b83b43846d" (UID: "0cd1858d-e47d-4405-ad1d-33b83b43846d"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:20.280595 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.280466 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "0cd1858d-e47d-4405-ad1d-33b83b43846d" (UID: "0cd1858d-e47d-4405-ad1d-33b83b43846d"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:20.281244 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.281218 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-thanos-prometheus-http-client-file" pod "0cd1858d-e47d-4405-ad1d-33b83b43846d" (UID: "0cd1858d-e47d-4405-ad1d-33b83b43846d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:20.281385 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.281313 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd1858d-e47d-4405-ad1d-33b83b43846d-tls-assets" pod "0cd1858d-e47d-4405-ad1d-33b83b43846d" (UID: "0cd1858d-e47d-4405-ad1d-33b83b43846d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:20.281547 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.281526 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-secret-prometheus-web-tls-ca" pod "0cd1858d-e47d-4405-ad1d-33b83b43846d" (UID: "0cd1858d-e47d-4405-ad1d-33b83b43846d"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:20.281968 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.281943 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-config" pod "0cd1858d-e47d-4405-ad1d-33b83b43846d" (UID: "0cd1858d-e47d-4405-ad1d-33b83b43846d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:20.282069 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.281975 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd1858d-e47d-4405-ad1d-33b83b43846d-kube-api-access-jrk55" pod "0cd1858d-e47d-4405-ad1d-33b83b43846d" (UID: "0cd1858d-e47d-4405-ad1d-33b83b43846d"). InnerVolumeSpecName "kube-api-access-jrk55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:20.282286 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.282256 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd1858d-e47d-4405-ad1d-33b83b43846d-config-out" pod "0cd1858d-e47d-4405-ad1d-33b83b43846d" (UID: "0cd1858d-e47d-4405-ad1d-33b83b43846d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:19:20.283525 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.283501 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4" pod "0cd1858d-e47d-4405-ad1d-33b83b43846d" (UID: "0cd1858d-e47d-4405-ad1d-33b83b43846d"). InnerVolumeSpecName "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:19:20.293574 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.293550 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config" pod "0cd1858d-e47d-4405-ad1d-33b83b43846d" (UID: "0cd1858d-e47d-4405-ad1d-33b83b43846d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:20.379662 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.379604 5882 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:20.379834 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.379682 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:19:20.379834 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.379701 5882 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-secret-prometheus-web-tls-ca\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:20.379834 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.379713 5882 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0cd1858d-e47d-4405-ad1d-33b83b43846d-tls-assets\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:20.379834 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.379723 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:20.379834 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.379735 5882 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0cd1858d-e47d-4405-ad1d-33b83b43846d-config-out\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:20.379834 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.379744 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:20.379834 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.379758 5882 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:20.379834 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.379772 5882 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:20.379834 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.379786 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrk55\" (UniqueName: \"kubernetes.io/projected/0cd1858d-e47d-4405-ad1d-33b83b43846d-kube-api-access-jrk55\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:20.379834 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.379799 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0cd1858d-e47d-4405-ad1d-33b83b43846d-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:20.379834 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.379817 5882 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0cd1858d-e47d-4405-ad1d-33b83b43846d-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:20.392531 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.392512 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:19:20.480773 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.480743 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-927d18ef2eb99137c68fbe5e0bb8ea629bb5219ed56cdede72da7c0f045be95e\"" Oct 30 04:19:20.794920 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.794872 5882 generic.go:356] "Generic (PLEG): container finished" podID="0cd1858d-e47d-4405-ad1d-33b83b43846d" containerID="e2fe712d1b3582d040888a7389a8a7c1db2ea319bb6bdf53226531ff9b6620bc" exitCode=0 Oct 30 04:19:20.794920 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.794924 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0cd1858d-e47d-4405-ad1d-33b83b43846d","Type":"ContainerDied","Data":"e2fe712d1b3582d040888a7389a8a7c1db2ea319bb6bdf53226531ff9b6620bc"} Oct 30 04:19:20.795144 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.794958 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:20.795144 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.794994 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"0cd1858d-e47d-4405-ad1d-33b83b43846d","Type":"ContainerDied","Data":"9b40d3e7d97c218125156632477bf8eefbf45002e115fc0affd58b8888baf078"} Oct 30 04:19:20.795144 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.795037 5882 scope.go:122] "RemoveContainer" containerID="12ae11190d08e587d760a511d365d63a23dcdc4ef34628d784a3816eaeafbdda" Oct 30 04:19:20.803240 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.803219 5882 scope.go:122] "RemoveContainer" containerID="0cac6c3f5f51df269857622a1bbc8d13606d4e9a66a685edf78212f17e558188" Oct 30 04:19:20.810969 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.810953 5882 scope.go:122] "RemoveContainer" containerID="e2fe712d1b3582d040888a7389a8a7c1db2ea319bb6bdf53226531ff9b6620bc" Oct 30 04:19:20.818452 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.818433 5882 scope.go:122] "RemoveContainer" containerID="c68f4fd6143ccce635e4e812c6c4d556df3c2e10a791599b1041ca3dcac1035c" Oct 30 04:19:20.825802 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.825770 5882 scope.go:122] "RemoveContainer" containerID="12ae11190d08e587d760a511d365d63a23dcdc4ef34628d784a3816eaeafbdda" Oct 30 04:19:20.826064 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:19:20.826038 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12ae11190d08e587d760a511d365d63a23dcdc4ef34628d784a3816eaeafbdda\": container with ID starting with 12ae11190d08e587d760a511d365d63a23dcdc4ef34628d784a3816eaeafbdda not found: ID does not exist" containerID="12ae11190d08e587d760a511d365d63a23dcdc4ef34628d784a3816eaeafbdda" Oct 30 04:19:20.826110 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.826073 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12ae11190d08e587d760a511d365d63a23dcdc4ef34628d784a3816eaeafbdda"} err="failed to get container status \"12ae11190d08e587d760a511d365d63a23dcdc4ef34628d784a3816eaeafbdda\": rpc error: code = NotFound desc = could not find container \"12ae11190d08e587d760a511d365d63a23dcdc4ef34628d784a3816eaeafbdda\": container with ID starting with 12ae11190d08e587d760a511d365d63a23dcdc4ef34628d784a3816eaeafbdda not found: ID does not exist" Oct 30 04:19:20.826110 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.826092 5882 scope.go:122] "RemoveContainer" containerID="0cac6c3f5f51df269857622a1bbc8d13606d4e9a66a685edf78212f17e558188" Oct 30 04:19:20.826312 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:19:20.826296 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cac6c3f5f51df269857622a1bbc8d13606d4e9a66a685edf78212f17e558188\": container with ID starting with 0cac6c3f5f51df269857622a1bbc8d13606d4e9a66a685edf78212f17e558188 not found: ID does not exist" containerID="0cac6c3f5f51df269857622a1bbc8d13606d4e9a66a685edf78212f17e558188" Oct 30 04:19:20.826353 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.826316 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cac6c3f5f51df269857622a1bbc8d13606d4e9a66a685edf78212f17e558188"} err="failed to get container status \"0cac6c3f5f51df269857622a1bbc8d13606d4e9a66a685edf78212f17e558188\": rpc error: code = NotFound desc = could not find container \"0cac6c3f5f51df269857622a1bbc8d13606d4e9a66a685edf78212f17e558188\": container with ID starting with 0cac6c3f5f51df269857622a1bbc8d13606d4e9a66a685edf78212f17e558188 not found: ID does not exist" Oct 30 04:19:20.826353 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.826334 5882 scope.go:122] "RemoveContainer" containerID="e2fe712d1b3582d040888a7389a8a7c1db2ea319bb6bdf53226531ff9b6620bc" Oct 30 04:19:20.826533 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:19:20.826515 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2fe712d1b3582d040888a7389a8a7c1db2ea319bb6bdf53226531ff9b6620bc\": container with ID starting with e2fe712d1b3582d040888a7389a8a7c1db2ea319bb6bdf53226531ff9b6620bc not found: ID does not exist" containerID="e2fe712d1b3582d040888a7389a8a7c1db2ea319bb6bdf53226531ff9b6620bc" Oct 30 04:19:20.826576 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.826536 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2fe712d1b3582d040888a7389a8a7c1db2ea319bb6bdf53226531ff9b6620bc"} err="failed to get container status \"e2fe712d1b3582d040888a7389a8a7c1db2ea319bb6bdf53226531ff9b6620bc\": rpc error: code = NotFound desc = could not find container \"e2fe712d1b3582d040888a7389a8a7c1db2ea319bb6bdf53226531ff9b6620bc\": container with ID starting with e2fe712d1b3582d040888a7389a8a7c1db2ea319bb6bdf53226531ff9b6620bc not found: ID does not exist" Oct 30 04:19:20.826576 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.826550 5882 scope.go:122] "RemoveContainer" containerID="c68f4fd6143ccce635e4e812c6c4d556df3c2e10a791599b1041ca3dcac1035c" Oct 30 04:19:20.826776 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:19:20.826754 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c68f4fd6143ccce635e4e812c6c4d556df3c2e10a791599b1041ca3dcac1035c\": container with ID starting with c68f4fd6143ccce635e4e812c6c4d556df3c2e10a791599b1041ca3dcac1035c not found: ID does not exist" containerID="c68f4fd6143ccce635e4e812c6c4d556df3c2e10a791599b1041ca3dcac1035c" Oct 30 04:19:20.826834 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.826786 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c68f4fd6143ccce635e4e812c6c4d556df3c2e10a791599b1041ca3dcac1035c"} err="failed to get container status \"c68f4fd6143ccce635e4e812c6c4d556df3c2e10a791599b1041ca3dcac1035c\": rpc error: code = NotFound desc = could not find container \"c68f4fd6143ccce635e4e812c6c4d556df3c2e10a791599b1041ca3dcac1035c\": container with ID starting with c68f4fd6143ccce635e4e812c6c4d556df3c2e10a791599b1041ca3dcac1035c not found: ID does not exist" Oct 30 04:19:20.874914 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.874877 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:19:20.893127 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:20.893100 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:19:22.010203 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:22.010168 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cd1858d-e47d-4405-ad1d-33b83b43846d" path="/var/lib/kubelet/pods/0cd1858d-e47d-4405-ad1d-33b83b43846d/volumes" Oct 30 04:19:23.941770 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:19:23.941736 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:19:27.695666 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.695599 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll"] Oct 30 04:19:27.696038 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.695946 5882 cpu_manager.go:414] "containerMap: removing container" podUID="78072d40-0ad6-4a50-a475-652f1cdd980c" containerName="alertmanager" Oct 30 04:19:27.696038 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.695958 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78072d40-0ad6-4a50-a475-652f1cdd980c" containerName="alertmanager" Oct 30 04:19:27.696038 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.695973 5882 cpu_manager.go:414] "containerMap: removing container" podUID="78072d40-0ad6-4a50-a475-652f1cdd980c" containerName="init-config-reloader" Oct 30 04:19:27.696038 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.695980 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78072d40-0ad6-4a50-a475-652f1cdd980c" containerName="init-config-reloader" Oct 30 04:19:27.696038 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.695994 5882 cpu_manager.go:414] "containerMap: removing container" podUID="1cb2478f-259b-4eae-9fee-4d527b0c0143" containerName="perses" Oct 30 04:19:27.696038 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.695999 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb2478f-259b-4eae-9fee-4d527b0c0143" containerName="perses" Oct 30 04:19:27.696038 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696008 5882 cpu_manager.go:414] "containerMap: removing container" podUID="0cd1858d-e47d-4405-ad1d-33b83b43846d" containerName="prometheus" Oct 30 04:19:27.696038 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696013 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd1858d-e47d-4405-ad1d-33b83b43846d" containerName="prometheus" Oct 30 04:19:27.696038 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696019 5882 cpu_manager.go:414] "containerMap: removing container" podUID="b795fd60-5d7d-43aa-96ec-4b130206e6f1" containerName="ta-container" Oct 30 04:19:27.696038 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696023 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b795fd60-5d7d-43aa-96ec-4b130206e6f1" containerName="ta-container" Oct 30 04:19:27.696038 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696032 5882 cpu_manager.go:414] "containerMap: removing container" podUID="0cd1858d-e47d-4405-ad1d-33b83b43846d" containerName="thanos-sidecar" Oct 30 04:19:27.696038 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696036 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd1858d-e47d-4405-ad1d-33b83b43846d" containerName="thanos-sidecar" Oct 30 04:19:27.696404 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696045 5882 cpu_manager.go:414] "containerMap: removing container" podUID="84697a88-0f07-445d-9284-b26d20b6d7f1" containerName="ta-container" Oct 30 04:19:27.696404 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696050 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84697a88-0f07-445d-9284-b26d20b6d7f1" containerName="ta-container" Oct 30 04:19:27.696404 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696065 5882 cpu_manager.go:414] "containerMap: removing container" podUID="0cd1858d-e47d-4405-ad1d-33b83b43846d" containerName="config-reloader" Oct 30 04:19:27.696404 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696070 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd1858d-e47d-4405-ad1d-33b83b43846d" containerName="config-reloader" Oct 30 04:19:27.696404 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696077 5882 cpu_manager.go:414] "containerMap: removing container" podUID="78072d40-0ad6-4a50-a475-652f1cdd980c" containerName="config-reloader" Oct 30 04:19:27.696404 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696082 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78072d40-0ad6-4a50-a475-652f1cdd980c" containerName="config-reloader" Oct 30 04:19:27.696404 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696089 5882 cpu_manager.go:414] "containerMap: removing container" podUID="0cd1858d-e47d-4405-ad1d-33b83b43846d" containerName="init-config-reloader" Oct 30 04:19:27.696404 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696094 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd1858d-e47d-4405-ad1d-33b83b43846d" containerName="init-config-reloader" Oct 30 04:19:27.696404 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696138 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="84697a88-0f07-445d-9284-b26d20b6d7f1" containerName="ta-container" Oct 30 04:19:27.696404 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696146 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd1858d-e47d-4405-ad1d-33b83b43846d" containerName="config-reloader" Oct 30 04:19:27.696404 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696152 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="78072d40-0ad6-4a50-a475-652f1cdd980c" containerName="config-reloader" Oct 30 04:19:27.696404 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696158 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="78072d40-0ad6-4a50-a475-652f1cdd980c" containerName="alertmanager" Oct 30 04:19:27.696404 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696166 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb2478f-259b-4eae-9fee-4d527b0c0143" containerName="perses" Oct 30 04:19:27.696404 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696171 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd1858d-e47d-4405-ad1d-33b83b43846d" containerName="prometheus" Oct 30 04:19:27.696404 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696176 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b795fd60-5d7d-43aa-96ec-4b130206e6f1" containerName="ta-container" Oct 30 04:19:27.696404 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.696181 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd1858d-e47d-4405-ad1d-33b83b43846d" containerName="thanos-sidecar" Oct 30 04:19:27.700700 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.700679 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll" Oct 30 04:19:27.718485 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.718456 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll"] Oct 30 04:19:27.731419 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.731392 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 04:19:27.731542 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.731396 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 04:19:27.731542 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.731453 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 04:19:27.738917 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.738883 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qm5p\" (UniqueName: \"kubernetes.io/projected/0b8d3c67-6238-4b04-9761-dec3d942194a-kube-api-access-6qm5p\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll\" (UID: \"0b8d3c67-6238-4b04-9761-dec3d942194a\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll" Oct 30 04:19:27.839534 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.839495 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qm5p\" (UniqueName: \"kubernetes.io/projected/0b8d3c67-6238-4b04-9761-dec3d942194a-kube-api-access-6qm5p\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll\" (UID: \"0b8d3c67-6238-4b04-9761-dec3d942194a\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll" Oct 30 04:19:27.864915 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:27.864885 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qm5p\" (UniqueName: \"kubernetes.io/projected/0b8d3c67-6238-4b04-9761-dec3d942194a-kube-api-access-6qm5p\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll\" (UID: \"0b8d3c67-6238-4b04-9761-dec3d942194a\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll" Oct 30 04:19:28.039744 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.039720 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 04:19:28.044892 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.044873 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll" Oct 30 04:19:28.341048 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.341016 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll"] Oct 30 04:19:28.342699 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:19:28.342667 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b8d3c67_6238_4b04_9761_dec3d942194a.slice/crio-ea03e3f5c803c14b2a2eb9483f090a30e6b6522501745a1fe44f931282017a11 WatchSource:0}: Error finding container ea03e3f5c803c14b2a2eb9483f090a30e6b6522501745a1fe44f931282017a11: Status 404 returned error can't find the container with id ea03e3f5c803c14b2a2eb9483f090a30e6b6522501745a1fe44f931282017a11 Oct 30 04:19:28.699864 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.699782 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:19:28.703522 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.703498 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:28.717174 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.717140 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:19:28.720348 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.720327 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:28.724829 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.724805 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:19:28.736688 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.736664 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:19:28.739274 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.739254 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 30 04:19:28.740474 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.740447 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:19:28.747315 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.747237 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:28.747501 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.747478 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ed10b143-7033-4b0f-8eaf-ca2050be68ac-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:28.747628 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.747522 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqkj8\" (UniqueName: \"kubernetes.io/projected/ed10b143-7033-4b0f-8eaf-ca2050be68ac-kube-api-access-dqkj8\") pod \"data-science-collector-collector-1\" (UID: \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:28.747707 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.747638 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:28.747707 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.747659 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:28.762506 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.762473 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:19:28.762506 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.762473 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:19:28.762736 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.762648 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-wvqg5\"" Oct 30 04:19:28.762736 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.762681 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:19:28.814111 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.814081 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp"] Oct 30 04:19:28.817484 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.817462 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" Oct 30 04:19:28.832498 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.832472 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll" event={"ID":"0b8d3c67-6238-4b04-9761-dec3d942194a","Type":"ContainerStarted","Data":"51d1e53b2040eb8cf1ea961df4e6e248c57215882dadaed46824df55479fa3e0"} Oct 30 04:19:28.832655 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.832506 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll" event={"ID":"0b8d3c67-6238-4b04-9761-dec3d942194a","Type":"ContainerStarted","Data":"ea03e3f5c803c14b2a2eb9483f090a30e6b6522501745a1fe44f931282017a11"} Oct 30 04:19:28.834372 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.834356 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-bjf7t\"" Oct 30 04:19:28.834490 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.834473 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 30 04:19:28.838373 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.838348 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp"] Oct 30 04:19:28.840837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.840816 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 30 04:19:28.848341 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.848306 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:28.848480 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.848360 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ed10b143-7033-4b0f-8eaf-ca2050be68ac-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:28.848480 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.848387 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqkj8\" (UniqueName: \"kubernetes.io/projected/ed10b143-7033-4b0f-8eaf-ca2050be68ac-kube-api-access-dqkj8\") pod \"data-science-collector-collector-1\" (UID: \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:28.848480 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.848410 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/4a13a26d-0e0b-4817-adda-af696009974d-plugins\") pod \"data-science-perses-0\" (UID: \"4a13a26d-0e0b-4817-adda-af696009974d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:28.848480 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.848470 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:28.848736 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.848490 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:28.848736 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.848535 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a13a26d-0e0b-4817-adda-af696009974d-config\") pod \"data-science-perses-0\" (UID: \"4a13a26d-0e0b-4817-adda-af696009974d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:28.848736 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.848675 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6w9s\" (UniqueName: \"kubernetes.io/projected/4a13a26d-0e0b-4817-adda-af696009974d-kube-api-access-w6w9s\") pod \"data-science-perses-0\" (UID: \"4a13a26d-0e0b-4817-adda-af696009974d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:28.849208 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.849173 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ed10b143-7033-4b0f-8eaf-ca2050be68ac-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:28.850846 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.850822 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:28.850966 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.850921 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:28.851255 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.851237 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:28.873476 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.873450 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqkj8\" (UniqueName: \"kubernetes.io/projected/ed10b143-7033-4b0f-8eaf-ca2050be68ac-kube-api-access-dqkj8\") pod \"data-science-collector-collector-1\" (UID: \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:28.910062 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.910006 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll" podStartSLOduration=1.909990672 podStartE2EDuration="1.909990672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:28.909039876 +0000 UTC m=+5881.023354852" watchObservedRunningTime="2026-10-30 04:19:28.909990672 +0000 UTC m=+5881.024305647" Oct 30 04:19:28.949232 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.949197 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl6m8\" (UniqueName: \"kubernetes.io/projected/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-kube-api-access-wl6m8\") pod \"data-science-collector-targetallocator-6c5b676cd5-cp4tp\" (UID: \"18bfa56e-ecfc-40a1-9bcc-0cee9754b826\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" Oct 30 04:19:28.949393 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.949247 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/4a13a26d-0e0b-4817-adda-af696009974d-plugins\") pod \"data-science-perses-0\" (UID: \"4a13a26d-0e0b-4817-adda-af696009974d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:28.949393 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.949336 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6c5b676cd5-cp4tp\" (UID: \"18bfa56e-ecfc-40a1-9bcc-0cee9754b826\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" Oct 30 04:19:28.949468 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.949412 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a13a26d-0e0b-4817-adda-af696009974d-config\") pod \"data-science-perses-0\" (UID: \"4a13a26d-0e0b-4817-adda-af696009974d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:28.949513 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.949500 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-ta-internal\") pod \"data-science-collector-targetallocator-6c5b676cd5-cp4tp\" (UID: \"18bfa56e-ecfc-40a1-9bcc-0cee9754b826\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" Oct 30 04:19:28.949553 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.949542 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6w9s\" (UniqueName: \"kubernetes.io/projected/4a13a26d-0e0b-4817-adda-af696009974d-kube-api-access-w6w9s\") pod \"data-science-perses-0\" (UID: \"4a13a26d-0e0b-4817-adda-af696009974d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:28.949750 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.949731 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/4a13a26d-0e0b-4817-adda-af696009974d-plugins\") pod \"data-science-perses-0\" (UID: \"4a13a26d-0e0b-4817-adda-af696009974d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:28.950192 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.950150 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a13a26d-0e0b-4817-adda-af696009974d-config\") pod \"data-science-perses-0\" (UID: \"4a13a26d-0e0b-4817-adda-af696009974d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:28.975191 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:28.975162 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6w9s\" (UniqueName: \"kubernetes.io/projected/4a13a26d-0e0b-4817-adda-af696009974d-kube-api-access-w6w9s\") pod \"data-science-perses-0\" (UID: \"4a13a26d-0e0b-4817-adda-af696009974d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:29.016323 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.016290 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:29.050002 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.049970 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6c5b676cd5-cp4tp\" (UID: \"18bfa56e-ecfc-40a1-9bcc-0cee9754b826\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" Oct 30 04:19:29.050197 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.050039 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-ta-internal\") pod \"data-science-collector-targetallocator-6c5b676cd5-cp4tp\" (UID: \"18bfa56e-ecfc-40a1-9bcc-0cee9754b826\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" Oct 30 04:19:29.050197 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.050078 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl6m8\" (UniqueName: \"kubernetes.io/projected/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-kube-api-access-wl6m8\") pod \"data-science-collector-targetallocator-6c5b676cd5-cp4tp\" (UID: \"18bfa56e-ecfc-40a1-9bcc-0cee9754b826\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" Oct 30 04:19:29.050983 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.050955 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-ta-internal\") pod \"data-science-collector-targetallocator-6c5b676cd5-cp4tp\" (UID: \"18bfa56e-ecfc-40a1-9bcc-0cee9754b826\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" Oct 30 04:19:29.052475 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.052449 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6c5b676cd5-cp4tp\" (UID: \"18bfa56e-ecfc-40a1-9bcc-0cee9754b826\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" Oct 30 04:19:29.080218 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.080188 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl6m8\" (UniqueName: \"kubernetes.io/projected/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-kube-api-access-wl6m8\") pod \"data-science-collector-targetallocator-6c5b676cd5-cp4tp\" (UID: \"18bfa56e-ecfc-40a1-9bcc-0cee9754b826\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" Oct 30 04:19:29.129215 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.129158 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" Oct 30 04:19:29.314951 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.314923 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:19:29.315799 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:19:29.315768 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded10b143_7033_4b0f_8eaf_ca2050be68ac.slice/crio-c368656dbf9559556000fc39efcd38a1c29fe9ce6130be512a7c3f9937599876 WatchSource:0}: Error finding container c368656dbf9559556000fc39efcd38a1c29fe9ce6130be512a7c3f9937599876: Status 404 returned error can't find the container with id c368656dbf9559556000fc39efcd38a1c29fe9ce6130be512a7c3f9937599876 Oct 30 04:19:29.318132 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.318100 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:19:29.318223 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.318174 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:19:29.429690 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.429574 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp"] Oct 30 04:19:29.430886 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:19:29.430849 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18bfa56e_ecfc_40a1_9bcc_0cee9754b826.slice/crio-d3e0abe12774930d8f22a3fe10d2783e0207ce00c0754cb21d600edf9a4738f4 WatchSource:0}: Error finding container d3e0abe12774930d8f22a3fe10d2783e0207ce00c0754cb21d600edf9a4738f4: Status 404 returned error can't find the container with id d3e0abe12774930d8f22a3fe10d2783e0207ce00c0754cb21d600edf9a4738f4 Oct 30 04:19:29.433448 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.433413 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:19:29.433533 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.433483 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:19:29.839318 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.839281 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" event={"ID":"18bfa56e-ecfc-40a1-9bcc-0cee9754b826","Type":"ContainerStarted","Data":"95ebdab8e7824882a3819f00c0548caf6b82fc1c7cb761f58aa4cb613131bf8f"} Oct 30 04:19:29.839813 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.839330 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" event={"ID":"18bfa56e-ecfc-40a1-9bcc-0cee9754b826","Type":"ContainerStarted","Data":"d3e0abe12774930d8f22a3fe10d2783e0207ce00c0754cb21d600edf9a4738f4"} Oct 30 04:19:29.839813 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.839359 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" Oct 30 04:19:29.840758 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.840728 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" podUID="18bfa56e-ecfc-40a1-9bcc-0cee9754b826" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.171:8080/readyz\": dial tcp 10.134.0.171:8080: connect: connection refused" Oct 30 04:19:29.840970 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.840945 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ed10b143-7033-4b0f-8eaf-ca2050be68ac","Type":"ContainerStarted","Data":"5cbe73f66fabbcc63de75922cb43ef1ede6423c9c73e5b67c113896c05ec12d6"} Oct 30 04:19:29.841014 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.840992 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ed10b143-7033-4b0f-8eaf-ca2050be68ac","Type":"ContainerStarted","Data":"c368656dbf9559556000fc39efcd38a1c29fe9ce6130be512a7c3f9937599876"} Oct 30 04:19:29.884503 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.884439 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" podStartSLOduration=1.884422849 podStartE2EDuration="1.884422849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:29.883008734 +0000 UTC m=+5881.997323721" watchObservedRunningTime="2026-10-30 04:19:29.884422849 +0000 UTC m=+5881.998737825" Oct 30 04:19:29.928922 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:29.928869 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.928853412 podStartE2EDuration="1.928853412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:29.928133288 +0000 UTC m=+5882.042448264" watchObservedRunningTime="2026-10-30 04:19:29.928853412 +0000 UTC m=+5882.043168383" Oct 30 04:19:30.847281 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:30.847250 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" Oct 30 04:19:31.975466 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:31.975422 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"4a13a26d-0e0b-4817-adda-af696009974d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:32.008263 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.008226 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"4a13a26d-0e0b-4817-adda-af696009974d\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:32.076084 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.076049 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"4a13a26d-0e0b-4817-adda-af696009974d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:32.076264 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.076183 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"4a13a26d-0e0b-4817-adda-af696009974d\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:32.092224 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.092194 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"4a13a26d-0e0b-4817-adda-af696009974d\") DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:32.195658 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.195606 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"4a13a26d-0e0b-4817-adda-af696009974d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2c422b71c1e0b8fdf34659f2cef7846e06b4353ed88027dc99ade5d7fc29e9d6/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:32.227335 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.227274 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"4a13a26d-0e0b-4817-adda-af696009974d\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:32.332770 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.332735 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:32.344363 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.344338 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:19:32.349378 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.349356 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.371051 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.371029 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-5km5j\"" Oct 30 04:19:32.371198 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.371072 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 04:19:32.371286 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.371272 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 04:19:32.374556 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.374531 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:19:32.379793 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.379759 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:19:32.402562 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.402391 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:19:32.407173 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.406764 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.435000 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.434971 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:19:32.470863 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.470836 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:19:32.471041 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.471021 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 04:19:32.471102 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.471024 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 04:19:32.471146 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.471133 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 04:19:32.471223 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.471208 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 04:19:32.471277 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.471214 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 04:19:32.471277 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.471242 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-mcw24\"" Oct 30 04:19:32.471359 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.471299 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 04:19:32.480775 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.480717 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.480775 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.480751 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.480932 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.480778 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.480932 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.480890 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.481008 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.480966 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.481008 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.480993 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btxr6\" (UniqueName: \"kubernetes.io/projected/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-kube-api-access-btxr6\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.481084 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.481013 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.494662 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.494605 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 04:19:32.494934 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.494917 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 04:19:32.494934 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.494925 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 04:19:32.581471 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.581442 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.581676 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.581480 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.581676 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.581506 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.581676 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.581639 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.581864 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.581681 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.581864 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.581709 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5979b0e7-e3e6-436d-a626-d47d1f860a28-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.581864 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.581842 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.582023 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.581875 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.582023 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.581940 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.582023 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.581955 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.582023 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.582014 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.582217 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.582068 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5fn8\" (UniqueName: \"kubernetes.io/projected/5979b0e7-e3e6-436d-a626-d47d1f860a28-kube-api-access-z5fn8\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.582217 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.582093 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.582217 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.582165 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.582388 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.582246 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.582388 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.582288 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btxr6\" (UniqueName: \"kubernetes.io/projected/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-kube-api-access-btxr6\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.582388 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.582320 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.582388 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.582379 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.582748 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.582416 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5979b0e7-e3e6-436d-a626-d47d1f860a28-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.582748 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.582444 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.584502 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.584476 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.584698 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.584678 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.584768 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.584707 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.584768 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.584684 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.585070 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.585053 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.607270 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.607243 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btxr6\" (UniqueName: \"kubernetes.io/projected/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-kube-api-access-btxr6\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.630772 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.630744 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:19:32.631249 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:19:32.631227 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a13a26d_0e0b_4817_adda_af696009974d.slice/crio-0a856ce0014d4c1ebca18ba4be23f79d764f9f5c3c42e893d167bf3fee4fe65a WatchSource:0}: Error finding container 0a856ce0014d4c1ebca18ba4be23f79d764f9f5c3c42e893d167bf3fee4fe65a: Status 404 returned error can't find the container with id 0a856ce0014d4c1ebca18ba4be23f79d764f9f5c3c42e893d167bf3fee4fe65a Oct 30 04:19:32.675724 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.675696 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:32.682963 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.682937 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.683022 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.682990 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.683073 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.683044 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5979b0e7-e3e6-436d-a626-d47d1f860a28-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.683108 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.683075 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.683108 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.683095 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.683213 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.683117 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.683213 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.683165 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5fn8\" (UniqueName: \"kubernetes.io/projected/5979b0e7-e3e6-436d-a626-d47d1f860a28-kube-api-access-z5fn8\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.683213 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.683199 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.683357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.683246 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.683357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.683303 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.683357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.683334 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5979b0e7-e3e6-436d-a626-d47d1f860a28-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.683533 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.683362 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.683604 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.683579 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.683753 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.683710 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.684260 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.684233 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.685981 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.685955 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.686519 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.686492 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.686750 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.686730 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.687090 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.687064 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.687302 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.687279 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.687392 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.687368 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5979b0e7-e3e6-436d-a626-d47d1f860a28-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.688369 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.688343 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.688676 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.688650 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5979b0e7-e3e6-436d-a626-d47d1f860a28-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.712942 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.712909 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5fn8\" (UniqueName: \"kubernetes.io/projected/5979b0e7-e3e6-436d-a626-d47d1f860a28-kube-api-access-z5fn8\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:32.856421 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.856377 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"4a13a26d-0e0b-4817-adda-af696009974d","Type":"ContainerStarted","Data":"a0fffb1c1a5535a59fc82945cf7117b91f9514958841ddb2c7fe8ef08904a246"} Oct 30 04:19:32.856421 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.856428 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"4a13a26d-0e0b-4817-adda-af696009974d","Type":"ContainerStarted","Data":"0a856ce0014d4c1ebca18ba4be23f79d764f9f5c3c42e893d167bf3fee4fe65a"} Oct 30 04:19:32.856716 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.856456 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:32.900094 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.900041 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=4.90001405 podStartE2EDuration="4.90001405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:32.899683383 +0000 UTC m=+5885.013998358" watchObservedRunningTime="2026-10-30 04:19:32.90001405 +0000 UTC m=+5885.014329029" Oct 30 04:19:32.987125 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:32.987099 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:19:32.988649 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:19:32.988610 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5c4add_ffce_4b84_8b51_b0caecc1f3b5.slice/crio-5f88c5e8b555bbaaae620016820424b54dead750aea53aa1fb0b6349c9bd5618 WatchSource:0}: Error finding container 5f88c5e8b555bbaaae620016820424b54dead750aea53aa1fb0b6349c9bd5618: Status 404 returned error can't find the container with id 5f88c5e8b555bbaaae620016820424b54dead750aea53aa1fb0b6349c9bd5618 Oct 30 04:19:33.123663 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:33.123630 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll"] Oct 30 04:19:33.123848 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:33.123790 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll" podUID="0b8d3c67-6238-4b04-9761-dec3d942194a" containerName="thanos-querier" containerID="cri-o://51d1e53b2040eb8cf1ea961df4e6e248c57215882dadaed46824df55479fa3e0" gracePeriod=30 Oct 30 04:19:33.472238 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:33.472210 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll" Oct 30 04:19:33.490864 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:33.490813 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b8d3c67-6238-4b04-9761-dec3d942194a-kube-api-access-6qm5p\" (UniqueName: \"kubernetes.io/projected/0b8d3c67-6238-4b04-9761-dec3d942194a-kube-api-access-6qm5p\") pod \"0b8d3c67-6238-4b04-9761-dec3d942194a\" (UID: \"0b8d3c67-6238-4b04-9761-dec3d942194a\") " Oct 30 04:19:33.492841 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:33.492819 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8d3c67-6238-4b04-9761-dec3d942194a-kube-api-access-6qm5p" pod "0b8d3c67-6238-4b04-9761-dec3d942194a" (UID: "0b8d3c67-6238-4b04-9761-dec3d942194a"). InnerVolumeSpecName "kube-api-access-6qm5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:33.591931 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:33.591892 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qm5p\" (UniqueName: \"kubernetes.io/projected/0b8d3c67-6238-4b04-9761-dec3d942194a-kube-api-access-6qm5p\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:33.868171 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:33.868127 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5","Type":"ContainerStarted","Data":"6188b21a274ce3e4e63c0d6bd8d814f57dcaea418980886b855fae3d403b5fe9"} Oct 30 04:19:33.868310 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:33.868213 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5","Type":"ContainerStarted","Data":"5f88c5e8b555bbaaae620016820424b54dead750aea53aa1fb0b6349c9bd5618"} Oct 30 04:19:33.869668 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:33.869638 5882 generic.go:356] "Generic (PLEG): container finished" podID="0b8d3c67-6238-4b04-9761-dec3d942194a" containerID="51d1e53b2040eb8cf1ea961df4e6e248c57215882dadaed46824df55479fa3e0" exitCode=0 Oct 30 04:19:33.869772 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:33.869677 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll" Oct 30 04:19:33.869772 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:33.869725 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll" event={"ID":"0b8d3c67-6238-4b04-9761-dec3d942194a","Type":"ContainerDied","Data":"51d1e53b2040eb8cf1ea961df4e6e248c57215882dadaed46824df55479fa3e0"} Oct 30 04:19:33.869903 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:33.869768 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll" event={"ID":"0b8d3c67-6238-4b04-9761-dec3d942194a","Type":"ContainerDied","Data":"ea03e3f5c803c14b2a2eb9483f090a30e6b6522501745a1fe44f931282017a11"} Oct 30 04:19:33.869903 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:33.869794 5882 scope.go:122] "RemoveContainer" containerID="51d1e53b2040eb8cf1ea961df4e6e248c57215882dadaed46824df55479fa3e0" Oct 30 04:19:33.878788 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:33.878768 5882 scope.go:122] "RemoveContainer" containerID="51d1e53b2040eb8cf1ea961df4e6e248c57215882dadaed46824df55479fa3e0" Oct 30 04:19:33.879103 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:19:33.879083 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51d1e53b2040eb8cf1ea961df4e6e248c57215882dadaed46824df55479fa3e0\": container with ID starting with 51d1e53b2040eb8cf1ea961df4e6e248c57215882dadaed46824df55479fa3e0 not found: ID does not exist" containerID="51d1e53b2040eb8cf1ea961df4e6e248c57215882dadaed46824df55479fa3e0" Oct 30 04:19:33.879155 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:33.879113 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51d1e53b2040eb8cf1ea961df4e6e248c57215882dadaed46824df55479fa3e0"} err="failed to get container status \"51d1e53b2040eb8cf1ea961df4e6e248c57215882dadaed46824df55479fa3e0\": rpc error: code = NotFound desc = could not find container \"51d1e53b2040eb8cf1ea961df4e6e248c57215882dadaed46824df55479fa3e0\": container with ID starting with 51d1e53b2040eb8cf1ea961df4e6e248c57215882dadaed46824df55479fa3e0 not found: ID does not exist" Oct 30 04:19:33.956689 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:33.956656 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll"] Oct 30 04:19:33.961037 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:33.961012 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dwmll"] Oct 30 04:19:34.010551 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:34.010515 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b8d3c67-6238-4b04-9761-dec3d942194a" path="/var/lib/kubelet/pods/0b8d3c67-6238-4b04-9761-dec3d942194a/volumes" Oct 30 04:19:37.888310 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:37.888275 5882 generic.go:356] "Generic (PLEG): container finished" podID="3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" containerID="6188b21a274ce3e4e63c0d6bd8d814f57dcaea418980886b855fae3d403b5fe9" exitCode=0 Oct 30 04:19:37.888716 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:37.888346 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5","Type":"ContainerDied","Data":"6188b21a274ce3e4e63c0d6bd8d814f57dcaea418980886b855fae3d403b5fe9"} Oct 30 04:19:38.894730 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:38.894688 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5","Type":"ContainerStarted","Data":"0af55161445b7af6fa2176020e7b9d92f21a9429146e35a13a7cb1e4135d9a99"} Oct 30 04:19:38.895179 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:38.894758 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5","Type":"ContainerStarted","Data":"4189e054fe3142f17b49127b1ca2bc9d837255ee2d63d10cb3b2886e98ee1da5"} Oct 30 04:19:38.895179 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:38.894950 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:38.972628 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:38.972560 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.378566721 podStartE2EDuration="6.972546179s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.593979458s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:38.958444555 +0000 UTC m=+5891.072759529" watchObservedRunningTime="2026-10-30 04:19:38.972546179 +0000 UTC m=+5891.086861181" Oct 30 04:19:42.162886 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:42.162850 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:42.195902 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:42.195870 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:42.264184 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:42.264152 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:42.264333 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:42.264266 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:42.280337 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:42.280309 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") DevicePath \"csi-927d18ef2eb99137c68fbe5e0bb8ea629bb5219ed56cdede72da7c0f045be95e\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:42.384399 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:42.384370 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a535375d22e374ba333968465471c7604a06f9d95a513910fe4bb083b629e986/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:42.413999 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:42.413942 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:42.626699 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:42.626650 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:42.948558 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:19:42.948522 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5979b0e7_e3e6_436d_a626_d47d1f860a28.slice/crio-bc6cc21edcfbe82fa21dddecedd391d50dd19655e3d471a885fd13036193ee25 WatchSource:0}: Error finding container bc6cc21edcfbe82fa21dddecedd391d50dd19655e3d471a885fd13036193ee25: Status 404 returned error can't find the container with id bc6cc21edcfbe82fa21dddecedd391d50dd19655e3d471a885fd13036193ee25 Oct 30 04:19:42.962537 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:42.962513 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:19:43.871770 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:43.871737 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:19:43.919599 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:43.919557 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"5979b0e7-e3e6-436d-a626-d47d1f860a28","Type":"ContainerStarted","Data":"1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca"} Oct 30 04:19:43.919828 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:43.919657 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"5979b0e7-e3e6-436d-a626-d47d1f860a28","Type":"ContainerStarted","Data":"bc6cc21edcfbe82fa21dddecedd391d50dd19655e3d471a885fd13036193ee25"} Oct 30 04:19:44.900433 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:44.900399 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:19:46.933643 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:46.933534 5882 generic.go:356] "Generic (PLEG): container finished" podID="5979b0e7-e3e6-436d-a626-d47d1f860a28" containerID="1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca" exitCode=0 Oct 30 04:19:46.933643 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:46.933612 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"5979b0e7-e3e6-436d-a626-d47d1f860a28","Type":"ContainerDied","Data":"1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca"} Oct 30 04:19:47.941548 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:47.941510 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"5979b0e7-e3e6-436d-a626-d47d1f860a28","Type":"ContainerStarted","Data":"e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869"} Oct 30 04:19:47.942009 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:47.941601 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"5979b0e7-e3e6-436d-a626-d47d1f860a28","Type":"ContainerStarted","Data":"ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747"} Oct 30 04:19:47.942009 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:47.941650 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"5979b0e7-e3e6-436d-a626-d47d1f860a28","Type":"ContainerStarted","Data":"99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e"} Oct 30 04:19:48.025389 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:48.024868 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=12.830259178 podStartE2EDuration="16.024851161s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.194591983s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:48.010095232 +0000 UTC m=+5900.124410217" watchObservedRunningTime="2026-10-30 04:19:48.024851161 +0000 UTC m=+5900.139166137" Oct 30 04:19:49.744582 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:49.744551 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:19:49.745055 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:49.744765 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="ed10b143-7033-4b0f-8eaf-ca2050be68ac" containerName="otc-container" containerID="cri-o://5cbe73f66fabbcc63de75922cb43ef1ede6423c9c73e5b67c113896c05ec12d6" gracePeriod=30 Oct 30 04:19:49.975364 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:49.975327 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg"] Oct 30 04:19:49.975755 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:49.975741 5882 cpu_manager.go:414] "containerMap: removing container" podUID="0b8d3c67-6238-4b04-9761-dec3d942194a" containerName="thanos-querier" Oct 30 04:19:49.975805 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:49.975756 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b8d3c67-6238-4b04-9761-dec3d942194a" containerName="thanos-querier" Oct 30 04:19:49.975841 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:49.975816 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b8d3c67-6238-4b04-9761-dec3d942194a" containerName="thanos-querier" Oct 30 04:19:49.978874 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:49.978859 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" Oct 30 04:19:50.002473 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.002400 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg"] Oct 30 04:19:50.036277 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.036246 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gdpg\" (UniqueName: \"kubernetes.io/projected/007a10cc-ee5a-45d0-ab05-201155877f10-kube-api-access-4gdpg\") pod \"data-science-collector-targetallocator-5dd6975f86-vxrsg\" (UID: \"007a10cc-ee5a-45d0-ab05-201155877f10\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" Oct 30 04:19:50.036451 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.036310 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/007a10cc-ee5a-45d0-ab05-201155877f10-ta-internal\") pod \"data-science-collector-targetallocator-5dd6975f86-vxrsg\" (UID: \"007a10cc-ee5a-45d0-ab05-201155877f10\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" Oct 30 04:19:50.036451 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.036372 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/007a10cc-ee5a-45d0-ab05-201155877f10-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5dd6975f86-vxrsg\" (UID: \"007a10cc-ee5a-45d0-ab05-201155877f10\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" Oct 30 04:19:50.137414 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.137381 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/007a10cc-ee5a-45d0-ab05-201155877f10-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5dd6975f86-vxrsg\" (UID: \"007a10cc-ee5a-45d0-ab05-201155877f10\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" Oct 30 04:19:50.137576 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.137452 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gdpg\" (UniqueName: \"kubernetes.io/projected/007a10cc-ee5a-45d0-ab05-201155877f10-kube-api-access-4gdpg\") pod \"data-science-collector-targetallocator-5dd6975f86-vxrsg\" (UID: \"007a10cc-ee5a-45d0-ab05-201155877f10\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" Oct 30 04:19:50.137576 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.137511 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/007a10cc-ee5a-45d0-ab05-201155877f10-ta-internal\") pod \"data-science-collector-targetallocator-5dd6975f86-vxrsg\" (UID: \"007a10cc-ee5a-45d0-ab05-201155877f10\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" Oct 30 04:19:50.138180 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.138158 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/007a10cc-ee5a-45d0-ab05-201155877f10-ta-internal\") pod \"data-science-collector-targetallocator-5dd6975f86-vxrsg\" (UID: \"007a10cc-ee5a-45d0-ab05-201155877f10\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" Oct 30 04:19:50.139789 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.139770 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/007a10cc-ee5a-45d0-ab05-201155877f10-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5dd6975f86-vxrsg\" (UID: \"007a10cc-ee5a-45d0-ab05-201155877f10\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" Oct 30 04:19:50.162563 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.162534 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gdpg\" (UniqueName: \"kubernetes.io/projected/007a10cc-ee5a-45d0-ab05-201155877f10-kube-api-access-4gdpg\") pod \"data-science-collector-targetallocator-5dd6975f86-vxrsg\" (UID: \"007a10cc-ee5a-45d0-ab05-201155877f10\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" Oct 30 04:19:50.290661 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.290561 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" Oct 30 04:19:50.584049 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.584006 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg"] Oct 30 04:19:50.586372 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:19:50.586345 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod007a10cc_ee5a_45d0_ab05_201155877f10.slice/crio-7325ecd4054799a8227e947450228b002d658c7f6fb128c7cc336baab9bf7154 WatchSource:0}: Error finding container 7325ecd4054799a8227e947450228b002d658c7f6fb128c7cc336baab9bf7154: Status 404 returned error can't find the container with id 7325ecd4054799a8227e947450228b002d658c7f6fb128c7cc336baab9bf7154 Oct 30 04:19:50.588661 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.588608 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:19:50.588743 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.588686 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:19:50.956755 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.956716 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" event={"ID":"007a10cc-ee5a-45d0-ab05-201155877f10","Type":"ContainerStarted","Data":"f8bbf79b25f358c125412f2065016cf091d01ad926ec30a5502749c128133e84"} Oct 30 04:19:50.957194 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.956773 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" Oct 30 04:19:50.957194 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.956788 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" event={"ID":"007a10cc-ee5a-45d0-ab05-201155877f10","Type":"ContainerStarted","Data":"7325ecd4054799a8227e947450228b002d658c7f6fb128c7cc336baab9bf7154"} Oct 30 04:19:50.957999 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.957960 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" podUID="007a10cc-ee5a-45d0-ab05-201155877f10" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.174:8080/readyz\": dial tcp 10.134.0.174:8080: connect: connection refused" Oct 30 04:19:50.958588 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.958562 5882 generic.go:356] "Generic (PLEG): container finished" podID="ed10b143-7033-4b0f-8eaf-ca2050be68ac" containerID="5cbe73f66fabbcc63de75922cb43ef1ede6423c9c73e5b67c113896c05ec12d6" exitCode=0 Oct 30 04:19:50.958708 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.958639 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ed10b143-7033-4b0f-8eaf-ca2050be68ac","Type":"ContainerDied","Data":"5cbe73f66fabbcc63de75922cb43ef1ede6423c9c73e5b67c113896c05ec12d6"} Oct 30 04:19:50.981183 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:50.981163 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:51.001790 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:51.001747 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" podStartSLOduration=2.001734068 podStartE2EDuration="2.001734068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:51.000922319 +0000 UTC m=+5903.115237296" watchObservedRunningTime="2026-10-30 04:19:51.001734068 +0000 UTC m=+5903.116049043" Oct 30 04:19:51.046987 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:51.046953 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed10b143-7033-4b0f-8eaf-ca2050be68ac-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ed10b143-7033-4b0f-8eaf-ca2050be68ac-otc-internal\") pod \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\" (UID: \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\") " Oct 30 04:19:51.047157 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:51.047011 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed10b143-7033-4b0f-8eaf-ca2050be68ac-kube-api-access-dqkj8\" (UniqueName: \"kubernetes.io/projected/ed10b143-7033-4b0f-8eaf-ca2050be68ac-kube-api-access-dqkj8\") pod \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\" (UID: \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\") " Oct 30 04:19:51.047157 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:51.047100 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-tls-certs\") pod \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\" (UID: \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\") " Oct 30 04:19:51.047296 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:51.047183 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-data-science-collector-ta-client-cert\") pod \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\" (UID: \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\") " Oct 30 04:19:51.047296 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:51.047226 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-monitoring-tls-certs\") pod \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\" (UID: \"ed10b143-7033-4b0f-8eaf-ca2050be68ac\") " Oct 30 04:19:51.049054 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:51.049004 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed10b143-7033-4b0f-8eaf-ca2050be68ac-otc-internal" pod "ed10b143-7033-4b0f-8eaf-ca2050be68ac" (UID: "ed10b143-7033-4b0f-8eaf-ca2050be68ac"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:51.049986 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:51.049963 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-monitoring-tls-certs" pod "ed10b143-7033-4b0f-8eaf-ca2050be68ac" (UID: "ed10b143-7033-4b0f-8eaf-ca2050be68ac"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:51.050087 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:51.049985 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed10b143-7033-4b0f-8eaf-ca2050be68ac-kube-api-access-dqkj8" pod "ed10b143-7033-4b0f-8eaf-ca2050be68ac" (UID: "ed10b143-7033-4b0f-8eaf-ca2050be68ac"). InnerVolumeSpecName "kube-api-access-dqkj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:51.050537 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:51.050512 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-tls-certs" pod "ed10b143-7033-4b0f-8eaf-ca2050be68ac" (UID: "ed10b143-7033-4b0f-8eaf-ca2050be68ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:51.050723 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:51.050703 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-data-science-collector-ta-client-cert" pod "ed10b143-7033-4b0f-8eaf-ca2050be68ac" (UID: "ed10b143-7033-4b0f-8eaf-ca2050be68ac"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:51.149022 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:51.148991 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:51.149130 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:51.149028 5882 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-monitoring-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:51.149130 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:51.149045 5882 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ed10b143-7033-4b0f-8eaf-ca2050be68ac-otc-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:51.149130 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:51.149059 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqkj8\" (UniqueName: \"kubernetes.io/projected/ed10b143-7033-4b0f-8eaf-ca2050be68ac-kube-api-access-dqkj8\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:51.149130 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:51.149073 5882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed10b143-7033-4b0f-8eaf-ca2050be68ac-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:51.965277 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:51.965245 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:51.965277 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:51.965255 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ed10b143-7033-4b0f-8eaf-ca2050be68ac","Type":"ContainerDied","Data":"c368656dbf9559556000fc39efcd38a1c29fe9ce6130be512a7c3f9937599876"} Oct 30 04:19:51.965802 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:51.965325 5882 scope.go:122] "RemoveContainer" containerID="5cbe73f66fabbcc63de75922cb43ef1ede6423c9c73e5b67c113896c05ec12d6" Oct 30 04:19:51.966959 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:51.966940 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" Oct 30 04:19:52.055966 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.055932 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp"] Oct 30 04:19:52.056155 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.056127 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" podUID="18bfa56e-ecfc-40a1-9bcc-0cee9754b826" containerName="ta-container" containerID="cri-o://95ebdab8e7824882a3819f00c0548caf6b82fc1c7cb761f58aa4cb613131bf8f" gracePeriod=30 Oct 30 04:19:52.069161 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.069131 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:19:52.075341 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.075317 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:19:52.306864 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.306831 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" Oct 30 04:19:52.375672 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.371961 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-ta-internal\" (UniqueName: \"kubernetes.io/configmap/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-ta-internal\") pod \"18bfa56e-ecfc-40a1-9bcc-0cee9754b826\" (UID: \"18bfa56e-ecfc-40a1-9bcc-0cee9754b826\") " Oct 30 04:19:52.375672 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.372068 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-kube-api-access-wl6m8\" (UniqueName: \"kubernetes.io/projected/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-kube-api-access-wl6m8\") pod \"18bfa56e-ecfc-40a1-9bcc-0cee9754b826\" (UID: \"18bfa56e-ecfc-40a1-9bcc-0cee9754b826\") " Oct 30 04:19:52.375672 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.372140 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-data-science-collector-ta-server-cert\") pod \"18bfa56e-ecfc-40a1-9bcc-0cee9754b826\" (UID: \"18bfa56e-ecfc-40a1-9bcc-0cee9754b826\") " Oct 30 04:19:52.376072 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.376039 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-ta-internal" pod "18bfa56e-ecfc-40a1-9bcc-0cee9754b826" (UID: "18bfa56e-ecfc-40a1-9bcc-0cee9754b826"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:52.377878 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.377849 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-data-science-collector-ta-server-cert" pod "18bfa56e-ecfc-40a1-9bcc-0cee9754b826" (UID: "18bfa56e-ecfc-40a1-9bcc-0cee9754b826"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:52.378052 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.378034 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-kube-api-access-wl6m8" pod "18bfa56e-ecfc-40a1-9bcc-0cee9754b826" (UID: "18bfa56e-ecfc-40a1-9bcc-0cee9754b826"). InnerVolumeSpecName "kube-api-access-wl6m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:52.473764 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.473734 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:52.473764 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.473761 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl6m8\" (UniqueName: \"kubernetes.io/projected/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-kube-api-access-wl6m8\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:52.473975 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.473774 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/18bfa56e-ecfc-40a1-9bcc-0cee9754b826-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:19:52.627576 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.627543 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:52.971495 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.971411 5882 generic.go:356] "Generic (PLEG): container finished" podID="18bfa56e-ecfc-40a1-9bcc-0cee9754b826" containerID="95ebdab8e7824882a3819f00c0548caf6b82fc1c7cb761f58aa4cb613131bf8f" exitCode=0 Oct 30 04:19:52.971495 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.971453 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" Oct 30 04:19:52.972021 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.971500 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" event={"ID":"18bfa56e-ecfc-40a1-9bcc-0cee9754b826","Type":"ContainerDied","Data":"95ebdab8e7824882a3819f00c0548caf6b82fc1c7cb761f58aa4cb613131bf8f"} Oct 30 04:19:52.972021 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.971542 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp" event={"ID":"18bfa56e-ecfc-40a1-9bcc-0cee9754b826","Type":"ContainerDied","Data":"d3e0abe12774930d8f22a3fe10d2783e0207ce00c0754cb21d600edf9a4738f4"} Oct 30 04:19:52.972021 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.971566 5882 scope.go:122] "RemoveContainer" containerID="95ebdab8e7824882a3819f00c0548caf6b82fc1c7cb761f58aa4cb613131bf8f" Oct 30 04:19:52.981230 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.981211 5882 scope.go:122] "RemoveContainer" containerID="95ebdab8e7824882a3819f00c0548caf6b82fc1c7cb761f58aa4cb613131bf8f" Oct 30 04:19:52.981501 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:19:52.981481 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ebdab8e7824882a3819f00c0548caf6b82fc1c7cb761f58aa4cb613131bf8f\": container with ID starting with 95ebdab8e7824882a3819f00c0548caf6b82fc1c7cb761f58aa4cb613131bf8f not found: ID does not exist" containerID="95ebdab8e7824882a3819f00c0548caf6b82fc1c7cb761f58aa4cb613131bf8f" Oct 30 04:19:52.981564 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:52.981513 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ebdab8e7824882a3819f00c0548caf6b82fc1c7cb761f58aa4cb613131bf8f"} err="failed to get container status \"95ebdab8e7824882a3819f00c0548caf6b82fc1c7cb761f58aa4cb613131bf8f\": rpc error: code = NotFound desc = could not find container \"95ebdab8e7824882a3819f00c0548caf6b82fc1c7cb761f58aa4cb613131bf8f\": container with ID starting with 95ebdab8e7824882a3819f00c0548caf6b82fc1c7cb761f58aa4cb613131bf8f not found: ID does not exist" Oct 30 04:19:53.026948 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:53.026911 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp"] Oct 30 04:19:53.040820 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:53.040790 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c5b676cd5-cp4tp"] Oct 30 04:19:53.945866 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:19:53.943784 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:19:54.009216 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:54.009181 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18bfa56e-ecfc-40a1-9bcc-0cee9754b826" path="/var/lib/kubelet/pods/18bfa56e-ecfc-40a1-9bcc-0cee9754b826/volumes" Oct 30 04:19:54.009588 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:54.009568 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed10b143-7033-4b0f-8eaf-ca2050be68ac" path="/var/lib/kubelet/pods/ed10b143-7033-4b0f-8eaf-ca2050be68ac/volumes" Oct 30 04:19:57.627831 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:57.627793 5882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:57.633289 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:57.633268 5882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:19:58.001382 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:19:58.001356 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:05.675081 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:05.675044 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w"] Oct 30 04:20:05.675468 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:05.675416 5882 cpu_manager.go:414] "containerMap: removing container" podUID="ed10b143-7033-4b0f-8eaf-ca2050be68ac" containerName="otc-container" Oct 30 04:20:05.675468 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:05.675428 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed10b143-7033-4b0f-8eaf-ca2050be68ac" containerName="otc-container" Oct 30 04:20:05.675468 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:05.675451 5882 cpu_manager.go:414] "containerMap: removing container" podUID="18bfa56e-ecfc-40a1-9bcc-0cee9754b826" containerName="ta-container" Oct 30 04:20:05.675468 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:05.675456 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18bfa56e-ecfc-40a1-9bcc-0cee9754b826" containerName="ta-container" Oct 30 04:20:05.675601 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:05.675514 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed10b143-7033-4b0f-8eaf-ca2050be68ac" containerName="otc-container" Oct 30 04:20:05.675601 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:05.675526 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="18bfa56e-ecfc-40a1-9bcc-0cee9754b826" containerName="ta-container" Oct 30 04:20:05.679878 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:05.679858 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" Oct 30 04:20:05.710756 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:05.710731 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w"] Oct 30 04:20:05.793940 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:05.793902 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/4428c0a1-c791-490c-a0bb-b2ea894c3d18-ta-internal\") pod \"data-science-collector-targetallocator-5bf47c8d75-72f2w\" (UID: \"4428c0a1-c791-490c-a0bb-b2ea894c3d18\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" Oct 30 04:20:05.794089 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:05.793965 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7l7x\" (UniqueName: \"kubernetes.io/projected/4428c0a1-c791-490c-a0bb-b2ea894c3d18-kube-api-access-z7l7x\") pod \"data-science-collector-targetallocator-5bf47c8d75-72f2w\" (UID: \"4428c0a1-c791-490c-a0bb-b2ea894c3d18\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" Oct 30 04:20:05.794089 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:05.794066 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/4428c0a1-c791-490c-a0bb-b2ea894c3d18-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5bf47c8d75-72f2w\" (UID: \"4428c0a1-c791-490c-a0bb-b2ea894c3d18\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" Oct 30 04:20:05.895436 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:05.895398 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/4428c0a1-c791-490c-a0bb-b2ea894c3d18-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5bf47c8d75-72f2w\" (UID: \"4428c0a1-c791-490c-a0bb-b2ea894c3d18\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" Oct 30 04:20:05.895599 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:05.895490 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/4428c0a1-c791-490c-a0bb-b2ea894c3d18-ta-internal\") pod \"data-science-collector-targetallocator-5bf47c8d75-72f2w\" (UID: \"4428c0a1-c791-490c-a0bb-b2ea894c3d18\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" Oct 30 04:20:05.895599 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:05.895548 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7l7x\" (UniqueName: \"kubernetes.io/projected/4428c0a1-c791-490c-a0bb-b2ea894c3d18-kube-api-access-z7l7x\") pod \"data-science-collector-targetallocator-5bf47c8d75-72f2w\" (UID: \"4428c0a1-c791-490c-a0bb-b2ea894c3d18\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" Oct 30 04:20:05.896182 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:05.896156 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/4428c0a1-c791-490c-a0bb-b2ea894c3d18-ta-internal\") pod \"data-science-collector-targetallocator-5bf47c8d75-72f2w\" (UID: \"4428c0a1-c791-490c-a0bb-b2ea894c3d18\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" Oct 30 04:20:05.897833 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:05.897810 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/4428c0a1-c791-490c-a0bb-b2ea894c3d18-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5bf47c8d75-72f2w\" (UID: \"4428c0a1-c791-490c-a0bb-b2ea894c3d18\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" Oct 30 04:20:05.921032 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:05.921005 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7l7x\" (UniqueName: \"kubernetes.io/projected/4428c0a1-c791-490c-a0bb-b2ea894c3d18-kube-api-access-z7l7x\") pod \"data-science-collector-targetallocator-5bf47c8d75-72f2w\" (UID: \"4428c0a1-c791-490c-a0bb-b2ea894c3d18\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" Oct 30 04:20:05.991785 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:05.991697 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" Oct 30 04:20:06.285189 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:06.285158 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w"] Oct 30 04:20:06.287582 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:20:06.287541 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4428c0a1_c791_490c_a0bb_b2ea894c3d18.slice/crio-9aff1c2e22ede4819406f7eda428c83bbf35269bad93b07ae349394291b51756 WatchSource:0}: Error finding container 9aff1c2e22ede4819406f7eda428c83bbf35269bad93b07ae349394291b51756: Status 404 returned error can't find the container with id 9aff1c2e22ede4819406f7eda428c83bbf35269bad93b07ae349394291b51756 Oct 30 04:20:06.289884 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:06.289852 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:20:06.289973 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:06.289908 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:20:07.036743 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:07.036705 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" event={"ID":"4428c0a1-c791-490c-a0bb-b2ea894c3d18","Type":"ContainerStarted","Data":"f60e12db9ed8ce560b469406d18addf7968df17b5bf3e305f14f05219372cb2a"} Oct 30 04:20:07.037187 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:07.036752 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" event={"ID":"4428c0a1-c791-490c-a0bb-b2ea894c3d18","Type":"ContainerStarted","Data":"9aff1c2e22ede4819406f7eda428c83bbf35269bad93b07ae349394291b51756"} Oct 30 04:20:07.037187 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:07.036883 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" Oct 30 04:20:07.038602 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:07.038581 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" Oct 30 04:20:07.082785 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:07.082734 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" podStartSLOduration=2.08272043 podStartE2EDuration="2.08272043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:07.08191377 +0000 UTC m=+5919.196228747" watchObservedRunningTime="2026-10-30 04:20:07.08272043 +0000 UTC m=+5919.197035404" Oct 30 04:20:07.159521 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:07.159491 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg"] Oct 30 04:20:07.159927 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:07.159898 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" podUID="007a10cc-ee5a-45d0-ab05-201155877f10" containerName="ta-container" containerID="cri-o://f8bbf79b25f358c125412f2065016cf091d01ad926ec30a5502749c128133e84" gracePeriod=30 Oct 30 04:20:07.396491 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:07.396466 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" Oct 30 04:20:07.508242 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:07.508210 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/007a10cc-ee5a-45d0-ab05-201155877f10-ta-internal\" (UniqueName: \"kubernetes.io/configmap/007a10cc-ee5a-45d0-ab05-201155877f10-ta-internal\") pod \"007a10cc-ee5a-45d0-ab05-201155877f10\" (UID: \"007a10cc-ee5a-45d0-ab05-201155877f10\") " Oct 30 04:20:07.508242 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:07.508248 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007a10cc-ee5a-45d0-ab05-201155877f10-kube-api-access-4gdpg\" (UniqueName: \"kubernetes.io/projected/007a10cc-ee5a-45d0-ab05-201155877f10-kube-api-access-4gdpg\") pod \"007a10cc-ee5a-45d0-ab05-201155877f10\" (UID: \"007a10cc-ee5a-45d0-ab05-201155877f10\") " Oct 30 04:20:07.508453 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:07.508270 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/007a10cc-ee5a-45d0-ab05-201155877f10-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/007a10cc-ee5a-45d0-ab05-201155877f10-data-science-collector-ta-server-cert\") pod \"007a10cc-ee5a-45d0-ab05-201155877f10\" (UID: \"007a10cc-ee5a-45d0-ab05-201155877f10\") " Oct 30 04:20:07.508749 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:07.508726 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007a10cc-ee5a-45d0-ab05-201155877f10-ta-internal" pod "007a10cc-ee5a-45d0-ab05-201155877f10" (UID: "007a10cc-ee5a-45d0-ab05-201155877f10"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:07.510313 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:07.510289 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007a10cc-ee5a-45d0-ab05-201155877f10-kube-api-access-4gdpg" pod "007a10cc-ee5a-45d0-ab05-201155877f10" (UID: "007a10cc-ee5a-45d0-ab05-201155877f10"). InnerVolumeSpecName "kube-api-access-4gdpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:07.510413 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:07.510350 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/007a10cc-ee5a-45d0-ab05-201155877f10-data-science-collector-ta-server-cert" pod "007a10cc-ee5a-45d0-ab05-201155877f10" (UID: "007a10cc-ee5a-45d0-ab05-201155877f10"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:07.609466 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:07.609398 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/007a10cc-ee5a-45d0-ab05-201155877f10-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:07.609466 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:07.609425 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gdpg\" (UniqueName: \"kubernetes.io/projected/007a10cc-ee5a-45d0-ab05-201155877f10-kube-api-access-4gdpg\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:07.609466 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:07.609437 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/007a10cc-ee5a-45d0-ab05-201155877f10-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:08.041960 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:08.041931 5882 generic.go:356] "Generic (PLEG): container finished" podID="007a10cc-ee5a-45d0-ab05-201155877f10" containerID="f8bbf79b25f358c125412f2065016cf091d01ad926ec30a5502749c128133e84" exitCode=0 Oct 30 04:20:08.042341 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:08.041967 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" Oct 30 04:20:08.042341 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:08.042009 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" event={"ID":"007a10cc-ee5a-45d0-ab05-201155877f10","Type":"ContainerDied","Data":"f8bbf79b25f358c125412f2065016cf091d01ad926ec30a5502749c128133e84"} Oct 30 04:20:08.042341 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:08.042049 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg" event={"ID":"007a10cc-ee5a-45d0-ab05-201155877f10","Type":"ContainerDied","Data":"7325ecd4054799a8227e947450228b002d658c7f6fb128c7cc336baab9bf7154"} Oct 30 04:20:08.042341 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:08.042067 5882 scope.go:122] "RemoveContainer" containerID="f8bbf79b25f358c125412f2065016cf091d01ad926ec30a5502749c128133e84" Oct 30 04:20:08.050086 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:08.050069 5882 scope.go:122] "RemoveContainer" containerID="f8bbf79b25f358c125412f2065016cf091d01ad926ec30a5502749c128133e84" Oct 30 04:20:08.050344 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:20:08.050324 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8bbf79b25f358c125412f2065016cf091d01ad926ec30a5502749c128133e84\": container with ID starting with f8bbf79b25f358c125412f2065016cf091d01ad926ec30a5502749c128133e84 not found: ID does not exist" containerID="f8bbf79b25f358c125412f2065016cf091d01ad926ec30a5502749c128133e84" Oct 30 04:20:08.050404 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:08.050351 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8bbf79b25f358c125412f2065016cf091d01ad926ec30a5502749c128133e84"} err="failed to get container status \"f8bbf79b25f358c125412f2065016cf091d01ad926ec30a5502749c128133e84\": rpc error: code = NotFound desc = could not find container \"f8bbf79b25f358c125412f2065016cf091d01ad926ec30a5502749c128133e84\": container with ID starting with f8bbf79b25f358c125412f2065016cf091d01ad926ec30a5502749c128133e84 not found: ID does not exist" Oct 30 04:20:08.098494 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:08.098464 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg"] Oct 30 04:20:08.112318 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:08.112286 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5dd6975f86-vxrsg"] Oct 30 04:20:10.010038 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.010002 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="007a10cc-ee5a-45d0-ab05-201155877f10" path="/var/lib/kubelet/pods/007a10cc-ee5a-45d0-ab05-201155877f10/volumes" Oct 30 04:20:10.495402 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.495350 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:20:10.495647 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.495606 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" containerName="alertmanager" containerID="cri-o://4189e054fe3142f17b49127b1ca2bc9d837255ee2d63d10cb3b2886e98ee1da5" gracePeriod=120 Oct 30 04:20:10.495756 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.495674 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" containerName="config-reloader" containerID="cri-o://0af55161445b7af6fa2176020e7b9d92f21a9429146e35a13a7cb1e4135d9a99" gracePeriod=120 Oct 30 04:20:10.512798 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.512476 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:20:10.513151 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.512801 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="5979b0e7-e3e6-436d-a626-d47d1f860a28" containerName="prometheus" containerID="cri-o://99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e" gracePeriod=600 Oct 30 04:20:10.513151 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.512827 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="5979b0e7-e3e6-436d-a626-d47d1f860a28" containerName="thanos-sidecar" containerID="cri-o://e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869" gracePeriod=600 Oct 30 04:20:10.513151 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.512967 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="5979b0e7-e3e6-436d-a626-d47d1f860a28" containerName="config-reloader" containerID="cri-o://ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747" gracePeriod=600 Oct 30 04:20:10.760809 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.760785 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:10.837171 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.837140 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-secret-prometheus-web-tls-ca\") pod \"5979b0e7-e3e6-436d-a626-d47d1f860a28\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " Oct 30 04:20:10.837372 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.837181 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"5979b0e7-e3e6-436d-a626-d47d1f860a28\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " Oct 30 04:20:10.837372 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.837289 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"5979b0e7-e3e6-436d-a626-d47d1f860a28\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " Oct 30 04:20:10.837372 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.837326 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"5979b0e7-e3e6-436d-a626-d47d1f860a28\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " Oct 30 04:20:10.837372 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.837343 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config\") pod \"5979b0e7-e3e6-436d-a626-d47d1f860a28\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " Oct 30 04:20:10.837635 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.837398 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5979b0e7-e3e6-436d-a626-d47d1f860a28-config-out\" (UniqueName: \"kubernetes.io/empty-dir/5979b0e7-e3e6-436d-a626-d47d1f860a28-config-out\") pod \"5979b0e7-e3e6-436d-a626-d47d1f860a28\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " Oct 30 04:20:10.837635 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.837425 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-config\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-config\") pod \"5979b0e7-e3e6-436d-a626-d47d1f860a28\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " Oct 30 04:20:10.837635 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.837475 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5979b0e7-e3e6-436d-a626-d47d1f860a28-kube-api-access-z5fn8\" (UniqueName: \"kubernetes.io/projected/5979b0e7-e3e6-436d-a626-d47d1f860a28-kube-api-access-z5fn8\") pod \"5979b0e7-e3e6-436d-a626-d47d1f860a28\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " Oct 30 04:20:10.837635 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.837508 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"5979b0e7-e3e6-436d-a626-d47d1f860a28\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " Oct 30 04:20:10.837635 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.837571 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"5979b0e7-e3e6-436d-a626-d47d1f860a28\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " Oct 30 04:20:10.837635 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.837627 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5979b0e7-e3e6-436d-a626-d47d1f860a28-tls-assets\" (UniqueName: \"kubernetes.io/projected/5979b0e7-e3e6-436d-a626-d47d1f860a28-tls-assets\") pod \"5979b0e7-e3e6-436d-a626-d47d1f860a28\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " Oct 30 04:20:10.837977 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.837666 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-thanos-prometheus-http-client-file\") pod \"5979b0e7-e3e6-436d-a626-d47d1f860a28\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " Oct 30 04:20:10.837977 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.837684 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-0" pod "5979b0e7-e3e6-436d-a626-d47d1f860a28" (UID: "5979b0e7-e3e6-436d-a626-d47d1f860a28"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:10.837977 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.837710 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"5979b0e7-e3e6-436d-a626-d47d1f860a28\" (UID: \"5979b0e7-e3e6-436d-a626-d47d1f860a28\") " Oct 30 04:20:10.838146 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.838050 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.838409 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.838384 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-2" pod "5979b0e7-e3e6-436d-a626-d47d1f860a28" (UID: "5979b0e7-e3e6-436d-a626-d47d1f860a28"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:10.838973 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.838934 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-1" pod "5979b0e7-e3e6-436d-a626-d47d1f860a28" (UID: "5979b0e7-e3e6-436d-a626-d47d1f860a28"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:10.840700 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.840673 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5979b0e7-e3e6-436d-a626-d47d1f860a28-kube-api-access-z5fn8" pod "5979b0e7-e3e6-436d-a626-d47d1f860a28" (UID: "5979b0e7-e3e6-436d-a626-d47d1f860a28"). InnerVolumeSpecName "kube-api-access-z5fn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:10.841021 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.840996 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5979b0e7-e3e6-436d-a626-d47d1f860a28-config-out" pod "5979b0e7-e3e6-436d-a626-d47d1f860a28" (UID: "5979b0e7-e3e6-436d-a626-d47d1f860a28"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:20:10.842148 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.842038 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-secret-prometheus-web-tls-ca" pod "5979b0e7-e3e6-436d-a626-d47d1f860a28" (UID: "5979b0e7-e3e6-436d-a626-d47d1f860a28"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:10.842148 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.842121 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5979b0e7-e3e6-436d-a626-d47d1f860a28-tls-assets" pod "5979b0e7-e3e6-436d-a626-d47d1f860a28" (UID: "5979b0e7-e3e6-436d-a626-d47d1f860a28"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:10.842503 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.842444 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-config" pod "5979b0e7-e3e6-436d-a626-d47d1f860a28" (UID: "5979b0e7-e3e6-436d-a626-d47d1f860a28"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:10.842664 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.842509 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "5979b0e7-e3e6-436d-a626-d47d1f860a28" (UID: "5979b0e7-e3e6-436d-a626-d47d1f860a28"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:10.842664 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.842557 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "5979b0e7-e3e6-436d-a626-d47d1f860a28" (UID: "5979b0e7-e3e6-436d-a626-d47d1f860a28"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:10.843449 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.843425 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-thanos-prometheus-http-client-file" pod "5979b0e7-e3e6-436d-a626-d47d1f860a28" (UID: "5979b0e7-e3e6-436d-a626-d47d1f860a28"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:10.845388 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.845364 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4" pod "5979b0e7-e3e6-436d-a626-d47d1f860a28" (UID: "5979b0e7-e3e6-436d-a626-d47d1f860a28"). InnerVolumeSpecName "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:20:10.854858 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.854826 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config" pod "5979b0e7-e3e6-436d-a626-d47d1f860a28" (UID: "5979b0e7-e3e6-436d-a626-d47d1f860a28"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:10.939456 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.939420 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.939456 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.939450 5882 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-secret-prometheus-web-tls-ca\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.939681 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.939479 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:20:10.939681 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.939490 5882 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.939681 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.939500 5882 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.939681 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.939509 5882 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5979b0e7-e3e6-436d-a626-d47d1f860a28-config-out\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.939681 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.939517 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.939681 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.939525 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5fn8\" (UniqueName: \"kubernetes.io/projected/5979b0e7-e3e6-436d-a626-d47d1f860a28-kube-api-access-z5fn8\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.939681 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.939535 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5979b0e7-e3e6-436d-a626-d47d1f860a28-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.939681 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.939544 5882 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.939681 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.939553 5882 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5979b0e7-e3e6-436d-a626-d47d1f860a28-tls-assets\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.939681 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.939562 5882 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5979b0e7-e3e6-436d-a626-d47d1f860a28-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.950405 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:10.950387 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:20:11.040773 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.040743 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-927d18ef2eb99137c68fbe5e0bb8ea629bb5219ed56cdede72da7c0f045be95e\"" Oct 30 04:20:11.059221 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.059187 5882 generic.go:356] "Generic (PLEG): container finished" podID="5979b0e7-e3e6-436d-a626-d47d1f860a28" containerID="e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869" exitCode=0 Oct 30 04:20:11.059221 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.059216 5882 generic.go:356] "Generic (PLEG): container finished" podID="5979b0e7-e3e6-436d-a626-d47d1f860a28" containerID="ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747" exitCode=0 Oct 30 04:20:11.059447 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.059241 5882 generic.go:356] "Generic (PLEG): container finished" podID="5979b0e7-e3e6-436d-a626-d47d1f860a28" containerID="99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e" exitCode=0 Oct 30 04:20:11.059447 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.059256 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:11.059447 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.059261 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"5979b0e7-e3e6-436d-a626-d47d1f860a28","Type":"ContainerDied","Data":"e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869"} Oct 30 04:20:11.059447 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.059330 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"5979b0e7-e3e6-436d-a626-d47d1f860a28","Type":"ContainerDied","Data":"ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747"} Oct 30 04:20:11.059447 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.059359 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"5979b0e7-e3e6-436d-a626-d47d1f860a28","Type":"ContainerDied","Data":"99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e"} Oct 30 04:20:11.059447 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.059387 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"5979b0e7-e3e6-436d-a626-d47d1f860a28","Type":"ContainerDied","Data":"bc6cc21edcfbe82fa21dddecedd391d50dd19655e3d471a885fd13036193ee25"} Oct 30 04:20:11.059447 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.059401 5882 scope.go:122] "RemoveContainer" containerID="e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869" Oct 30 04:20:11.062105 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.062079 5882 generic.go:356] "Generic (PLEG): container finished" podID="3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" containerID="0af55161445b7af6fa2176020e7b9d92f21a9429146e35a13a7cb1e4135d9a99" exitCode=0 Oct 30 04:20:11.062198 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.062105 5882 generic.go:356] "Generic (PLEG): container finished" podID="3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" containerID="4189e054fe3142f17b49127b1ca2bc9d837255ee2d63d10cb3b2886e98ee1da5" exitCode=0 Oct 30 04:20:11.062198 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.062160 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5","Type":"ContainerDied","Data":"0af55161445b7af6fa2176020e7b9d92f21a9429146e35a13a7cb1e4135d9a99"} Oct 30 04:20:11.062325 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.062212 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5","Type":"ContainerDied","Data":"4189e054fe3142f17b49127b1ca2bc9d837255ee2d63d10cb3b2886e98ee1da5"} Oct 30 04:20:11.072817 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.072797 5882 scope.go:122] "RemoveContainer" containerID="ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747" Oct 30 04:20:11.081794 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.081774 5882 scope.go:122] "RemoveContainer" containerID="99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e" Oct 30 04:20:11.103627 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.094008 5882 scope.go:122] "RemoveContainer" containerID="1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca" Oct 30 04:20:11.111355 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.111252 5882 scope.go:122] "RemoveContainer" containerID="e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869" Oct 30 04:20:11.111629 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:20:11.111590 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869\": container with ID starting with e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869 not found: ID does not exist" containerID="e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869" Oct 30 04:20:11.111718 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.111644 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869"} err="failed to get container status \"e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869\": rpc error: code = NotFound desc = could not find container \"e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869\": container with ID starting with e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869 not found: ID does not exist" Oct 30 04:20:11.111718 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.111673 5882 scope.go:122] "RemoveContainer" containerID="ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747" Oct 30 04:20:11.112009 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:20:11.111983 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747\": container with ID starting with ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747 not found: ID does not exist" containerID="ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747" Oct 30 04:20:11.112110 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.112020 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747"} err="failed to get container status \"ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747\": rpc error: code = NotFound desc = could not find container \"ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747\": container with ID starting with ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747 not found: ID does not exist" Oct 30 04:20:11.112110 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.112043 5882 scope.go:122] "RemoveContainer" containerID="99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e" Oct 30 04:20:11.112381 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:20:11.112362 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e\": container with ID starting with 99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e not found: ID does not exist" containerID="99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e" Oct 30 04:20:11.112455 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.112389 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e"} err="failed to get container status \"99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e\": rpc error: code = NotFound desc = could not find container \"99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e\": container with ID starting with 99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e not found: ID does not exist" Oct 30 04:20:11.112455 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.112410 5882 scope.go:122] "RemoveContainer" containerID="1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca" Oct 30 04:20:11.112771 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:20:11.112751 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca\": container with ID starting with 1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca not found: ID does not exist" containerID="1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca" Oct 30 04:20:11.112846 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.112777 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca"} err="failed to get container status \"1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca\": rpc error: code = NotFound desc = could not find container \"1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca\": container with ID starting with 1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca not found: ID does not exist" Oct 30 04:20:11.112846 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.112798 5882 scope.go:122] "RemoveContainer" containerID="e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869" Oct 30 04:20:11.113076 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.113047 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869"} err="failed to get container status \"e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869\": rpc error: code = NotFound desc = could not find container \"e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869\": container with ID starting with e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869 not found: ID does not exist" Oct 30 04:20:11.113076 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.113068 5882 scope.go:122] "RemoveContainer" containerID="ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747" Oct 30 04:20:11.113407 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.113355 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747"} err="failed to get container status \"ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747\": rpc error: code = NotFound desc = could not find container \"ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747\": container with ID starting with ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747 not found: ID does not exist" Oct 30 04:20:11.113407 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.113383 5882 scope.go:122] "RemoveContainer" containerID="99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e" Oct 30 04:20:11.113678 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.113651 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e"} err="failed to get container status \"99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e\": rpc error: code = NotFound desc = could not find container \"99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e\": container with ID starting with 99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e not found: ID does not exist" Oct 30 04:20:11.113770 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.113680 5882 scope.go:122] "RemoveContainer" containerID="1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca" Oct 30 04:20:11.114039 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.114019 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca"} err="failed to get container status \"1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca\": rpc error: code = NotFound desc = could not find container \"1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca\": container with ID starting with 1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca not found: ID does not exist" Oct 30 04:20:11.114039 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.114038 5882 scope.go:122] "RemoveContainer" containerID="e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869" Oct 30 04:20:11.114368 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.114343 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869"} err="failed to get container status \"e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869\": rpc error: code = NotFound desc = could not find container \"e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869\": container with ID starting with e267a38060f67d05e0137f08b6f5c8e871c9cf3421c4b6157a81351019d94869 not found: ID does not exist" Oct 30 04:20:11.114368 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.114364 5882 scope.go:122] "RemoveContainer" containerID="ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747" Oct 30 04:20:11.114690 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.114669 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747"} err="failed to get container status \"ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747\": rpc error: code = NotFound desc = could not find container \"ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747\": container with ID starting with ba6e8f72ec99f8cf00151c74fbe94f768b218401854234cb3919471067054747 not found: ID does not exist" Oct 30 04:20:11.114799 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.114692 5882 scope.go:122] "RemoveContainer" containerID="99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e" Oct 30 04:20:11.114998 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.114969 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e"} err="failed to get container status \"99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e\": rpc error: code = NotFound desc = could not find container \"99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e\": container with ID starting with 99b8b42cb4436c826ee653c1e65260e01c839738d85fb8df329814465aa7e35e not found: ID does not exist" Oct 30 04:20:11.114998 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.114998 5882 scope.go:122] "RemoveContainer" containerID="1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca" Oct 30 04:20:11.115281 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.115249 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca"} err="failed to get container status \"1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca\": rpc error: code = NotFound desc = could not find container \"1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca\": container with ID starting with 1b0780cae0efc4cf960cb9621b91c09a57267d45b186b6088e1b1eacd3c5b0ca not found: ID does not exist" Oct 30 04:20:11.132004 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.131977 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:20:11.141138 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.141112 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:11.145473 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.145450 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:20:11.221480 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.221449 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w"] Oct 30 04:20:11.221689 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.221607 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" podUID="4428c0a1-c791-490c-a0bb-b2ea894c3d18" containerName="ta-container" containerID="cri-o://f60e12db9ed8ce560b469406d18addf7968df17b5bf3e305f14f05219372cb2a" gracePeriod=30 Oct 30 04:20:11.242203 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.242175 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-config-volume\" (UniqueName: \"kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-config-volume\") pod \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " Oct 30 04:20:11.242305 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.242237 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-web-config\" (UniqueName: \"kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-web-config\") pod \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " Oct 30 04:20:11.242305 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.242285 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-kube-api-access-btxr6\" (UniqueName: \"kubernetes.io/projected/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-kube-api-access-btxr6\") pod \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " Oct 30 04:20:11.242379 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.242325 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-alertmanager-data-science-monitoringstack-db\") pod \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " Oct 30 04:20:11.242379 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.242368 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-config-out\") pod \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " Oct 30 04:20:11.242461 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.242389 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-cluster-tls-config\") pod \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " Oct 30 04:20:11.242461 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.242410 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-tls-assets\" (UniqueName: \"kubernetes.io/projected/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-tls-assets\") pod \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\" (UID: \"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5\") " Oct 30 04:20:11.242756 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.242725 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-alertmanager-data-science-monitoringstack-db" pod "3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" (UID: "3b5c4add-ffce-4b84-8b51-b0caecc1f3b5"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:20:11.244457 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.244426 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-config-out" pod "3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" (UID: "3b5c4add-ffce-4b84-8b51-b0caecc1f3b5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:20:11.244663 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.244641 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-config-volume" pod "3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" (UID: "3b5c4add-ffce-4b84-8b51-b0caecc1f3b5"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:11.244807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.244784 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-kube-api-access-btxr6" pod "3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" (UID: "3b5c4add-ffce-4b84-8b51-b0caecc1f3b5"). InnerVolumeSpecName "kube-api-access-btxr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:11.244892 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.244851 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-tls-assets" pod "3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" (UID: "3b5c4add-ffce-4b84-8b51-b0caecc1f3b5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:11.251610 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.251585 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-cluster-tls-config" pod "3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" (UID: "3b5c4add-ffce-4b84-8b51-b0caecc1f3b5"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:11.258466 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.258403 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:20:11.258679 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.258630 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="4a13a26d-0e0b-4817-adda-af696009974d" containerName="perses" containerID="cri-o://a0fffb1c1a5535a59fc82945cf7117b91f9514958841ddb2c7fe8ef08904a246" gracePeriod=30 Oct 30 04:20:11.263265 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.263238 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-web-config" pod "3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" (UID: "3b5c4add-ffce-4b84-8b51-b0caecc1f3b5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:11.343444 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.343408 5882 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:11.343444 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.343436 5882 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-config-out\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:11.343444 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.343446 5882 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-cluster-tls-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:11.343444 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.343455 5882 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-tls-assets\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:11.343794 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.343463 5882 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-config-volume\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:11.343794 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.343470 5882 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-web-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:11.343794 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.343479 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btxr6\" (UniqueName: \"kubernetes.io/projected/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5-kube-api-access-btxr6\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:11.505700 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.505568 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" Oct 30 04:20:11.540157 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.540132 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:11.645680 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.645647 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a13a26d-0e0b-4817-adda-af696009974d-kube-api-access-w6w9s\" (UniqueName: \"kubernetes.io/projected/4a13a26d-0e0b-4817-adda-af696009974d-kube-api-access-w6w9s\") pod \"4a13a26d-0e0b-4817-adda-af696009974d\" (UID: \"4a13a26d-0e0b-4817-adda-af696009974d\") " Oct 30 04:20:11.645826 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.645696 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a13a26d-0e0b-4817-adda-af696009974d-plugins\" (UniqueName: \"kubernetes.io/empty-dir/4a13a26d-0e0b-4817-adda-af696009974d-plugins\") pod \"4a13a26d-0e0b-4817-adda-af696009974d\" (UID: \"4a13a26d-0e0b-4817-adda-af696009974d\") " Oct 30 04:20:11.645826 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.645726 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4428c0a1-c791-490c-a0bb-b2ea894c3d18-ta-internal\" (UniqueName: \"kubernetes.io/configmap/4428c0a1-c791-490c-a0bb-b2ea894c3d18-ta-internal\") pod \"4428c0a1-c791-490c-a0bb-b2ea894c3d18\" (UID: \"4428c0a1-c791-490c-a0bb-b2ea894c3d18\") " Oct 30 04:20:11.645826 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.645749 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4428c0a1-c791-490c-a0bb-b2ea894c3d18-kube-api-access-z7l7x\" (UniqueName: \"kubernetes.io/projected/4428c0a1-c791-490c-a0bb-b2ea894c3d18-kube-api-access-z7l7x\") pod \"4428c0a1-c791-490c-a0bb-b2ea894c3d18\" (UID: \"4428c0a1-c791-490c-a0bb-b2ea894c3d18\") " Oct 30 04:20:11.645826 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.645776 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4428c0a1-c791-490c-a0bb-b2ea894c3d18-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/4428c0a1-c791-490c-a0bb-b2ea894c3d18-data-science-collector-ta-server-cert\") pod \"4428c0a1-c791-490c-a0bb-b2ea894c3d18\" (UID: \"4428c0a1-c791-490c-a0bb-b2ea894c3d18\") " Oct 30 04:20:11.646061 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.645887 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"4a13a26d-0e0b-4817-adda-af696009974d\" (UID: \"4a13a26d-0e0b-4817-adda-af696009974d\") " Oct 30 04:20:11.646061 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.645926 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a13a26d-0e0b-4817-adda-af696009974d-config\" (UniqueName: \"kubernetes.io/configmap/4a13a26d-0e0b-4817-adda-af696009974d-config\") pod \"4a13a26d-0e0b-4817-adda-af696009974d\" (UID: \"4a13a26d-0e0b-4817-adda-af696009974d\") " Oct 30 04:20:11.646181 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.646094 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4428c0a1-c791-490c-a0bb-b2ea894c3d18-ta-internal" pod "4428c0a1-c791-490c-a0bb-b2ea894c3d18" (UID: "4428c0a1-c791-490c-a0bb-b2ea894c3d18"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:11.646328 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.646244 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/4428c0a1-c791-490c-a0bb-b2ea894c3d18-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:11.646426 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.646407 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a13a26d-0e0b-4817-adda-af696009974d-config" pod "4a13a26d-0e0b-4817-adda-af696009974d" (UID: "4a13a26d-0e0b-4817-adda-af696009974d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:11.648174 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.648148 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4428c0a1-c791-490c-a0bb-b2ea894c3d18-kube-api-access-z7l7x" pod "4428c0a1-c791-490c-a0bb-b2ea894c3d18" (UID: "4428c0a1-c791-490c-a0bb-b2ea894c3d18"). InnerVolumeSpecName "kube-api-access-z7l7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:11.648357 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.648331 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4428c0a1-c791-490c-a0bb-b2ea894c3d18-data-science-collector-ta-server-cert" pod "4428c0a1-c791-490c-a0bb-b2ea894c3d18" (UID: "4428c0a1-c791-490c-a0bb-b2ea894c3d18"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:11.648431 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.648403 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a13a26d-0e0b-4817-adda-af696009974d-kube-api-access-w6w9s" pod "4a13a26d-0e0b-4817-adda-af696009974d" (UID: "4a13a26d-0e0b-4817-adda-af696009974d"). InnerVolumeSpecName "kube-api-access-w6w9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:11.648820 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.648797 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e" pod "4a13a26d-0e0b-4817-adda-af696009974d" (UID: "4a13a26d-0e0b-4817-adda-af696009974d"). InnerVolumeSpecName "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:20:11.747106 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.747074 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:20:11.747106 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.747103 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a13a26d-0e0b-4817-adda-af696009974d-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:11.747106 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.747113 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6w9s\" (UniqueName: \"kubernetes.io/projected/4a13a26d-0e0b-4817-adda-af696009974d-kube-api-access-w6w9s\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:11.747393 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.747123 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7l7x\" (UniqueName: \"kubernetes.io/projected/4428c0a1-c791-490c-a0bb-b2ea894c3d18-kube-api-access-z7l7x\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:11.747393 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.747132 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/4428c0a1-c791-490c-a0bb-b2ea894c3d18-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:11.760650 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.760579 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:20:11.793674 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.793644 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a13a26d-0e0b-4817-adda-af696009974d-plugins" pod "4a13a26d-0e0b-4817-adda-af696009974d" (UID: "4a13a26d-0e0b-4817-adda-af696009974d"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:20:11.847893 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.847860 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" Oct 30 04:20:11.847893 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:11.847890 5882 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/4a13a26d-0e0b-4817-adda-af696009974d-plugins\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:12.010750 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.010679 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5979b0e7-e3e6-436d-a626-d47d1f860a28" path="/var/lib/kubelet/pods/5979b0e7-e3e6-436d-a626-d47d1f860a28/volumes" Oct 30 04:20:12.070675 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.070638 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"3b5c4add-ffce-4b84-8b51-b0caecc1f3b5","Type":"ContainerDied","Data":"5f88c5e8b555bbaaae620016820424b54dead750aea53aa1fb0b6349c9bd5618"} Oct 30 04:20:12.071124 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.070720 5882 scope.go:122] "RemoveContainer" containerID="0af55161445b7af6fa2176020e7b9d92f21a9429146e35a13a7cb1e4135d9a99" Oct 30 04:20:12.071124 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.070656 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:12.072242 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.072076 5882 generic.go:356] "Generic (PLEG): container finished" podID="4a13a26d-0e0b-4817-adda-af696009974d" containerID="a0fffb1c1a5535a59fc82945cf7117b91f9514958841ddb2c7fe8ef08904a246" exitCode=0 Oct 30 04:20:12.072242 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.072175 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"4a13a26d-0e0b-4817-adda-af696009974d","Type":"ContainerDied","Data":"a0fffb1c1a5535a59fc82945cf7117b91f9514958841ddb2c7fe8ef08904a246"} Oct 30 04:20:12.072242 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.072183 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:12.072242 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.072203 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"4a13a26d-0e0b-4817-adda-af696009974d","Type":"ContainerDied","Data":"0a856ce0014d4c1ebca18ba4be23f79d764f9f5c3c42e893d167bf3fee4fe65a"} Oct 30 04:20:12.073577 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.073557 5882 generic.go:356] "Generic (PLEG): container finished" podID="4428c0a1-c791-490c-a0bb-b2ea894c3d18" containerID="f60e12db9ed8ce560b469406d18addf7968df17b5bf3e305f14f05219372cb2a" exitCode=0 Oct 30 04:20:12.073689 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.073587 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" event={"ID":"4428c0a1-c791-490c-a0bb-b2ea894c3d18","Type":"ContainerDied","Data":"f60e12db9ed8ce560b469406d18addf7968df17b5bf3e305f14f05219372cb2a"} Oct 30 04:20:12.073689 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.073627 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" event={"ID":"4428c0a1-c791-490c-a0bb-b2ea894c3d18","Type":"ContainerDied","Data":"9aff1c2e22ede4819406f7eda428c83bbf35269bad93b07ae349394291b51756"} Oct 30 04:20:12.073689 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.073670 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w" Oct 30 04:20:12.079900 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.079883 5882 scope.go:122] "RemoveContainer" containerID="4189e054fe3142f17b49127b1ca2bc9d837255ee2d63d10cb3b2886e98ee1da5" Oct 30 04:20:12.087303 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.087268 5882 scope.go:122] "RemoveContainer" containerID="6188b21a274ce3e4e63c0d6bd8d814f57dcaea418980886b855fae3d403b5fe9" Oct 30 04:20:12.094483 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.094468 5882 scope.go:122] "RemoveContainer" containerID="a0fffb1c1a5535a59fc82945cf7117b91f9514958841ddb2c7fe8ef08904a246" Oct 30 04:20:12.101435 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.101404 5882 scope.go:122] "RemoveContainer" containerID="a0fffb1c1a5535a59fc82945cf7117b91f9514958841ddb2c7fe8ef08904a246" Oct 30 04:20:12.101709 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:20:12.101675 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0fffb1c1a5535a59fc82945cf7117b91f9514958841ddb2c7fe8ef08904a246\": container with ID starting with a0fffb1c1a5535a59fc82945cf7117b91f9514958841ddb2c7fe8ef08904a246 not found: ID does not exist" containerID="a0fffb1c1a5535a59fc82945cf7117b91f9514958841ddb2c7fe8ef08904a246" Oct 30 04:20:12.101776 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.101722 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0fffb1c1a5535a59fc82945cf7117b91f9514958841ddb2c7fe8ef08904a246"} err="failed to get container status \"a0fffb1c1a5535a59fc82945cf7117b91f9514958841ddb2c7fe8ef08904a246\": rpc error: code = NotFound desc = could not find container \"a0fffb1c1a5535a59fc82945cf7117b91f9514958841ddb2c7fe8ef08904a246\": container with ID starting with a0fffb1c1a5535a59fc82945cf7117b91f9514958841ddb2c7fe8ef08904a246 not found: ID does not exist" Oct 30 04:20:12.101776 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.101740 5882 scope.go:122] "RemoveContainer" containerID="f60e12db9ed8ce560b469406d18addf7968df17b5bf3e305f14f05219372cb2a" Oct 30 04:20:12.108756 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.108731 5882 scope.go:122] "RemoveContainer" containerID="f60e12db9ed8ce560b469406d18addf7968df17b5bf3e305f14f05219372cb2a" Oct 30 04:20:12.109052 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:20:12.108984 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f60e12db9ed8ce560b469406d18addf7968df17b5bf3e305f14f05219372cb2a\": container with ID starting with f60e12db9ed8ce560b469406d18addf7968df17b5bf3e305f14f05219372cb2a not found: ID does not exist" containerID="f60e12db9ed8ce560b469406d18addf7968df17b5bf3e305f14f05219372cb2a" Oct 30 04:20:12.109126 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.109062 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f60e12db9ed8ce560b469406d18addf7968df17b5bf3e305f14f05219372cb2a"} err="failed to get container status \"f60e12db9ed8ce560b469406d18addf7968df17b5bf3e305f14f05219372cb2a\": rpc error: code = NotFound desc = could not find container \"f60e12db9ed8ce560b469406d18addf7968df17b5bf3e305f14f05219372cb2a\": container with ID starting with f60e12db9ed8ce560b469406d18addf7968df17b5bf3e305f14f05219372cb2a not found: ID does not exist" Oct 30 04:20:12.127495 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.127473 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:20:12.131790 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.131768 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:20:12.169305 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.169279 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:20:12.175568 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.175540 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:20:12.212667 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.212608 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w"] Oct 30 04:20:12.218311 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:12.218282 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5bf47c8d75-72f2w"] Oct 30 04:20:14.008786 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:14.008755 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" path="/var/lib/kubelet/pods/3b5c4add-ffce-4b84-8b51-b0caecc1f3b5/volumes" Oct 30 04:20:14.009231 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:14.009217 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4428c0a1-c791-490c-a0bb-b2ea894c3d18" path="/var/lib/kubelet/pods/4428c0a1-c791-490c-a0bb-b2ea894c3d18/volumes" Oct 30 04:20:14.009589 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:14.009577 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a13a26d-0e0b-4817-adda-af696009974d" path="/var/lib/kubelet/pods/4a13a26d-0e0b-4817-adda-af696009974d/volumes" Oct 30 04:20:17.230115 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230079 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh"] Oct 30 04:20:17.230505 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230416 5882 cpu_manager.go:414] "containerMap: removing container" podUID="3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" containerName="config-reloader" Oct 30 04:20:17.230505 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230426 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" containerName="config-reloader" Oct 30 04:20:17.230505 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230438 5882 cpu_manager.go:414] "containerMap: removing container" podUID="3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" containerName="alertmanager" Oct 30 04:20:17.230505 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230445 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" containerName="alertmanager" Oct 30 04:20:17.230505 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230461 5882 cpu_manager.go:414] "containerMap: removing container" podUID="3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" containerName="init-config-reloader" Oct 30 04:20:17.230505 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230466 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" containerName="init-config-reloader" Oct 30 04:20:17.230505 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230476 5882 cpu_manager.go:414] "containerMap: removing container" podUID="5979b0e7-e3e6-436d-a626-d47d1f860a28" containerName="config-reloader" Oct 30 04:20:17.230505 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230480 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5979b0e7-e3e6-436d-a626-d47d1f860a28" containerName="config-reloader" Oct 30 04:20:17.230505 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230486 5882 cpu_manager.go:414] "containerMap: removing container" podUID="007a10cc-ee5a-45d0-ab05-201155877f10" containerName="ta-container" Oct 30 04:20:17.230505 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230490 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007a10cc-ee5a-45d0-ab05-201155877f10" containerName="ta-container" Oct 30 04:20:17.230505 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230499 5882 cpu_manager.go:414] "containerMap: removing container" podUID="4428c0a1-c791-490c-a0bb-b2ea894c3d18" containerName="ta-container" Oct 30 04:20:17.230505 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230504 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4428c0a1-c791-490c-a0bb-b2ea894c3d18" containerName="ta-container" Oct 30 04:20:17.231107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230515 5882 cpu_manager.go:414] "containerMap: removing container" podUID="5979b0e7-e3e6-436d-a626-d47d1f860a28" containerName="init-config-reloader" Oct 30 04:20:17.231107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230519 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5979b0e7-e3e6-436d-a626-d47d1f860a28" containerName="init-config-reloader" Oct 30 04:20:17.231107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230529 5882 cpu_manager.go:414] "containerMap: removing container" podUID="5979b0e7-e3e6-436d-a626-d47d1f860a28" containerName="prometheus" Oct 30 04:20:17.231107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230534 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5979b0e7-e3e6-436d-a626-d47d1f860a28" containerName="prometheus" Oct 30 04:20:17.231107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230540 5882 cpu_manager.go:414] "containerMap: removing container" podUID="5979b0e7-e3e6-436d-a626-d47d1f860a28" containerName="thanos-sidecar" Oct 30 04:20:17.231107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230544 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5979b0e7-e3e6-436d-a626-d47d1f860a28" containerName="thanos-sidecar" Oct 30 04:20:17.231107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230551 5882 cpu_manager.go:414] "containerMap: removing container" podUID="4a13a26d-0e0b-4817-adda-af696009974d" containerName="perses" Oct 30 04:20:17.231107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230555 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a13a26d-0e0b-4817-adda-af696009974d" containerName="perses" Oct 30 04:20:17.231107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230607 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" containerName="config-reloader" Oct 30 04:20:17.231107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230628 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b5c4add-ffce-4b84-8b51-b0caecc1f3b5" containerName="alertmanager" Oct 30 04:20:17.231107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230640 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5979b0e7-e3e6-436d-a626-d47d1f860a28" containerName="config-reloader" Oct 30 04:20:17.231107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230649 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="007a10cc-ee5a-45d0-ab05-201155877f10" containerName="ta-container" Oct 30 04:20:17.231107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230657 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4428c0a1-c791-490c-a0bb-b2ea894c3d18" containerName="ta-container" Oct 30 04:20:17.231107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230665 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5979b0e7-e3e6-436d-a626-d47d1f860a28" containerName="prometheus" Oct 30 04:20:17.231107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230671 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5979b0e7-e3e6-436d-a626-d47d1f860a28" containerName="thanos-sidecar" Oct 30 04:20:17.231107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.230677 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a13a26d-0e0b-4817-adda-af696009974d" containerName="perses" Oct 30 04:20:17.235793 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.235774 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" Oct 30 04:20:17.269804 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.269773 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 04:20:17.269948 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.269841 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 04:20:17.269948 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.269893 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-wfww4\"" Oct 30 04:20:17.269948 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.269902 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 04:20:17.269948 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.269910 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 04:20:17.270265 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.270248 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh"] Oct 30 04:20:17.294794 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.294767 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2zzj\" (UniqueName: \"kubernetes.io/projected/612f19e4-0114-4d9d-b23b-48fc92b83dd0-kube-api-access-p2zzj\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-6bhnh\" (UID: \"612f19e4-0114-4d9d-b23b-48fc92b83dd0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" Oct 30 04:20:17.294912 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.294837 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/612f19e4-0114-4d9d-b23b-48fc92b83dd0-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-6bhnh\" (UID: \"612f19e4-0114-4d9d-b23b-48fc92b83dd0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" Oct 30 04:20:17.294981 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.294960 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/612f19e4-0114-4d9d-b23b-48fc92b83dd0-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-6bhnh\" (UID: \"612f19e4-0114-4d9d-b23b-48fc92b83dd0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" Oct 30 04:20:17.396361 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.396328 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/612f19e4-0114-4d9d-b23b-48fc92b83dd0-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-6bhnh\" (UID: \"612f19e4-0114-4d9d-b23b-48fc92b83dd0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" Oct 30 04:20:17.396507 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.396369 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2zzj\" (UniqueName: \"kubernetes.io/projected/612f19e4-0114-4d9d-b23b-48fc92b83dd0-kube-api-access-p2zzj\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-6bhnh\" (UID: \"612f19e4-0114-4d9d-b23b-48fc92b83dd0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" Oct 30 04:20:17.396507 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.396427 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/612f19e4-0114-4d9d-b23b-48fc92b83dd0-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-6bhnh\" (UID: \"612f19e4-0114-4d9d-b23b-48fc92b83dd0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" Oct 30 04:20:17.397171 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.397144 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/612f19e4-0114-4d9d-b23b-48fc92b83dd0-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-6bhnh\" (UID: \"612f19e4-0114-4d9d-b23b-48fc92b83dd0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" Oct 30 04:20:17.398938 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.398917 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/612f19e4-0114-4d9d-b23b-48fc92b83dd0-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-6bhnh\" (UID: \"612f19e4-0114-4d9d-b23b-48fc92b83dd0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" Oct 30 04:20:17.421581 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.421551 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2zzj\" (UniqueName: \"kubernetes.io/projected/612f19e4-0114-4d9d-b23b-48fc92b83dd0-kube-api-access-p2zzj\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-6bhnh\" (UID: \"612f19e4-0114-4d9d-b23b-48fc92b83dd0\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" Oct 30 04:20:17.551221 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.551186 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" Oct 30 04:20:17.844710 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.844681 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh"] Oct 30 04:20:17.846143 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:20:17.846112 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612f19e4_0114_4d9d_b23b_48fc92b83dd0.slice/crio-f8c28063a79aa9356e5646e8a0d2e034f0cfcbd4ad19799a863618bc5b9cf743 WatchSource:0}: Error finding container f8c28063a79aa9356e5646e8a0d2e034f0cfcbd4ad19799a863618bc5b9cf743: Status 404 returned error can't find the container with id f8c28063a79aa9356e5646e8a0d2e034f0cfcbd4ad19799a863618bc5b9cf743 Oct 30 04:20:17.907997 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:17.907973 5882 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:20:18.105339 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.105263 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" event={"ID":"612f19e4-0114-4d9d-b23b-48fc92b83dd0","Type":"ContainerStarted","Data":"b30db35d3aacddc7bc654aad15f3c4139c2f7cd8e63a3ac4b5e4c9687e2cf425"} Oct 30 04:20:18.105339 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.105319 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" event={"ID":"612f19e4-0114-4d9d-b23b-48fc92b83dd0","Type":"ContainerStarted","Data":"f8c28063a79aa9356e5646e8a0d2e034f0cfcbd4ad19799a863618bc5b9cf743"} Oct 30 04:20:18.739672 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.739635 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:20:18.754353 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.754319 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:20:18.766335 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.766308 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:20:18.777968 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.777937 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:20:18.778143 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.778007 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:20:18.788034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.788008 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-r7bt7\"" Oct 30 04:20:18.788181 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.788064 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:20:18.788228 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.788189 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:20:18.788910 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.788893 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:20:18.792725 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.792707 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:18.809428 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.807731 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:20:18.870203 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.870108 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds"] Oct 30 04:20:18.872473 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.872442 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 30 04:20:18.872691 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.872672 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 04:20:18.873878 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.873859 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" Oct 30 04:20:18.897687 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.897660 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds"] Oct 30 04:20:18.911035 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.910993 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"6a4625de-0cb2-4aa9-8288-ef658dafc193\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:20:18.911194 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.911047 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r594v\" (UniqueName: \"kubernetes.io/projected/c78e806a-f197-4c83-a43f-6306f3a73539-kube-api-access-r594v\") pod \"data-science-perses-0\" (UID: \"c78e806a-f197-4c83-a43f-6306f3a73539\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:18.911194 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.911155 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c78e806a-f197-4c83-a43f-6306f3a73539-config\") pod \"data-science-perses-0\" (UID: \"c78e806a-f197-4c83-a43f-6306f3a73539\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:18.911321 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.911193 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/c78e806a-f197-4c83-a43f-6306f3a73539-plugins\") pod \"data-science-perses-0\" (UID: \"c78e806a-f197-4c83-a43f-6306f3a73539\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:18.911384 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.911320 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6a4625de-0cb2-4aa9-8288-ef658dafc193-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6a4625de-0cb2-4aa9-8288-ef658dafc193\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:20:18.911445 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.911380 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6a4625de-0cb2-4aa9-8288-ef658dafc193\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:20:18.911445 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.911407 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6a4625de-0cb2-4aa9-8288-ef658dafc193\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:20:18.911543 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.911523 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpnms\" (UniqueName: \"kubernetes.io/projected/6a4625de-0cb2-4aa9-8288-ef658dafc193-kube-api-access-kpnms\") pod \"data-science-collector-collector-0\" (UID: \"6a4625de-0cb2-4aa9-8288-ef658dafc193\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:20:18.957836 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.957804 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 30 04:20:18.957836 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.957809 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-d5kln\"" Oct 30 04:20:18.972449 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:18.972424 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 30 04:20:19.011920 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.011894 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpnms\" (UniqueName: \"kubernetes.io/projected/6a4625de-0cb2-4aa9-8288-ef658dafc193-kube-api-access-kpnms\") pod \"data-science-collector-collector-0\" (UID: \"6a4625de-0cb2-4aa9-8288-ef658dafc193\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:20:19.012034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.011941 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"6a4625de-0cb2-4aa9-8288-ef658dafc193\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:20:19.012104 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.012053 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r594v\" (UniqueName: \"kubernetes.io/projected/c78e806a-f197-4c83-a43f-6306f3a73539-kube-api-access-r594v\") pod \"data-science-perses-0\" (UID: \"c78e806a-f197-4c83-a43f-6306f3a73539\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:19.012170 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.012114 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c78e806a-f197-4c83-a43f-6306f3a73539-config\") pod \"data-science-perses-0\" (UID: \"c78e806a-f197-4c83-a43f-6306f3a73539\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:19.012170 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.012146 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/c78e806a-f197-4c83-a43f-6306f3a73539-plugins\") pod \"data-science-perses-0\" (UID: \"c78e806a-f197-4c83-a43f-6306f3a73539\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:19.012280 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.012191 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2crbt\" (UniqueName: \"kubernetes.io/projected/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-kube-api-access-2crbt\") pod \"data-science-collector-targetallocator-67b4965b4f-bhlds\" (UID: \"e4c5811e-e6f4-4800-9e34-9ad9d40690d2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" Oct 30 04:20:19.012280 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.012239 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6a4625de-0cb2-4aa9-8288-ef658dafc193-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6a4625de-0cb2-4aa9-8288-ef658dafc193\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:20:19.012390 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.012349 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6a4625de-0cb2-4aa9-8288-ef658dafc193\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:20:19.012509 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.012387 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6a4625de-0cb2-4aa9-8288-ef658dafc193\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:20:19.012509 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.012427 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-ta-internal\") pod \"data-science-collector-targetallocator-67b4965b4f-bhlds\" (UID: \"e4c5811e-e6f4-4800-9e34-9ad9d40690d2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" Oct 30 04:20:19.012509 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.012456 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-67b4965b4f-bhlds\" (UID: \"e4c5811e-e6f4-4800-9e34-9ad9d40690d2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" Oct 30 04:20:19.012702 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.012531 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/c78e806a-f197-4c83-a43f-6306f3a73539-plugins\") pod \"data-science-perses-0\" (UID: \"c78e806a-f197-4c83-a43f-6306f3a73539\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:19.012929 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.012904 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c78e806a-f197-4c83-a43f-6306f3a73539-config\") pod \"data-science-perses-0\" (UID: \"c78e806a-f197-4c83-a43f-6306f3a73539\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:19.013044 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.012958 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6a4625de-0cb2-4aa9-8288-ef658dafc193-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6a4625de-0cb2-4aa9-8288-ef658dafc193\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:20:19.014349 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.014320 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"6a4625de-0cb2-4aa9-8288-ef658dafc193\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:20:19.014456 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.014439 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6a4625de-0cb2-4aa9-8288-ef658dafc193\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:20:19.014591 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.014576 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6a4625de-0cb2-4aa9-8288-ef658dafc193\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:20:19.039715 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.039691 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpnms\" (UniqueName: \"kubernetes.io/projected/6a4625de-0cb2-4aa9-8288-ef658dafc193-kube-api-access-kpnms\") pod \"data-science-collector-collector-0\" (UID: \"6a4625de-0cb2-4aa9-8288-ef658dafc193\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:20:19.039881 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.039865 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r594v\" (UniqueName: \"kubernetes.io/projected/c78e806a-f197-4c83-a43f-6306f3a73539-kube-api-access-r594v\") pod \"data-science-perses-0\" (UID: \"c78e806a-f197-4c83-a43f-6306f3a73539\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:19.074027 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.073996 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:20:19.111147 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.111117 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" event={"ID":"612f19e4-0114-4d9d-b23b-48fc92b83dd0","Type":"ContainerStarted","Data":"ec2235ab704a4b7c7be64cc6886443d51215c926aa5861af1b538030ef40534c"} Oct 30 04:20:19.113812 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.113784 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-ta-internal\") pod \"data-science-collector-targetallocator-67b4965b4f-bhlds\" (UID: \"e4c5811e-e6f4-4800-9e34-9ad9d40690d2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" Oct 30 04:20:19.113965 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.113823 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-67b4965b4f-bhlds\" (UID: \"e4c5811e-e6f4-4800-9e34-9ad9d40690d2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" Oct 30 04:20:19.113965 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.113931 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2crbt\" (UniqueName: \"kubernetes.io/projected/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-kube-api-access-2crbt\") pod \"data-science-collector-targetallocator-67b4965b4f-bhlds\" (UID: \"e4c5811e-e6f4-4800-9e34-9ad9d40690d2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" Oct 30 04:20:19.114419 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.114393 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-ta-internal\") pod \"data-science-collector-targetallocator-67b4965b4f-bhlds\" (UID: \"e4c5811e-e6f4-4800-9e34-9ad9d40690d2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" Oct 30 04:20:19.116283 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.116261 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-67b4965b4f-bhlds\" (UID: \"e4c5811e-e6f4-4800-9e34-9ad9d40690d2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" Oct 30 04:20:19.139201 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.139176 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2crbt\" (UniqueName: \"kubernetes.io/projected/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-kube-api-access-2crbt\") pod \"data-science-collector-targetallocator-67b4965b4f-bhlds\" (UID: \"e4c5811e-e6f4-4800-9e34-9ad9d40690d2\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" Oct 30 04:20:19.155647 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.155399 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" podStartSLOduration=1.079946889 podStartE2EDuration="2.155382221s" totalImagesPullingTime="1.075435332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:19.155216081 +0000 UTC m=+5931.269531047" watchObservedRunningTime="2026-10-30 04:20:19.155382221 +0000 UTC m=+5931.269697193" Oct 30 04:20:19.188249 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.188208 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" Oct 30 04:20:19.371050 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.371022 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:20:19.372498 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:20:19.372469 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4625de_0cb2_4aa9_8288_ef658dafc193.slice/crio-1cd894af7991a8b11aac6d6430c228b1e6019d8034519c38db2f8b23c3657970 WatchSource:0}: Error finding container 1cd894af7991a8b11aac6d6430c228b1e6019d8034519c38db2f8b23c3657970: Status 404 returned error can't find the container with id 1cd894af7991a8b11aac6d6430c228b1e6019d8034519c38db2f8b23c3657970 Oct 30 04:20:19.374730 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.374698 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:20:19.374821 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.374755 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:20:19.479885 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.479856 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds"] Oct 30 04:20:19.481123 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:20:19.481099 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4c5811e_e6f4_4800_9e34_9ad9d40690d2.slice/crio-3da14286c99b0853e8e3f01554ec4c6cafa122b737563cf9f93694affcf81de2 WatchSource:0}: Error finding container 3da14286c99b0853e8e3f01554ec4c6cafa122b737563cf9f93694affcf81de2: Status 404 returned error can't find the container with id 3da14286c99b0853e8e3f01554ec4c6cafa122b737563cf9f93694affcf81de2 Oct 30 04:20:19.483888 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.483858 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:20:19.483968 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:19.483919 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:20:20.116659 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:20.116607 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6a4625de-0cb2-4aa9-8288-ef658dafc193","Type":"ContainerStarted","Data":"7455e03cbafa843a295d85b67dd628c45b25a72bd72f7e10dd299ce95c6085db"} Oct 30 04:20:20.117113 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:20.116682 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6a4625de-0cb2-4aa9-8288-ef658dafc193","Type":"ContainerStarted","Data":"1cd894af7991a8b11aac6d6430c228b1e6019d8034519c38db2f8b23c3657970"} Oct 30 04:20:20.118111 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:20.118084 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" event={"ID":"e4c5811e-e6f4-4800-9e34-9ad9d40690d2","Type":"ContainerStarted","Data":"199174d8478199ec1f42137da65ed089d2213827c03c5708f4c384f001b84c9b"} Oct 30 04:20:20.118238 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:20.118122 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" event={"ID":"e4c5811e-e6f4-4800-9e34-9ad9d40690d2","Type":"ContainerStarted","Data":"3da14286c99b0853e8e3f01554ec4c6cafa122b737563cf9f93694affcf81de2"} Oct 30 04:20:20.118421 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:20.118404 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" Oct 30 04:20:20.119781 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:20.119762 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" Oct 30 04:20:20.163386 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:20.163341 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.163330418 podStartE2EDuration="2.163330418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:20.162415095 +0000 UTC m=+5932.276730069" watchObservedRunningTime="2026-10-30 04:20:20.163330418 +0000 UTC m=+5932.277645397" Oct 30 04:20:22.352028 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.351967 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" podStartSLOduration=4.351948371 podStartE2EDuration="4.351948371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:20.210705683 +0000 UTC m=+5932.325020655" watchObservedRunningTime="2026-10-30 04:20:22.351948371 +0000 UTC m=+5934.466263348" Oct 30 04:20:22.353136 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.353111 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:20:22.358464 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.358444 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.376581 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.376556 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 04:20:22.376729 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.376557 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 04:20:22.376729 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.376644 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-fp654\"" Oct 30 04:20:22.389971 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.389942 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 04:20:22.389971 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.389946 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:20:22.392042 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.392016 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:20:22.446638 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.446581 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.446821 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.446698 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/794280a2-9fc7-425e-8f4b-9a0a25f89031-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.446821 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.446720 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vtdk\" (UniqueName: \"kubernetes.io/projected/794280a2-9fc7-425e-8f4b-9a0a25f89031-kube-api-access-6vtdk\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.446962 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.446813 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.446962 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.446883 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/794280a2-9fc7-425e-8f4b-9a0a25f89031-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.447054 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.446964 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.447054 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.446987 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/794280a2-9fc7-425e-8f4b-9a0a25f89031-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.474336 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.474303 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:20:22.478298 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.478277 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.507300 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.507269 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:20:22.548076 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.548045 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.548234 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.548089 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.548234 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.548117 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxlbq\" (UniqueName: \"kubernetes.io/projected/a582ce4e-0909-4e2e-acb4-956a215630d9-kube-api-access-qxlbq\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.548331 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.548261 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.548331 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.548297 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.548331 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.548322 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a582ce4e-0909-4e2e-acb4-956a215630d9-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.548458 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.548439 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/794280a2-9fc7-425e-8f4b-9a0a25f89031-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.548499 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.548471 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vtdk\" (UniqueName: \"kubernetes.io/projected/794280a2-9fc7-425e-8f4b-9a0a25f89031-kube-api-access-6vtdk\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.548553 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.548499 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.548553 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.548525 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.548682 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.548560 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/794280a2-9fc7-425e-8f4b-9a0a25f89031-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.548682 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.548584 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.548806 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.548680 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.548806 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.548708 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.548913 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.548816 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a582ce4e-0909-4e2e-acb4-956a215630d9-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.548913 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.548847 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.549032 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.548907 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.549032 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.548944 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.549032 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.548987 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/794280a2-9fc7-425e-8f4b-9a0a25f89031-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.549157 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.549033 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/794280a2-9fc7-425e-8f4b-9a0a25f89031-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.550848 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.550820 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/794280a2-9fc7-425e-8f4b-9a0a25f89031-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.551167 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.551147 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.551238 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.551200 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.551238 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.551200 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.551496 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.551480 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/794280a2-9fc7-425e-8f4b-9a0a25f89031-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.553326 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.553305 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 04:20:22.553326 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.553309 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 04:20:22.553490 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.553432 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 04:20:22.573828 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.573794 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vtdk\" (UniqueName: \"kubernetes.io/projected/794280a2-9fc7-425e-8f4b-9a0a25f89031-kube-api-access-6vtdk\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.577037 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.577017 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 04:20:22.577268 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.577250 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 04:20:22.577340 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.577252 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:20:22.577452 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.577434 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 04:20:22.577540 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.577450 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 04:20:22.577540 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.577458 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 04:20:22.577540 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.577478 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-fsr8m\"" Oct 30 04:20:22.650192 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.650108 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.650192 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.650156 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.650192 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.650183 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.650473 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.650211 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.650473 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.650278 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a582ce4e-0909-4e2e-acb4-956a215630d9-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.650473 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.650308 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.650473 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.650331 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.650473 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.650366 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.650473 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.650408 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxlbq\" (UniqueName: \"kubernetes.io/projected/a582ce4e-0909-4e2e-acb4-956a215630d9-kube-api-access-qxlbq\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.650473 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.650441 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.650974 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.650478 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.650974 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.650513 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a582ce4e-0909-4e2e-acb4-956a215630d9-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.650974 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.650950 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.651150 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.651129 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.651553 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.651525 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.653151 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.653105 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a582ce4e-0909-4e2e-acb4-956a215630d9-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.653344 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.653319 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.653458 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.653433 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a582ce4e-0909-4e2e-acb4-956a215630d9-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.653757 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.653716 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.654085 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.654068 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.654229 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.654206 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.655107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.655085 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.655317 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.655300 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.676247 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.676227 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxlbq\" (UniqueName: \"kubernetes.io/projected/a582ce4e-0909-4e2e-acb4-956a215630d9-kube-api-access-qxlbq\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:22.681066 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.681047 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:22.751320 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.751293 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"c78e806a-f197-4c83-a43f-6306f3a73539\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:22.784136 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.784103 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"c78e806a-f197-4c83-a43f-6306f3a73539\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:22.852504 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.852473 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"c78e806a-f197-4c83-a43f-6306f3a73539\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:22.852684 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.852566 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"c78e806a-f197-4c83-a43f-6306f3a73539\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:22.868719 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.868688 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"c78e806a-f197-4c83-a43f-6306f3a73539\") DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:22.967953 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.967916 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"c78e806a-f197-4c83-a43f-6306f3a73539\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2c422b71c1e0b8fdf34659f2cef7846e06b4353ed88027dc99ade5d7fc29e9d6/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:22.991382 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:22.991354 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:20:23.001675 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:23.001651 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"c78e806a-f197-4c83-a43f-6306f3a73539\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:23.133822 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:23.133789 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"794280a2-9fc7-425e-8f4b-9a0a25f89031","Type":"ContainerStarted","Data":"4d88e354a718a42e1f547b920ebbf2c720901593b62f7d147ebeb36a2f332972"} Oct 30 04:20:23.314407 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:23.314374 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:23.607550 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:23.607517 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:20:23.608311 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:20:23.608276 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc78e806a_f197_4c83_a43f_6306f3a73539.slice/crio-2e6c721448e646ac10d402545d088aafd98fb64837a78de9073f53c06e4fbbb7 WatchSource:0}: Error finding container 2e6c721448e646ac10d402545d088aafd98fb64837a78de9073f53c06e4fbbb7: Status 404 returned error can't find the container with id 2e6c721448e646ac10d402545d088aafd98fb64837a78de9073f53c06e4fbbb7 Oct 30 04:20:23.944136 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:20:23.944053 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:20:24.140514 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:24.140474 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"c78e806a-f197-4c83-a43f-6306f3a73539","Type":"ContainerStarted","Data":"c5d9bf9ed6f0789e23fbfafb388d4109eed1015e5058c1491d461585326d9c3d"} Oct 30 04:20:24.140732 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:24.140523 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"c78e806a-f197-4c83-a43f-6306f3a73539","Type":"ContainerStarted","Data":"2e6c721448e646ac10d402545d088aafd98fb64837a78de9073f53c06e4fbbb7"} Oct 30 04:20:24.140732 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:24.140651 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:24.142197 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:24.142168 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"794280a2-9fc7-425e-8f4b-9a0a25f89031","Type":"ContainerStarted","Data":"75f588b157b6fa4a3abb78c3b3aa13288ab91ae342ef5f242bcc4164b9b1892c"} Oct 30 04:20:24.185477 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:24.185413 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=6.18539649 podStartE2EDuration="6.18539649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:24.18453548 +0000 UTC m=+5936.298850457" watchObservedRunningTime="2026-10-30 04:20:24.18539649 +0000 UTC m=+5936.299711462" Oct 30 04:20:25.375869 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:25.375829 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:25.408913 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:25.408872 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:25.476937 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:25.476898 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:25.477127 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:25.477066 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:25.493126 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:25.493098 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") DevicePath \"csi-927d18ef2eb99137c68fbe5e0bb8ea629bb5219ed56cdede72da7c0f045be95e\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:25.598647 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:25.598598 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a535375d22e374ba333968465471c7604a06f9d95a513910fe4bb083b629e986/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:25.630057 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:25.629988 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:25.798588 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:25.798546 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:26.148042 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:20:26.148008 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda582ce4e_0909_4e2e_acb4_956a215630d9.slice/crio-a02b6deb384b099aa486d9c0538db187acee16f8853f0d704d320d7cacd67db0 WatchSource:0}: Error finding container a02b6deb384b099aa486d9c0538db187acee16f8853f0d704d320d7cacd67db0: Status 404 returned error can't find the container with id a02b6deb384b099aa486d9c0538db187acee16f8853f0d704d320d7cacd67db0 Oct 30 04:20:26.152211 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:26.152183 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:20:27.159308 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:27.159272 5882 generic.go:356] "Generic (PLEG): container finished" podID="794280a2-9fc7-425e-8f4b-9a0a25f89031" containerID="75f588b157b6fa4a3abb78c3b3aa13288ab91ae342ef5f242bcc4164b9b1892c" exitCode=0 Oct 30 04:20:27.159808 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:27.159347 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"794280a2-9fc7-425e-8f4b-9a0a25f89031","Type":"ContainerDied","Data":"75f588b157b6fa4a3abb78c3b3aa13288ab91ae342ef5f242bcc4164b9b1892c"} Oct 30 04:20:27.161049 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:27.161022 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a582ce4e-0909-4e2e-acb4-956a215630d9","Type":"ContainerStarted","Data":"516671a81e2dd8f2a80f5ace91442941cfc76e2475a71c89e495c70b25a9a434"} Oct 30 04:20:27.161160 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:27.161092 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a582ce4e-0909-4e2e-acb4-956a215630d9","Type":"ContainerStarted","Data":"a02b6deb384b099aa486d9c0538db187acee16f8853f0d704d320d7cacd67db0"} Oct 30 04:20:28.168014 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:28.167927 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"794280a2-9fc7-425e-8f4b-9a0a25f89031","Type":"ContainerStarted","Data":"2d7e07c5e5c2385e472afcddfbeb7f8a9554eb3cf543191decb05a01a85037c2"} Oct 30 04:20:28.168014 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:28.168005 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"794280a2-9fc7-425e-8f4b-9a0a25f89031","Type":"ContainerStarted","Data":"886b28f978320fcb5d31bfa97da1e8f6e8fdf79cf028236d4d8c59dd01ab4bda"} Oct 30 04:20:28.168452 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:28.168315 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:28.220931 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:28.220875 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.034089618 podStartE2EDuration="6.220862218s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.1867726s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:28.220784726 +0000 UTC m=+5940.335099701" watchObservedRunningTime="2026-10-30 04:20:28.220862218 +0000 UTC m=+5940.335177190" Oct 30 04:20:30.178393 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:30.178360 5882 generic.go:356] "Generic (PLEG): container finished" podID="a582ce4e-0909-4e2e-acb4-956a215630d9" containerID="516671a81e2dd8f2a80f5ace91442941cfc76e2475a71c89e495c70b25a9a434" exitCode=0 Oct 30 04:20:30.178811 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:30.178426 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a582ce4e-0909-4e2e-acb4-956a215630d9","Type":"ContainerDied","Data":"516671a81e2dd8f2a80f5ace91442941cfc76e2475a71c89e495c70b25a9a434"} Oct 30 04:20:31.186103 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:31.186069 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a582ce4e-0909-4e2e-acb4-956a215630d9","Type":"ContainerStarted","Data":"ae9fd1f4dff510f477128b002f999c2a1093eca6bef3b5bf7f730577b60fc1d6"} Oct 30 04:20:31.186517 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:31.186135 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a582ce4e-0909-4e2e-acb4-956a215630d9","Type":"ContainerStarted","Data":"a9d2b0ea4664f875c8a9931ed21b6665cafbe285cf3203ea536c0e4ffbd617ec"} Oct 30 04:20:31.186517 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:31.186158 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a582ce4e-0909-4e2e-acb4-956a215630d9","Type":"ContainerStarted","Data":"c556a7a6ff399ba1d13edd11ad1b563438475489c37058c77b951ae8669990f0"} Oct 30 04:20:31.271338 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:31.271283 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=5.979731553 podStartE2EDuration="9.271269264s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.291537711s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:31.25642295 +0000 UTC m=+5943.370737936" watchObservedRunningTime="2026-10-30 04:20:31.271269264 +0000 UTC m=+5943.385584258" Oct 30 04:20:34.174449 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:34.174415 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:20:35.148770 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:35.148740 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:20:35.799214 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:35.799178 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:40.799011 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:40.798929 5882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:40.804139 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:40.804116 5882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:41.236552 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:41.236474 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:20:43.012479 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:43.012445 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:20:43.012993 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:43.012674 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="6a4625de-0cb2-4aa9-8288-ef658dafc193" containerName="otc-container" containerID="cri-o://7455e03cbafa843a295d85b67dd628c45b25a72bd72f7e10dd299ce95c6085db" gracePeriod=30 Oct 30 04:20:43.999133 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:43.999099 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds"] Oct 30 04:20:43.999361 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:43.999315 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" podUID="e4c5811e-e6f4-4800-9e34-9ad9d40690d2" containerName="ta-container" containerID="cri-o://199174d8478199ec1f42137da65ed089d2213827c03c5708f4c384f001b84c9b" gracePeriod=30 Oct 30 04:20:44.247235 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:44.247199 5882 generic.go:356] "Generic (PLEG): container finished" podID="e4c5811e-e6f4-4800-9e34-9ad9d40690d2" containerID="199174d8478199ec1f42137da65ed089d2213827c03c5708f4c384f001b84c9b" exitCode=0 Oct 30 04:20:44.247591 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:44.247247 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" event={"ID":"e4c5811e-e6f4-4800-9e34-9ad9d40690d2","Type":"ContainerDied","Data":"199174d8478199ec1f42137da65ed089d2213827c03c5708f4c384f001b84c9b"} Oct 30 04:20:44.247591 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:44.247293 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" event={"ID":"e4c5811e-e6f4-4800-9e34-9ad9d40690d2","Type":"ContainerDied","Data":"3da14286c99b0853e8e3f01554ec4c6cafa122b737563cf9f93694affcf81de2"} Oct 30 04:20:44.247591 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:44.247305 5882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3da14286c99b0853e8e3f01554ec4c6cafa122b737563cf9f93694affcf81de2" Oct 30 04:20:44.250532 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:44.250481 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" Oct 30 04:20:44.354309 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:44.354269 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-data-science-collector-ta-server-cert\") pod \"e4c5811e-e6f4-4800-9e34-9ad9d40690d2\" (UID: \"e4c5811e-e6f4-4800-9e34-9ad9d40690d2\") " Oct 30 04:20:44.354484 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:44.354392 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-ta-internal\" (UniqueName: \"kubernetes.io/configmap/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-ta-internal\") pod \"e4c5811e-e6f4-4800-9e34-9ad9d40690d2\" (UID: \"e4c5811e-e6f4-4800-9e34-9ad9d40690d2\") " Oct 30 04:20:44.354484 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:44.354420 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-kube-api-access-2crbt\" (UniqueName: \"kubernetes.io/projected/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-kube-api-access-2crbt\") pod \"e4c5811e-e6f4-4800-9e34-9ad9d40690d2\" (UID: \"e4c5811e-e6f4-4800-9e34-9ad9d40690d2\") " Oct 30 04:20:44.354768 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:44.354741 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-ta-internal" pod "e4c5811e-e6f4-4800-9e34-9ad9d40690d2" (UID: "e4c5811e-e6f4-4800-9e34-9ad9d40690d2"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:44.356409 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:44.356386 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-kube-api-access-2crbt" pod "e4c5811e-e6f4-4800-9e34-9ad9d40690d2" (UID: "e4c5811e-e6f4-4800-9e34-9ad9d40690d2"). InnerVolumeSpecName "kube-api-access-2crbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:44.356484 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:44.356467 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-data-science-collector-ta-server-cert" pod "e4c5811e-e6f4-4800-9e34-9ad9d40690d2" (UID: "e4c5811e-e6f4-4800-9e34-9ad9d40690d2"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:44.455545 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:44.455510 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:44.455545 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:44.455541 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2crbt\" (UniqueName: \"kubernetes.io/projected/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-kube-api-access-2crbt\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:44.455774 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:44.455554 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/e4c5811e-e6f4-4800-9e34-9ad9d40690d2-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:45.254172 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.254121 5882 generic.go:356] "Generic (PLEG): container finished" podID="6a4625de-0cb2-4aa9-8288-ef658dafc193" containerID="7455e03cbafa843a295d85b67dd628c45b25a72bd72f7e10dd299ce95c6085db" exitCode=0 Oct 30 04:20:45.254607 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.254286 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6a4625de-0cb2-4aa9-8288-ef658dafc193","Type":"ContainerDied","Data":"7455e03cbafa843a295d85b67dd628c45b25a72bd72f7e10dd299ce95c6085db"} Oct 30 04:20:45.254607 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.254320 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds" Oct 30 04:20:45.310794 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.310760 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds"] Oct 30 04:20:45.324568 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.324538 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-67b4965b4f-bhlds"] Oct 30 04:20:45.356449 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.356415 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:20:45.465849 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.465766 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-monitoring-tls-certs\") pod \"6a4625de-0cb2-4aa9-8288-ef658dafc193\" (UID: \"6a4625de-0cb2-4aa9-8288-ef658dafc193\") " Oct 30 04:20:45.465849 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.465828 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a4625de-0cb2-4aa9-8288-ef658dafc193-otc-internal\" (UniqueName: \"kubernetes.io/configmap/6a4625de-0cb2-4aa9-8288-ef658dafc193-otc-internal\") pod \"6a4625de-0cb2-4aa9-8288-ef658dafc193\" (UID: \"6a4625de-0cb2-4aa9-8288-ef658dafc193\") " Oct 30 04:20:45.466049 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.465876 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a4625de-0cb2-4aa9-8288-ef658dafc193-kube-api-access-kpnms\" (UniqueName: \"kubernetes.io/projected/6a4625de-0cb2-4aa9-8288-ef658dafc193-kube-api-access-kpnms\") pod \"6a4625de-0cb2-4aa9-8288-ef658dafc193\" (UID: \"6a4625de-0cb2-4aa9-8288-ef658dafc193\") " Oct 30 04:20:45.466049 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.465952 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-data-science-collector-ta-client-cert\") pod \"6a4625de-0cb2-4aa9-8288-ef658dafc193\" (UID: \"6a4625de-0cb2-4aa9-8288-ef658dafc193\") " Oct 30 04:20:45.466049 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.465983 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-tls-certs\") pod \"6a4625de-0cb2-4aa9-8288-ef658dafc193\" (UID: \"6a4625de-0cb2-4aa9-8288-ef658dafc193\") " Oct 30 04:20:45.466208 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.466179 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a4625de-0cb2-4aa9-8288-ef658dafc193-otc-internal" pod "6a4625de-0cb2-4aa9-8288-ef658dafc193" (UID: "6a4625de-0cb2-4aa9-8288-ef658dafc193"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:45.467999 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.467971 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-data-science-collector-ta-client-cert" pod "6a4625de-0cb2-4aa9-8288-ef658dafc193" (UID: "6a4625de-0cb2-4aa9-8288-ef658dafc193"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:45.468107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.467997 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-monitoring-tls-certs" pod "6a4625de-0cb2-4aa9-8288-ef658dafc193" (UID: "6a4625de-0cb2-4aa9-8288-ef658dafc193"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:45.468107 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.468061 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-tls-certs" pod "6a4625de-0cb2-4aa9-8288-ef658dafc193" (UID: "6a4625de-0cb2-4aa9-8288-ef658dafc193"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:45.468185 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.468105 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a4625de-0cb2-4aa9-8288-ef658dafc193-kube-api-access-kpnms" pod "6a4625de-0cb2-4aa9-8288-ef658dafc193" (UID: "6a4625de-0cb2-4aa9-8288-ef658dafc193"). InnerVolumeSpecName "kube-api-access-kpnms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:45.567329 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.567283 5882 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6a4625de-0cb2-4aa9-8288-ef658dafc193-otc-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:45.567329 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.567323 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpnms\" (UniqueName: \"kubernetes.io/projected/6a4625de-0cb2-4aa9-8288-ef658dafc193-kube-api-access-kpnms\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:45.567329 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.567333 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:45.567543 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.567344 5882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:45.567543 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:45.567353 5882 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4625de-0cb2-4aa9-8288-ef658dafc193-monitoring-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:46.009390 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:46.009355 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c5811e-e6f4-4800-9e34-9ad9d40690d2" path="/var/lib/kubelet/pods/e4c5811e-e6f4-4800-9e34-9ad9d40690d2/volumes" Oct 30 04:20:46.260069 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:46.259992 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:20:46.260560 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:46.260000 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6a4625de-0cb2-4aa9-8288-ef658dafc193","Type":"ContainerDied","Data":"1cd894af7991a8b11aac6d6430c228b1e6019d8034519c38db2f8b23c3657970"} Oct 30 04:20:46.260560 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:46.260131 5882 scope.go:122] "RemoveContainer" containerID="7455e03cbafa843a295d85b67dd628c45b25a72bd72f7e10dd299ce95c6085db" Oct 30 04:20:46.316559 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:46.316526 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:20:46.331244 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:46.331214 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:20:48.010465 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:48.010432 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a4625de-0cb2-4aa9-8288-ef658dafc193" path="/var/lib/kubelet/pods/6a4625de-0cb2-4aa9-8288-ef658dafc193/volumes" Oct 30 04:20:53.942858 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:20:53.942817 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:20:54.544163 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.544126 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf"] Oct 30 04:20:54.544533 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.544520 5882 cpu_manager.go:414] "containerMap: removing container" podUID="6a4625de-0cb2-4aa9-8288-ef658dafc193" containerName="otc-container" Oct 30 04:20:54.544584 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.544534 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a4625de-0cb2-4aa9-8288-ef658dafc193" containerName="otc-container" Oct 30 04:20:54.544584 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.544547 5882 cpu_manager.go:414] "containerMap: removing container" podUID="e4c5811e-e6f4-4800-9e34-9ad9d40690d2" containerName="ta-container" Oct 30 04:20:54.544584 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.544552 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c5811e-e6f4-4800-9e34-9ad9d40690d2" containerName="ta-container" Oct 30 04:20:54.544731 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.544605 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c5811e-e6f4-4800-9e34-9ad9d40690d2" containerName="ta-container" Oct 30 04:20:54.544731 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.544648 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a4625de-0cb2-4aa9-8288-ef658dafc193" containerName="otc-container" Oct 30 04:20:54.547858 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.547839 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" Oct 30 04:20:54.579821 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.579795 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 30 04:20:54.579934 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.579909 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-d5kln\"" Oct 30 04:20:54.580875 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.580856 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf"] Oct 30 04:20:54.649927 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.649896 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 30 04:20:54.653282 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.653255 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/658e0853-8ce7-45fb-9383-02840e509cc8-ta-internal\") pod \"data-science-collector-targetallocator-5b8fbb66ff-qzdmf\" (UID: \"658e0853-8ce7-45fb-9383-02840e509cc8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" Oct 30 04:20:54.653387 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.653372 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/658e0853-8ce7-45fb-9383-02840e509cc8-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5b8fbb66ff-qzdmf\" (UID: \"658e0853-8ce7-45fb-9383-02840e509cc8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" Oct 30 04:20:54.653432 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.653406 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr8t5\" (UniqueName: \"kubernetes.io/projected/658e0853-8ce7-45fb-9383-02840e509cc8-kube-api-access-lr8t5\") pod \"data-science-collector-targetallocator-5b8fbb66ff-qzdmf\" (UID: \"658e0853-8ce7-45fb-9383-02840e509cc8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" Oct 30 04:20:54.754776 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.754736 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/658e0853-8ce7-45fb-9383-02840e509cc8-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5b8fbb66ff-qzdmf\" (UID: \"658e0853-8ce7-45fb-9383-02840e509cc8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" Oct 30 04:20:54.754960 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.754784 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr8t5\" (UniqueName: \"kubernetes.io/projected/658e0853-8ce7-45fb-9383-02840e509cc8-kube-api-access-lr8t5\") pod \"data-science-collector-targetallocator-5b8fbb66ff-qzdmf\" (UID: \"658e0853-8ce7-45fb-9383-02840e509cc8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" Oct 30 04:20:54.754960 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.754846 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/658e0853-8ce7-45fb-9383-02840e509cc8-ta-internal\") pod \"data-science-collector-targetallocator-5b8fbb66ff-qzdmf\" (UID: \"658e0853-8ce7-45fb-9383-02840e509cc8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" Oct 30 04:20:54.755519 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.755498 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/658e0853-8ce7-45fb-9383-02840e509cc8-ta-internal\") pod \"data-science-collector-targetallocator-5b8fbb66ff-qzdmf\" (UID: \"658e0853-8ce7-45fb-9383-02840e509cc8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" Oct 30 04:20:54.757259 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.757237 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/658e0853-8ce7-45fb-9383-02840e509cc8-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5b8fbb66ff-qzdmf\" (UID: \"658e0853-8ce7-45fb-9383-02840e509cc8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" Oct 30 04:20:54.779860 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.779830 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr8t5\" (UniqueName: \"kubernetes.io/projected/658e0853-8ce7-45fb-9383-02840e509cc8-kube-api-access-lr8t5\") pod \"data-science-collector-targetallocator-5b8fbb66ff-qzdmf\" (UID: \"658e0853-8ce7-45fb-9383-02840e509cc8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" Oct 30 04:20:54.859300 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:54.859224 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" Oct 30 04:20:55.150906 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:55.150872 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf"] Oct 30 04:20:55.151397 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:20:55.151377 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod658e0853_8ce7_45fb_9383_02840e509cc8.slice/crio-a75608ab7ecc013dd10ffc8e51dea8cddacc11026e2e7a520e0e83c0f198e86c WatchSource:0}: Error finding container a75608ab7ecc013dd10ffc8e51dea8cddacc11026e2e7a520e0e83c0f198e86c: Status 404 returned error can't find the container with id a75608ab7ecc013dd10ffc8e51dea8cddacc11026e2e7a520e0e83c0f198e86c Oct 30 04:20:55.153780 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:55.153737 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:20:55.153869 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:55.153810 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:20:55.301718 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:55.301684 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" event={"ID":"658e0853-8ce7-45fb-9383-02840e509cc8","Type":"ContainerStarted","Data":"10131b75e2667ebc43c2a38ed46bf0e5175c3d518981768049f38db0510b616d"} Oct 30 04:20:55.301918 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:55.301730 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" event={"ID":"658e0853-8ce7-45fb-9383-02840e509cc8","Type":"ContainerStarted","Data":"a75608ab7ecc013dd10ffc8e51dea8cddacc11026e2e7a520e0e83c0f198e86c"} Oct 30 04:20:55.301918 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:55.301834 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" Oct 30 04:20:55.302946 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:55.302927 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" podUID="658e0853-8ce7-45fb-9383-02840e509cc8" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.182:8080/readyz\": dial tcp 10.134.0.182:8080: connect: connection refused" Oct 30 04:20:55.347475 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:55.347418 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" podStartSLOduration=1.347400669 podStartE2EDuration="1.347400669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:55.346551084 +0000 UTC m=+5967.460866060" watchObservedRunningTime="2026-10-30 04:20:55.347400669 +0000 UTC m=+5967.461715644" Oct 30 04:20:56.307240 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:56.307208 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" Oct 30 04:20:59.131135 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.131102 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh"] Oct 30 04:20:59.131609 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.131329 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" podUID="612f19e4-0114-4d9d-b23b-48fc92b83dd0" containerName="kube-rbac-proxy" containerID="cri-o://b30db35d3aacddc7bc654aad15f3c4139c2f7cd8e63a3ac4b5e4c9687e2cf425" gracePeriod=30 Oct 30 04:20:59.131609 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.131403 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" podUID="612f19e4-0114-4d9d-b23b-48fc92b83dd0" containerName="prom-label-proxy" containerID="cri-o://ec2235ab704a4b7c7be64cc6886443d51215c926aa5861af1b538030ef40534c" gracePeriod=30 Oct 30 04:20:59.321709 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.321671 5882 generic.go:356] "Generic (PLEG): container finished" podID="612f19e4-0114-4d9d-b23b-48fc92b83dd0" containerID="ec2235ab704a4b7c7be64cc6886443d51215c926aa5861af1b538030ef40534c" exitCode=0 Oct 30 04:20:59.321709 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.321708 5882 generic.go:356] "Generic (PLEG): container finished" podID="612f19e4-0114-4d9d-b23b-48fc92b83dd0" containerID="b30db35d3aacddc7bc654aad15f3c4139c2f7cd8e63a3ac4b5e4c9687e2cf425" exitCode=0 Oct 30 04:20:59.321963 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.321747 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" event={"ID":"612f19e4-0114-4d9d-b23b-48fc92b83dd0","Type":"ContainerDied","Data":"ec2235ab704a4b7c7be64cc6886443d51215c926aa5861af1b538030ef40534c"} Oct 30 04:20:59.321963 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.321844 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" event={"ID":"612f19e4-0114-4d9d-b23b-48fc92b83dd0","Type":"ContainerDied","Data":"b30db35d3aacddc7bc654aad15f3c4139c2f7cd8e63a3ac4b5e4c9687e2cf425"} Oct 30 04:20:59.377656 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.377610 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" Oct 30 04:20:59.391755 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.391696 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/612f19e4-0114-4d9d-b23b-48fc92b83dd0-kube-api-access-p2zzj\" (UniqueName: \"kubernetes.io/projected/612f19e4-0114-4d9d-b23b-48fc92b83dd0-kube-api-access-p2zzj\") pod \"612f19e4-0114-4d9d-b23b-48fc92b83dd0\" (UID: \"612f19e4-0114-4d9d-b23b-48fc92b83dd0\") " Oct 30 04:20:59.391864 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.391766 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/612f19e4-0114-4d9d-b23b-48fc92b83dd0-config\" (UniqueName: \"kubernetes.io/configmap/612f19e4-0114-4d9d-b23b-48fc92b83dd0-config\") pod \"612f19e4-0114-4d9d-b23b-48fc92b83dd0\" (UID: \"612f19e4-0114-4d9d-b23b-48fc92b83dd0\") " Oct 30 04:20:59.391864 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.391838 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/612f19e4-0114-4d9d-b23b-48fc92b83dd0-tls\" (UniqueName: \"kubernetes.io/secret/612f19e4-0114-4d9d-b23b-48fc92b83dd0-tls\") pod \"612f19e4-0114-4d9d-b23b-48fc92b83dd0\" (UID: \"612f19e4-0114-4d9d-b23b-48fc92b83dd0\") " Oct 30 04:20:59.392193 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.392165 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612f19e4-0114-4d9d-b23b-48fc92b83dd0-config" pod "612f19e4-0114-4d9d-b23b-48fc92b83dd0" (UID: "612f19e4-0114-4d9d-b23b-48fc92b83dd0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:59.393771 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.393750 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612f19e4-0114-4d9d-b23b-48fc92b83dd0-kube-api-access-p2zzj" pod "612f19e4-0114-4d9d-b23b-48fc92b83dd0" (UID: "612f19e4-0114-4d9d-b23b-48fc92b83dd0"). InnerVolumeSpecName "kube-api-access-p2zzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:59.393840 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.393752 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/612f19e4-0114-4d9d-b23b-48fc92b83dd0-tls" pod "612f19e4-0114-4d9d-b23b-48fc92b83dd0" (UID: "612f19e4-0114-4d9d-b23b-48fc92b83dd0"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:59.493284 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.493247 5882 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/612f19e4-0114-4d9d-b23b-48fc92b83dd0-tls\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:59.493284 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.493287 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2zzj\" (UniqueName: \"kubernetes.io/projected/612f19e4-0114-4d9d-b23b-48fc92b83dd0-kube-api-access-p2zzj\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:59.493471 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.493304 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/612f19e4-0114-4d9d-b23b-48fc92b83dd0-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:20:59.656981 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.656892 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:20:59.657239 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.657206 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="a582ce4e-0909-4e2e-acb4-956a215630d9" containerName="prometheus" containerID="cri-o://c556a7a6ff399ba1d13edd11ad1b563438475489c37058c77b951ae8669990f0" gracePeriod=600 Oct 30 04:20:59.657385 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.657257 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="a582ce4e-0909-4e2e-acb4-956a215630d9" containerName="thanos-sidecar" containerID="cri-o://ae9fd1f4dff510f477128b002f999c2a1093eca6bef3b5bf7f730577b60fc1d6" gracePeriod=600 Oct 30 04:20:59.657385 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.657338 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="a582ce4e-0909-4e2e-acb4-956a215630d9" containerName="config-reloader" containerID="cri-o://a9d2b0ea4664f875c8a9931ed21b6665cafbe285cf3203ea536c0e4ffbd617ec" gracePeriod=600 Oct 30 04:20:59.664047 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.664021 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:20:59.664366 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.664341 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="794280a2-9fc7-425e-8f4b-9a0a25f89031" containerName="alertmanager" containerID="cri-o://886b28f978320fcb5d31bfa97da1e8f6e8fdf79cf028236d4d8c59dd01ab4bda" gracePeriod=120 Oct 30 04:20:59.664437 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:20:59.664365 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="794280a2-9fc7-425e-8f4b-9a0a25f89031" containerName="config-reloader" containerID="cri-o://2d7e07c5e5c2385e472afcddfbeb7f8a9554eb3cf543191decb05a01a85037c2" gracePeriod=120 Oct 30 04:21:00.319586 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.319557 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf"] Oct 30 04:21:00.320007 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.319780 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" podUID="658e0853-8ce7-45fb-9383-02840e509cc8" containerName="ta-container" containerID="cri-o://10131b75e2667ebc43c2a38ed46bf0e5175c3d518981768049f38db0510b616d" gracePeriod=30 Oct 30 04:21:00.330428 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.330400 5882 generic.go:356] "Generic (PLEG): container finished" podID="794280a2-9fc7-425e-8f4b-9a0a25f89031" containerID="2d7e07c5e5c2385e472afcddfbeb7f8a9554eb3cf543191decb05a01a85037c2" exitCode=0 Oct 30 04:21:00.330570 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.330479 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"794280a2-9fc7-425e-8f4b-9a0a25f89031","Type":"ContainerDied","Data":"2d7e07c5e5c2385e472afcddfbeb7f8a9554eb3cf543191decb05a01a85037c2"} Oct 30 04:21:00.333819 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.333789 5882 generic.go:356] "Generic (PLEG): container finished" podID="a582ce4e-0909-4e2e-acb4-956a215630d9" containerID="ae9fd1f4dff510f477128b002f999c2a1093eca6bef3b5bf7f730577b60fc1d6" exitCode=0 Oct 30 04:21:00.333819 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.333816 5882 generic.go:356] "Generic (PLEG): container finished" podID="a582ce4e-0909-4e2e-acb4-956a215630d9" containerID="a9d2b0ea4664f875c8a9931ed21b6665cafbe285cf3203ea536c0e4ffbd617ec" exitCode=0 Oct 30 04:21:00.333996 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.333819 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a582ce4e-0909-4e2e-acb4-956a215630d9","Type":"ContainerDied","Data":"ae9fd1f4dff510f477128b002f999c2a1093eca6bef3b5bf7f730577b60fc1d6"} Oct 30 04:21:00.333996 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.333896 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a582ce4e-0909-4e2e-acb4-956a215630d9","Type":"ContainerDied","Data":"a9d2b0ea4664f875c8a9931ed21b6665cafbe285cf3203ea536c0e4ffbd617ec"} Oct 30 04:21:00.335682 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.335658 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" event={"ID":"612f19e4-0114-4d9d-b23b-48fc92b83dd0","Type":"ContainerDied","Data":"f8c28063a79aa9356e5646e8a0d2e034f0cfcbd4ad19799a863618bc5b9cf743"} Oct 30 04:21:00.335805 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.335690 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh" Oct 30 04:21:00.335805 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.335718 5882 scope.go:122] "RemoveContainer" containerID="ec2235ab704a4b7c7be64cc6886443d51215c926aa5861af1b538030ef40534c" Oct 30 04:21:00.392926 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.392898 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:21:00.393103 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.393081 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="c78e806a-f197-4c83-a43f-6306f3a73539" containerName="perses" containerID="cri-o://c5d9bf9ed6f0789e23fbfafb388d4109eed1015e5058c1491d461585326d9c3d" gracePeriod=30 Oct 30 04:21:00.508118 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.508093 5882 scope.go:122] "RemoveContainer" containerID="b30db35d3aacddc7bc654aad15f3c4139c2f7cd8e63a3ac4b5e4c9687e2cf425" Oct 30 04:21:00.546382 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.545981 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh"] Oct 30 04:21:00.551242 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.551170 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-6bhnh"] Oct 30 04:21:00.563958 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.563671 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:00.604297 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.604267 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-config-volume\" (UniqueName: \"kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-config-volume\") pod \"794280a2-9fc7-425e-8f4b-9a0a25f89031\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " Oct 30 04:21:00.604438 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.604315 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/794280a2-9fc7-425e-8f4b-9a0a25f89031-kube-api-access-6vtdk\" (UniqueName: \"kubernetes.io/projected/794280a2-9fc7-425e-8f4b-9a0a25f89031-kube-api-access-6vtdk\") pod \"794280a2-9fc7-425e-8f4b-9a0a25f89031\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " Oct 30 04:21:00.604438 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.604362 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-cluster-tls-config\") pod \"794280a2-9fc7-425e-8f4b-9a0a25f89031\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " Oct 30 04:21:00.604438 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.604412 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/794280a2-9fc7-425e-8f4b-9a0a25f89031-tls-assets\" (UniqueName: \"kubernetes.io/projected/794280a2-9fc7-425e-8f4b-9a0a25f89031-tls-assets\") pod \"794280a2-9fc7-425e-8f4b-9a0a25f89031\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " Oct 30 04:21:00.604629 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.604467 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794280a2-9fc7-425e-8f4b-9a0a25f89031-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/794280a2-9fc7-425e-8f4b-9a0a25f89031-alertmanager-data-science-monitoringstack-db\") pod \"794280a2-9fc7-425e-8f4b-9a0a25f89031\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " Oct 30 04:21:00.604629 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.604508 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794280a2-9fc7-425e-8f4b-9a0a25f89031-config-out\" (UniqueName: \"kubernetes.io/empty-dir/794280a2-9fc7-425e-8f4b-9a0a25f89031-config-out\") pod \"794280a2-9fc7-425e-8f4b-9a0a25f89031\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " Oct 30 04:21:00.604629 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.604575 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-web-config\" (UniqueName: \"kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-web-config\") pod \"794280a2-9fc7-425e-8f4b-9a0a25f89031\" (UID: \"794280a2-9fc7-425e-8f4b-9a0a25f89031\") " Oct 30 04:21:00.605056 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.604945 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794280a2-9fc7-425e-8f4b-9a0a25f89031-alertmanager-data-science-monitoringstack-db" pod "794280a2-9fc7-425e-8f4b-9a0a25f89031" (UID: "794280a2-9fc7-425e-8f4b-9a0a25f89031"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:21:00.606938 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.606891 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-config-volume" pod "794280a2-9fc7-425e-8f4b-9a0a25f89031" (UID: "794280a2-9fc7-425e-8f4b-9a0a25f89031"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:00.607170 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.607130 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794280a2-9fc7-425e-8f4b-9a0a25f89031-kube-api-access-6vtdk" pod "794280a2-9fc7-425e-8f4b-9a0a25f89031" (UID: "794280a2-9fc7-425e-8f4b-9a0a25f89031"). InnerVolumeSpecName "kube-api-access-6vtdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:00.607649 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.607597 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794280a2-9fc7-425e-8f4b-9a0a25f89031-tls-assets" pod "794280a2-9fc7-425e-8f4b-9a0a25f89031" (UID: "794280a2-9fc7-425e-8f4b-9a0a25f89031"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:00.607738 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.607685 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794280a2-9fc7-425e-8f4b-9a0a25f89031-config-out" pod "794280a2-9fc7-425e-8f4b-9a0a25f89031" (UID: "794280a2-9fc7-425e-8f4b-9a0a25f89031"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:21:00.611951 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.611928 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-cluster-tls-config" pod "794280a2-9fc7-425e-8f4b-9a0a25f89031" (UID: "794280a2-9fc7-425e-8f4b-9a0a25f89031"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:00.624715 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.624681 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-web-config" pod "794280a2-9fc7-425e-8f4b-9a0a25f89031" (UID: "794280a2-9fc7-425e-8f4b-9a0a25f89031"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:00.656189 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.656167 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" Oct 30 04:21:00.658191 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.658173 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:00.705461 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.705434 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/658e0853-8ce7-45fb-9383-02840e509cc8-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/658e0853-8ce7-45fb-9383-02840e509cc8-data-science-collector-ta-server-cert\") pod \"658e0853-8ce7-45fb-9383-02840e509cc8\" (UID: \"658e0853-8ce7-45fb-9383-02840e509cc8\") " Oct 30 04:21:00.705642 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.705496 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c78e806a-f197-4c83-a43f-6306f3a73539-plugins\" (UniqueName: \"kubernetes.io/empty-dir/c78e806a-f197-4c83-a43f-6306f3a73539-plugins\") pod \"c78e806a-f197-4c83-a43f-6306f3a73539\" (UID: \"c78e806a-f197-4c83-a43f-6306f3a73539\") " Oct 30 04:21:00.705642 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.705524 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/658e0853-8ce7-45fb-9383-02840e509cc8-kube-api-access-lr8t5\" (UniqueName: \"kubernetes.io/projected/658e0853-8ce7-45fb-9383-02840e509cc8-kube-api-access-lr8t5\") pod \"658e0853-8ce7-45fb-9383-02840e509cc8\" (UID: \"658e0853-8ce7-45fb-9383-02840e509cc8\") " Oct 30 04:21:00.705785 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.705686 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"c78e806a-f197-4c83-a43f-6306f3a73539\" (UID: \"c78e806a-f197-4c83-a43f-6306f3a73539\") " Oct 30 04:21:00.705785 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.705770 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c78e806a-f197-4c83-a43f-6306f3a73539-kube-api-access-r594v\" (UniqueName: \"kubernetes.io/projected/c78e806a-f197-4c83-a43f-6306f3a73539-kube-api-access-r594v\") pod \"c78e806a-f197-4c83-a43f-6306f3a73539\" (UID: \"c78e806a-f197-4c83-a43f-6306f3a73539\") " Oct 30 04:21:00.705892 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.705805 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c78e806a-f197-4c83-a43f-6306f3a73539-config\" (UniqueName: \"kubernetes.io/configmap/c78e806a-f197-4c83-a43f-6306f3a73539-config\") pod \"c78e806a-f197-4c83-a43f-6306f3a73539\" (UID: \"c78e806a-f197-4c83-a43f-6306f3a73539\") " Oct 30 04:21:00.705892 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.705861 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/658e0853-8ce7-45fb-9383-02840e509cc8-ta-internal\" (UniqueName: \"kubernetes.io/configmap/658e0853-8ce7-45fb-9383-02840e509cc8-ta-internal\") pod \"658e0853-8ce7-45fb-9383-02840e509cc8\" (UID: \"658e0853-8ce7-45fb-9383-02840e509cc8\") " Oct 30 04:21:00.706102 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.706086 5882 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-cluster-tls-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:00.706164 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.706107 5882 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/794280a2-9fc7-425e-8f4b-9a0a25f89031-tls-assets\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:00.706164 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.706123 5882 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/794280a2-9fc7-425e-8f4b-9a0a25f89031-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:00.706164 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.706139 5882 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/794280a2-9fc7-425e-8f4b-9a0a25f89031-config-out\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:00.706164 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.706152 5882 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-web-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:00.706376 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.706164 5882 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/794280a2-9fc7-425e-8f4b-9a0a25f89031-config-volume\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:00.706376 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.706178 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vtdk\" (UniqueName: \"kubernetes.io/projected/794280a2-9fc7-425e-8f4b-9a0a25f89031-kube-api-access-6vtdk\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:00.706553 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.706532 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/658e0853-8ce7-45fb-9383-02840e509cc8-ta-internal" pod "658e0853-8ce7-45fb-9383-02840e509cc8" (UID: "658e0853-8ce7-45fb-9383-02840e509cc8"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:21:00.706604 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.706531 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c78e806a-f197-4c83-a43f-6306f3a73539-config" pod "c78e806a-f197-4c83-a43f-6306f3a73539" (UID: "c78e806a-f197-4c83-a43f-6306f3a73539"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:21:00.707745 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.707721 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/658e0853-8ce7-45fb-9383-02840e509cc8-kube-api-access-lr8t5" pod "658e0853-8ce7-45fb-9383-02840e509cc8" (UID: "658e0853-8ce7-45fb-9383-02840e509cc8"). InnerVolumeSpecName "kube-api-access-lr8t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:00.708206 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.708187 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c78e806a-f197-4c83-a43f-6306f3a73539-kube-api-access-r594v" pod "c78e806a-f197-4c83-a43f-6306f3a73539" (UID: "c78e806a-f197-4c83-a43f-6306f3a73539"). InnerVolumeSpecName "kube-api-access-r594v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:00.708881 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.708862 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e" pod "c78e806a-f197-4c83-a43f-6306f3a73539" (UID: "c78e806a-f197-4c83-a43f-6306f3a73539"). InnerVolumeSpecName "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:21:00.709803 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.709782 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/658e0853-8ce7-45fb-9383-02840e509cc8-data-science-collector-ta-server-cert" pod "658e0853-8ce7-45fb-9383-02840e509cc8" (UID: "658e0853-8ce7-45fb-9383-02840e509cc8"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:00.807437 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.807359 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/658e0853-8ce7-45fb-9383-02840e509cc8-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:00.807437 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.807387 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/658e0853-8ce7-45fb-9383-02840e509cc8-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:00.807437 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.807397 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr8t5\" (UniqueName: \"kubernetes.io/projected/658e0853-8ce7-45fb-9383-02840e509cc8-kube-api-access-lr8t5\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:00.807437 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.807428 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:21:00.807715 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.807442 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r594v\" (UniqueName: \"kubernetes.io/projected/c78e806a-f197-4c83-a43f-6306f3a73539-kube-api-access-r594v\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:00.807715 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.807455 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c78e806a-f197-4c83-a43f-6306f3a73539-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:00.821875 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.821764 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:21:00.864488 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.864450 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c78e806a-f197-4c83-a43f-6306f3a73539-plugins" pod "c78e806a-f197-4c83-a43f-6306f3a73539" (UID: "c78e806a-f197-4c83-a43f-6306f3a73539"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:21:00.908846 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.908814 5882 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/c78e806a-f197-4c83-a43f-6306f3a73539-plugins\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:00.908846 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:00.908843 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" Oct 30 04:21:01.237284 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.237247 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="a582ce4e-0909-4e2e-acb4-956a215630d9" containerName="prometheus" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:21:01.343381 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.343343 5882 generic.go:356] "Generic (PLEG): container finished" podID="658e0853-8ce7-45fb-9383-02840e509cc8" containerID="10131b75e2667ebc43c2a38ed46bf0e5175c3d518981768049f38db0510b616d" exitCode=0 Oct 30 04:21:01.343944 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.343400 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" Oct 30 04:21:01.343944 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.343424 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" event={"ID":"658e0853-8ce7-45fb-9383-02840e509cc8","Type":"ContainerDied","Data":"10131b75e2667ebc43c2a38ed46bf0e5175c3d518981768049f38db0510b616d"} Oct 30 04:21:01.343944 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.343476 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf" event={"ID":"658e0853-8ce7-45fb-9383-02840e509cc8","Type":"ContainerDied","Data":"a75608ab7ecc013dd10ffc8e51dea8cddacc11026e2e7a520e0e83c0f198e86c"} Oct 30 04:21:01.343944 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.343502 5882 scope.go:122] "RemoveContainer" containerID="10131b75e2667ebc43c2a38ed46bf0e5175c3d518981768049f38db0510b616d" Oct 30 04:21:01.348561 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.348086 5882 generic.go:356] "Generic (PLEG): container finished" podID="794280a2-9fc7-425e-8f4b-9a0a25f89031" containerID="886b28f978320fcb5d31bfa97da1e8f6e8fdf79cf028236d4d8c59dd01ab4bda" exitCode=0 Oct 30 04:21:01.348561 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.348260 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:01.348561 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.348399 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"794280a2-9fc7-425e-8f4b-9a0a25f89031","Type":"ContainerDied","Data":"886b28f978320fcb5d31bfa97da1e8f6e8fdf79cf028236d4d8c59dd01ab4bda"} Oct 30 04:21:01.348561 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.348474 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"794280a2-9fc7-425e-8f4b-9a0a25f89031","Type":"ContainerDied","Data":"4d88e354a718a42e1f547b920ebbf2c720901593b62f7d147ebeb36a2f332972"} Oct 30 04:21:01.351871 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.351842 5882 generic.go:356] "Generic (PLEG): container finished" podID="c78e806a-f197-4c83-a43f-6306f3a73539" containerID="c5d9bf9ed6f0789e23fbfafb388d4109eed1015e5058c1491d461585326d9c3d" exitCode=0 Oct 30 04:21:01.352000 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.351898 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"c78e806a-f197-4c83-a43f-6306f3a73539","Type":"ContainerDied","Data":"c5d9bf9ed6f0789e23fbfafb388d4109eed1015e5058c1491d461585326d9c3d"} Oct 30 04:21:01.352000 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.351926 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"c78e806a-f197-4c83-a43f-6306f3a73539","Type":"ContainerDied","Data":"2e6c721448e646ac10d402545d088aafd98fb64837a78de9073f53c06e4fbbb7"} Oct 30 04:21:01.352000 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.351987 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:01.371129 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.371111 5882 scope.go:122] "RemoveContainer" containerID="10131b75e2667ebc43c2a38ed46bf0e5175c3d518981768049f38db0510b616d" Oct 30 04:21:01.371427 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:01.371407 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10131b75e2667ebc43c2a38ed46bf0e5175c3d518981768049f38db0510b616d\": container with ID starting with 10131b75e2667ebc43c2a38ed46bf0e5175c3d518981768049f38db0510b616d not found: ID does not exist" containerID="10131b75e2667ebc43c2a38ed46bf0e5175c3d518981768049f38db0510b616d" Oct 30 04:21:01.371505 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.371434 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10131b75e2667ebc43c2a38ed46bf0e5175c3d518981768049f38db0510b616d"} err="failed to get container status \"10131b75e2667ebc43c2a38ed46bf0e5175c3d518981768049f38db0510b616d\": rpc error: code = NotFound desc = could not find container \"10131b75e2667ebc43c2a38ed46bf0e5175c3d518981768049f38db0510b616d\": container with ID starting with 10131b75e2667ebc43c2a38ed46bf0e5175c3d518981768049f38db0510b616d not found: ID does not exist" Oct 30 04:21:01.371505 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.371450 5882 scope.go:122] "RemoveContainer" containerID="2d7e07c5e5c2385e472afcddfbeb7f8a9554eb3cf543191decb05a01a85037c2" Oct 30 04:21:01.381742 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.381720 5882 scope.go:122] "RemoveContainer" containerID="886b28f978320fcb5d31bfa97da1e8f6e8fdf79cf028236d4d8c59dd01ab4bda" Oct 30 04:21:01.390862 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.390841 5882 scope.go:122] "RemoveContainer" containerID="75f588b157b6fa4a3abb78c3b3aa13288ab91ae342ef5f242bcc4164b9b1892c" Oct 30 04:21:01.399824 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.399798 5882 scope.go:122] "RemoveContainer" containerID="2d7e07c5e5c2385e472afcddfbeb7f8a9554eb3cf543191decb05a01a85037c2" Oct 30 04:21:01.400166 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:01.400133 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d7e07c5e5c2385e472afcddfbeb7f8a9554eb3cf543191decb05a01a85037c2\": container with ID starting with 2d7e07c5e5c2385e472afcddfbeb7f8a9554eb3cf543191decb05a01a85037c2 not found: ID does not exist" containerID="2d7e07c5e5c2385e472afcddfbeb7f8a9554eb3cf543191decb05a01a85037c2" Oct 30 04:21:01.400264 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.400175 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d7e07c5e5c2385e472afcddfbeb7f8a9554eb3cf543191decb05a01a85037c2"} err="failed to get container status \"2d7e07c5e5c2385e472afcddfbeb7f8a9554eb3cf543191decb05a01a85037c2\": rpc error: code = NotFound desc = could not find container \"2d7e07c5e5c2385e472afcddfbeb7f8a9554eb3cf543191decb05a01a85037c2\": container with ID starting with 2d7e07c5e5c2385e472afcddfbeb7f8a9554eb3cf543191decb05a01a85037c2 not found: ID does not exist" Oct 30 04:21:01.400264 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.400198 5882 scope.go:122] "RemoveContainer" containerID="886b28f978320fcb5d31bfa97da1e8f6e8fdf79cf028236d4d8c59dd01ab4bda" Oct 30 04:21:01.400514 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:01.400487 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"886b28f978320fcb5d31bfa97da1e8f6e8fdf79cf028236d4d8c59dd01ab4bda\": container with ID starting with 886b28f978320fcb5d31bfa97da1e8f6e8fdf79cf028236d4d8c59dd01ab4bda not found: ID does not exist" containerID="886b28f978320fcb5d31bfa97da1e8f6e8fdf79cf028236d4d8c59dd01ab4bda" Oct 30 04:21:01.400568 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.400523 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"886b28f978320fcb5d31bfa97da1e8f6e8fdf79cf028236d4d8c59dd01ab4bda"} err="failed to get container status \"886b28f978320fcb5d31bfa97da1e8f6e8fdf79cf028236d4d8c59dd01ab4bda\": rpc error: code = NotFound desc = could not find container \"886b28f978320fcb5d31bfa97da1e8f6e8fdf79cf028236d4d8c59dd01ab4bda\": container with ID starting with 886b28f978320fcb5d31bfa97da1e8f6e8fdf79cf028236d4d8c59dd01ab4bda not found: ID does not exist" Oct 30 04:21:01.400568 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.400543 5882 scope.go:122] "RemoveContainer" containerID="75f588b157b6fa4a3abb78c3b3aa13288ab91ae342ef5f242bcc4164b9b1892c" Oct 30 04:21:01.400880 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:01.400860 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75f588b157b6fa4a3abb78c3b3aa13288ab91ae342ef5f242bcc4164b9b1892c\": container with ID starting with 75f588b157b6fa4a3abb78c3b3aa13288ab91ae342ef5f242bcc4164b9b1892c not found: ID does not exist" containerID="75f588b157b6fa4a3abb78c3b3aa13288ab91ae342ef5f242bcc4164b9b1892c" Oct 30 04:21:01.400960 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.400885 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f588b157b6fa4a3abb78c3b3aa13288ab91ae342ef5f242bcc4164b9b1892c"} err="failed to get container status \"75f588b157b6fa4a3abb78c3b3aa13288ab91ae342ef5f242bcc4164b9b1892c\": rpc error: code = NotFound desc = could not find container \"75f588b157b6fa4a3abb78c3b3aa13288ab91ae342ef5f242bcc4164b9b1892c\": container with ID starting with 75f588b157b6fa4a3abb78c3b3aa13288ab91ae342ef5f242bcc4164b9b1892c not found: ID does not exist" Oct 30 04:21:01.400960 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.400904 5882 scope.go:122] "RemoveContainer" containerID="c5d9bf9ed6f0789e23fbfafb388d4109eed1015e5058c1491d461585326d9c3d" Oct 30 04:21:01.410296 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.410275 5882 scope.go:122] "RemoveContainer" containerID="c5d9bf9ed6f0789e23fbfafb388d4109eed1015e5058c1491d461585326d9c3d" Oct 30 04:21:01.410595 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:01.410573 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5d9bf9ed6f0789e23fbfafb388d4109eed1015e5058c1491d461585326d9c3d\": container with ID starting with c5d9bf9ed6f0789e23fbfafb388d4109eed1015e5058c1491d461585326d9c3d not found: ID does not exist" containerID="c5d9bf9ed6f0789e23fbfafb388d4109eed1015e5058c1491d461585326d9c3d" Oct 30 04:21:01.410682 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.410603 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5d9bf9ed6f0789e23fbfafb388d4109eed1015e5058c1491d461585326d9c3d"} err="failed to get container status \"c5d9bf9ed6f0789e23fbfafb388d4109eed1015e5058c1491d461585326d9c3d\": rpc error: code = NotFound desc = could not find container \"c5d9bf9ed6f0789e23fbfafb388d4109eed1015e5058c1491d461585326d9c3d\": container with ID starting with c5d9bf9ed6f0789e23fbfafb388d4109eed1015e5058c1491d461585326d9c3d not found: ID does not exist" Oct 30 04:21:01.433595 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.433570 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf"] Oct 30 04:21:01.438409 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.438384 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b8fbb66ff-qzdmf"] Oct 30 04:21:01.475694 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.475656 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:21:01.479866 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.479842 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:21:01.526739 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.526713 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:21:01.531199 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:01.531177 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:21:02.017246 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:02.017204 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="612f19e4-0114-4d9d-b23b-48fc92b83dd0" path="/var/lib/kubelet/pods/612f19e4-0114-4d9d-b23b-48fc92b83dd0/volumes" Oct 30 04:21:02.017782 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:02.017763 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="658e0853-8ce7-45fb-9383-02840e509cc8" path="/var/lib/kubelet/pods/658e0853-8ce7-45fb-9383-02840e509cc8/volumes" Oct 30 04:21:02.018231 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:02.018213 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="794280a2-9fc7-425e-8f4b-9a0a25f89031" path="/var/lib/kubelet/pods/794280a2-9fc7-425e-8f4b-9a0a25f89031/volumes" Oct 30 04:21:02.019276 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:02.019257 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c78e806a-f197-4c83-a43f-6306f3a73539" path="/var/lib/kubelet/pods/c78e806a-f197-4c83-a43f-6306f3a73539/volumes" Oct 30 04:21:05.911423 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.911398 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:05.953074 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.953050 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a582ce4e-0909-4e2e-acb4-956a215630d9-tls-assets\" (UniqueName: \"kubernetes.io/projected/a582ce4e-0909-4e2e-acb4-956a215630d9-tls-assets\") pod \"a582ce4e-0909-4e2e-acb4-956a215630d9\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " Oct 30 04:21:05.953207 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.953085 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-config\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-config\") pod \"a582ce4e-0909-4e2e-acb4-956a215630d9\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " Oct 30 04:21:05.953207 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.953106 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"a582ce4e-0909-4e2e-acb4-956a215630d9\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " Oct 30 04:21:05.953207 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.953127 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-thanos-prometheus-http-client-file\") pod \"a582ce4e-0909-4e2e-acb4-956a215630d9\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " Oct 30 04:21:05.953207 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.953154 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config\") pod \"a582ce4e-0909-4e2e-acb4-956a215630d9\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " Oct 30 04:21:05.953207 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.953195 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a582ce4e-0909-4e2e-acb4-956a215630d9-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a582ce4e-0909-4e2e-acb4-956a215630d9-config-out\") pod \"a582ce4e-0909-4e2e-acb4-956a215630d9\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " Oct 30 04:21:05.953492 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.953232 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"a582ce4e-0909-4e2e-acb4-956a215630d9\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " Oct 30 04:21:05.953492 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.953269 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"a582ce4e-0909-4e2e-acb4-956a215630d9\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " Oct 30 04:21:05.953492 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.953294 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-secret-prometheus-web-tls-ca\") pod \"a582ce4e-0909-4e2e-acb4-956a215630d9\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " Oct 30 04:21:05.953492 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.953384 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"a582ce4e-0909-4e2e-acb4-956a215630d9\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " Oct 30 04:21:05.953492 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.953439 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"a582ce4e-0909-4e2e-acb4-956a215630d9\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " Oct 30 04:21:05.953788 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.953503 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a582ce4e-0909-4e2e-acb4-956a215630d9-kube-api-access-qxlbq\" (UniqueName: \"kubernetes.io/projected/a582ce4e-0909-4e2e-acb4-956a215630d9-kube-api-access-qxlbq\") pod \"a582ce4e-0909-4e2e-acb4-956a215630d9\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " Oct 30 04:21:05.953788 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.953582 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-0" pod "a582ce4e-0909-4e2e-acb4-956a215630d9" (UID: "a582ce4e-0909-4e2e-acb4-956a215630d9"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:21:05.954866 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.954309 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"a582ce4e-0909-4e2e-acb4-956a215630d9\" (UID: \"a582ce4e-0909-4e2e-acb4-956a215630d9\") " Oct 30 04:21:05.954866 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.954565 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-1" pod "a582ce4e-0909-4e2e-acb4-956a215630d9" (UID: "a582ce4e-0909-4e2e-acb4-956a215630d9"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:21:05.954866 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.954584 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-2" pod "a582ce4e-0909-4e2e-acb4-956a215630d9" (UID: "a582ce4e-0909-4e2e-acb4-956a215630d9"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:21:05.954866 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.954801 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:05.954866 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.954824 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:05.954866 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.954840 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a582ce4e-0909-4e2e-acb4-956a215630d9-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:05.956204 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.956168 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-config" pod "a582ce4e-0909-4e2e-acb4-956a215630d9" (UID: "a582ce4e-0909-4e2e-acb4-956a215630d9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:05.956472 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.956453 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a582ce4e-0909-4e2e-acb4-956a215630d9-kube-api-access-qxlbq" pod "a582ce4e-0909-4e2e-acb4-956a215630d9" (UID: "a582ce4e-0909-4e2e-acb4-956a215630d9"). InnerVolumeSpecName "kube-api-access-qxlbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:05.957168 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.957145 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a582ce4e-0909-4e2e-acb4-956a215630d9-tls-assets" pod "a582ce4e-0909-4e2e-acb4-956a215630d9" (UID: "a582ce4e-0909-4e2e-acb4-956a215630d9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:05.957312 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.957159 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-secret-prometheus-web-tls-ca" pod "a582ce4e-0909-4e2e-acb4-956a215630d9" (UID: "a582ce4e-0909-4e2e-acb4-956a215630d9"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:05.957312 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.957206 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-thanos-prometheus-http-client-file" pod "a582ce4e-0909-4e2e-acb4-956a215630d9" (UID: "a582ce4e-0909-4e2e-acb4-956a215630d9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:05.957312 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.957260 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a582ce4e-0909-4e2e-acb4-956a215630d9-config-out" pod "a582ce4e-0909-4e2e-acb4-956a215630d9" (UID: "a582ce4e-0909-4e2e-acb4-956a215630d9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:21:05.957521 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.957496 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "a582ce4e-0909-4e2e-acb4-956a215630d9" (UID: "a582ce4e-0909-4e2e-acb4-956a215630d9"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:05.958867 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.958839 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "a582ce4e-0909-4e2e-acb4-956a215630d9" (UID: "a582ce4e-0909-4e2e-acb4-956a215630d9"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:05.959085 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.959067 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4" pod "a582ce4e-0909-4e2e-acb4-956a215630d9" (UID: "a582ce4e-0909-4e2e-acb4-956a215630d9"). InnerVolumeSpecName "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:21:05.968419 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:05.968394 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config" pod "a582ce4e-0909-4e2e-acb4-956a215630d9" (UID: "a582ce4e-0909-4e2e-acb4-956a215630d9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:06.055770 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.055742 5882 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a582ce4e-0909-4e2e-acb4-956a215630d9-config-out\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:06.055770 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.055769 5882 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:06.055957 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.055782 5882 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-secret-prometheus-web-tls-ca\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:06.055957 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.055818 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:21:06.055957 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.055828 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxlbq\" (UniqueName: \"kubernetes.io/projected/a582ce4e-0909-4e2e-acb4-956a215630d9-kube-api-access-qxlbq\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:06.055957 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.055837 5882 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:06.055957 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.055847 5882 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a582ce4e-0909-4e2e-acb4-956a215630d9-tls-assets\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:06.055957 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.055856 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:06.055957 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.055864 5882 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:06.055957 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.055877 5882 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a582ce4e-0909-4e2e-acb4-956a215630d9-web-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:06.066883 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.066863 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:21:06.156316 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.156276 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-927d18ef2eb99137c68fbe5e0bb8ea629bb5219ed56cdede72da7c0f045be95e\"" Oct 30 04:21:06.379343 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.379252 5882 generic.go:356] "Generic (PLEG): container finished" podID="a582ce4e-0909-4e2e-acb4-956a215630d9" containerID="c556a7a6ff399ba1d13edd11ad1b563438475489c37058c77b951ae8669990f0" exitCode=0 Oct 30 04:21:06.379343 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.379323 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a582ce4e-0909-4e2e-acb4-956a215630d9","Type":"ContainerDied","Data":"c556a7a6ff399ba1d13edd11ad1b563438475489c37058c77b951ae8669990f0"} Oct 30 04:21:06.379727 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.379342 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:06.379727 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.379375 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a582ce4e-0909-4e2e-acb4-956a215630d9","Type":"ContainerDied","Data":"a02b6deb384b099aa486d9c0538db187acee16f8853f0d704d320d7cacd67db0"} Oct 30 04:21:06.379727 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.379404 5882 scope.go:122] "RemoveContainer" containerID="ae9fd1f4dff510f477128b002f999c2a1093eca6bef3b5bf7f730577b60fc1d6" Oct 30 04:21:06.387326 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.387304 5882 scope.go:122] "RemoveContainer" containerID="a9d2b0ea4664f875c8a9931ed21b6665cafbe285cf3203ea536c0e4ffbd617ec" Oct 30 04:21:06.394987 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.394969 5882 scope.go:122] "RemoveContainer" containerID="c556a7a6ff399ba1d13edd11ad1b563438475489c37058c77b951ae8669990f0" Oct 30 04:21:06.402737 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.402722 5882 scope.go:122] "RemoveContainer" containerID="516671a81e2dd8f2a80f5ace91442941cfc76e2475a71c89e495c70b25a9a434" Oct 30 04:21:06.410314 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.410299 5882 scope.go:122] "RemoveContainer" containerID="ae9fd1f4dff510f477128b002f999c2a1093eca6bef3b5bf7f730577b60fc1d6" Oct 30 04:21:06.410574 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:06.410554 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9fd1f4dff510f477128b002f999c2a1093eca6bef3b5bf7f730577b60fc1d6\": container with ID starting with ae9fd1f4dff510f477128b002f999c2a1093eca6bef3b5bf7f730577b60fc1d6 not found: ID does not exist" containerID="ae9fd1f4dff510f477128b002f999c2a1093eca6bef3b5bf7f730577b60fc1d6" Oct 30 04:21:06.410630 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.410585 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae9fd1f4dff510f477128b002f999c2a1093eca6bef3b5bf7f730577b60fc1d6"} err="failed to get container status \"ae9fd1f4dff510f477128b002f999c2a1093eca6bef3b5bf7f730577b60fc1d6\": rpc error: code = NotFound desc = could not find container \"ae9fd1f4dff510f477128b002f999c2a1093eca6bef3b5bf7f730577b60fc1d6\": container with ID starting with ae9fd1f4dff510f477128b002f999c2a1093eca6bef3b5bf7f730577b60fc1d6 not found: ID does not exist" Oct 30 04:21:06.410630 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.410602 5882 scope.go:122] "RemoveContainer" containerID="a9d2b0ea4664f875c8a9931ed21b6665cafbe285cf3203ea536c0e4ffbd617ec" Oct 30 04:21:06.410879 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:06.410860 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d2b0ea4664f875c8a9931ed21b6665cafbe285cf3203ea536c0e4ffbd617ec\": container with ID starting with a9d2b0ea4664f875c8a9931ed21b6665cafbe285cf3203ea536c0e4ffbd617ec not found: ID does not exist" containerID="a9d2b0ea4664f875c8a9931ed21b6665cafbe285cf3203ea536c0e4ffbd617ec" Oct 30 04:21:06.410942 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.410884 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d2b0ea4664f875c8a9931ed21b6665cafbe285cf3203ea536c0e4ffbd617ec"} err="failed to get container status \"a9d2b0ea4664f875c8a9931ed21b6665cafbe285cf3203ea536c0e4ffbd617ec\": rpc error: code = NotFound desc = could not find container \"a9d2b0ea4664f875c8a9931ed21b6665cafbe285cf3203ea536c0e4ffbd617ec\": container with ID starting with a9d2b0ea4664f875c8a9931ed21b6665cafbe285cf3203ea536c0e4ffbd617ec not found: ID does not exist" Oct 30 04:21:06.410942 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.410908 5882 scope.go:122] "RemoveContainer" containerID="c556a7a6ff399ba1d13edd11ad1b563438475489c37058c77b951ae8669990f0" Oct 30 04:21:06.411133 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:06.411115 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c556a7a6ff399ba1d13edd11ad1b563438475489c37058c77b951ae8669990f0\": container with ID starting with c556a7a6ff399ba1d13edd11ad1b563438475489c37058c77b951ae8669990f0 not found: ID does not exist" containerID="c556a7a6ff399ba1d13edd11ad1b563438475489c37058c77b951ae8669990f0" Oct 30 04:21:06.411198 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.411141 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c556a7a6ff399ba1d13edd11ad1b563438475489c37058c77b951ae8669990f0"} err="failed to get container status \"c556a7a6ff399ba1d13edd11ad1b563438475489c37058c77b951ae8669990f0\": rpc error: code = NotFound desc = could not find container \"c556a7a6ff399ba1d13edd11ad1b563438475489c37058c77b951ae8669990f0\": container with ID starting with c556a7a6ff399ba1d13edd11ad1b563438475489c37058c77b951ae8669990f0 not found: ID does not exist" Oct 30 04:21:06.411198 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.411162 5882 scope.go:122] "RemoveContainer" containerID="516671a81e2dd8f2a80f5ace91442941cfc76e2475a71c89e495c70b25a9a434" Oct 30 04:21:06.411397 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:06.411378 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516671a81e2dd8f2a80f5ace91442941cfc76e2475a71c89e495c70b25a9a434\": container with ID starting with 516671a81e2dd8f2a80f5ace91442941cfc76e2475a71c89e495c70b25a9a434 not found: ID does not exist" containerID="516671a81e2dd8f2a80f5ace91442941cfc76e2475a71c89e495c70b25a9a434" Oct 30 04:21:06.411436 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.411402 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516671a81e2dd8f2a80f5ace91442941cfc76e2475a71c89e495c70b25a9a434"} err="failed to get container status \"516671a81e2dd8f2a80f5ace91442941cfc76e2475a71c89e495c70b25a9a434\": rpc error: code = NotFound desc = could not find container \"516671a81e2dd8f2a80f5ace91442941cfc76e2475a71c89e495c70b25a9a434\": container with ID starting with 516671a81e2dd8f2a80f5ace91442941cfc76e2475a71c89e495c70b25a9a434 not found: ID does not exist" Oct 30 04:21:06.438431 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.438399 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:21:06.452502 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:06.452470 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:21:08.009228 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:08.009191 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a582ce4e-0909-4e2e-acb4-956a215630d9" path="/var/lib/kubelet/pods/a582ce4e-0909-4e2e-acb4-956a215630d9/volumes" Oct 30 04:21:12.592071 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592037 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx"] Oct 30 04:21:12.592467 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592365 5882 cpu_manager.go:414] "containerMap: removing container" podUID="a582ce4e-0909-4e2e-acb4-956a215630d9" containerName="config-reloader" Oct 30 04:21:12.592467 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592375 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a582ce4e-0909-4e2e-acb4-956a215630d9" containerName="config-reloader" Oct 30 04:21:12.592467 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592386 5882 cpu_manager.go:414] "containerMap: removing container" podUID="612f19e4-0114-4d9d-b23b-48fc92b83dd0" containerName="prom-label-proxy" Oct 30 04:21:12.592467 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592391 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="612f19e4-0114-4d9d-b23b-48fc92b83dd0" containerName="prom-label-proxy" Oct 30 04:21:12.592467 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592402 5882 cpu_manager.go:414] "containerMap: removing container" podUID="794280a2-9fc7-425e-8f4b-9a0a25f89031" containerName="alertmanager" Oct 30 04:21:12.592467 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592407 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794280a2-9fc7-425e-8f4b-9a0a25f89031" containerName="alertmanager" Oct 30 04:21:12.592467 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592418 5882 cpu_manager.go:414] "containerMap: removing container" podUID="658e0853-8ce7-45fb-9383-02840e509cc8" containerName="ta-container" Oct 30 04:21:12.592467 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592423 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="658e0853-8ce7-45fb-9383-02840e509cc8" containerName="ta-container" Oct 30 04:21:12.592467 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592431 5882 cpu_manager.go:414] "containerMap: removing container" podUID="794280a2-9fc7-425e-8f4b-9a0a25f89031" containerName="config-reloader" Oct 30 04:21:12.592467 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592435 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794280a2-9fc7-425e-8f4b-9a0a25f89031" containerName="config-reloader" Oct 30 04:21:12.592467 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592444 5882 cpu_manager.go:414] "containerMap: removing container" podUID="c78e806a-f197-4c83-a43f-6306f3a73539" containerName="perses" Oct 30 04:21:12.592467 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592448 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c78e806a-f197-4c83-a43f-6306f3a73539" containerName="perses" Oct 30 04:21:12.592467 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592454 5882 cpu_manager.go:414] "containerMap: removing container" podUID="a582ce4e-0909-4e2e-acb4-956a215630d9" containerName="prometheus" Oct 30 04:21:12.592467 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592458 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a582ce4e-0909-4e2e-acb4-956a215630d9" containerName="prometheus" Oct 30 04:21:12.592921 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592479 5882 cpu_manager.go:414] "containerMap: removing container" podUID="612f19e4-0114-4d9d-b23b-48fc92b83dd0" containerName="kube-rbac-proxy" Oct 30 04:21:12.592921 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592484 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="612f19e4-0114-4d9d-b23b-48fc92b83dd0" containerName="kube-rbac-proxy" Oct 30 04:21:12.592921 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592492 5882 cpu_manager.go:414] "containerMap: removing container" podUID="794280a2-9fc7-425e-8f4b-9a0a25f89031" containerName="init-config-reloader" Oct 30 04:21:12.592921 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592496 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794280a2-9fc7-425e-8f4b-9a0a25f89031" containerName="init-config-reloader" Oct 30 04:21:12.592921 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592502 5882 cpu_manager.go:414] "containerMap: removing container" podUID="a582ce4e-0909-4e2e-acb4-956a215630d9" containerName="init-config-reloader" Oct 30 04:21:12.592921 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592507 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a582ce4e-0909-4e2e-acb4-956a215630d9" containerName="init-config-reloader" Oct 30 04:21:12.592921 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592513 5882 cpu_manager.go:414] "containerMap: removing container" podUID="a582ce4e-0909-4e2e-acb4-956a215630d9" containerName="thanos-sidecar" Oct 30 04:21:12.592921 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592517 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a582ce4e-0909-4e2e-acb4-956a215630d9" containerName="thanos-sidecar" Oct 30 04:21:12.592921 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592580 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="794280a2-9fc7-425e-8f4b-9a0a25f89031" containerName="alertmanager" Oct 30 04:21:12.592921 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592589 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="658e0853-8ce7-45fb-9383-02840e509cc8" containerName="ta-container" Oct 30 04:21:12.592921 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592595 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="794280a2-9fc7-425e-8f4b-9a0a25f89031" containerName="config-reloader" Oct 30 04:21:12.592921 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592600 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c78e806a-f197-4c83-a43f-6306f3a73539" containerName="perses" Oct 30 04:21:12.592921 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592605 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a582ce4e-0909-4e2e-acb4-956a215630d9" containerName="prometheus" Oct 30 04:21:12.592921 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592629 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="612f19e4-0114-4d9d-b23b-48fc92b83dd0" containerName="kube-rbac-proxy" Oct 30 04:21:12.592921 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592636 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a582ce4e-0909-4e2e-acb4-956a215630d9" containerName="thanos-sidecar" Oct 30 04:21:12.592921 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592642 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a582ce4e-0909-4e2e-acb4-956a215630d9" containerName="config-reloader" Oct 30 04:21:12.592921 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.592647 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="612f19e4-0114-4d9d-b23b-48fc92b83dd0" containerName="prom-label-proxy" Oct 30 04:21:12.597604 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.597586 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx" Oct 30 04:21:12.611734 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.611705 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx"] Oct 30 04:21:12.625253 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.625231 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 04:21:12.625253 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.625236 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\"" Oct 30 04:21:12.628180 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.628164 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 04:21:12.712559 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.712525 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txsg9\" (UniqueName: \"kubernetes.io/projected/134b06f4-cbbe-45e4-8ee4-3abc80ff9f19-kube-api-access-txsg9\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx\" (UID: \"134b06f4-cbbe-45e4-8ee4-3abc80ff9f19\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx" Oct 30 04:21:12.813946 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.813911 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txsg9\" (UniqueName: \"kubernetes.io/projected/134b06f4-cbbe-45e4-8ee4-3abc80ff9f19-kube-api-access-txsg9\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx\" (UID: \"134b06f4-cbbe-45e4-8ee4-3abc80ff9f19\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx" Oct 30 04:21:12.839069 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.839040 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txsg9\" (UniqueName: \"kubernetes.io/projected/134b06f4-cbbe-45e4-8ee4-3abc80ff9f19-kube-api-access-txsg9\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx\" (UID: \"134b06f4-cbbe-45e4-8ee4-3abc80ff9f19\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx" Oct 30 04:21:12.917754 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:12.917669 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx" Oct 30 04:21:13.208637 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.208598 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx"] Oct 30 04:21:13.210346 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:21:13.210315 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod134b06f4_cbbe_45e4_8ee4_3abc80ff9f19.slice/crio-48ed242be269e936c2e93a4c11ff6260fd8dc9c2c5867a4636afbae412756e13 WatchSource:0}: Error finding container 48ed242be269e936c2e93a4c11ff6260fd8dc9c2c5867a4636afbae412756e13: Status 404 returned error can't find the container with id 48ed242be269e936c2e93a4c11ff6260fd8dc9c2c5867a4636afbae412756e13 Oct 30 04:21:13.412945 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.412909 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx" event={"ID":"134b06f4-cbbe-45e4-8ee4-3abc80ff9f19","Type":"ContainerStarted","Data":"450643912acea109348c2f1cbe0cb6685578902bf45f68bfe8dfea2f5ae14a05"} Oct 30 04:21:13.413131 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.412954 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx" event={"ID":"134b06f4-cbbe-45e4-8ee4-3abc80ff9f19","Type":"ContainerStarted","Data":"48ed242be269e936c2e93a4c11ff6260fd8dc9c2c5867a4636afbae412756e13"} Oct 30 04:21:13.458523 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.458430 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx" podStartSLOduration=1.458416153 podStartE2EDuration="1.458416153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:13.458235804 +0000 UTC m=+5985.572550778" watchObservedRunningTime="2026-10-30 04:21:13.458416153 +0000 UTC m=+5985.572731124" Oct 30 04:21:13.574692 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.574653 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:21:13.578235 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.578211 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:13.597498 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.597469 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:21:13.603497 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.603451 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:21:13.611589 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.611565 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:21:13.614831 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.614816 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:13.619908 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.619882 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e2644fd9-640e-4c3b-b614-c876383484bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:13.620008 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.619967 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-589fg\" (UniqueName: \"kubernetes.io/projected/e2644fd9-640e-4c3b-b614-c876383484bc-kube-api-access-589fg\") pod \"data-science-collector-collector-1\" (UID: \"e2644fd9-640e-4c3b-b614-c876383484bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:13.620052 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.620007 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e2644fd9-640e-4c3b-b614-c876383484bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:13.620103 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.620060 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e2644fd9-640e-4c3b-b614-c876383484bc-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e2644fd9-640e-4c3b-b614-c876383484bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:13.620185 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.620168 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e2644fd9-640e-4c3b-b614-c876383484bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:13.633329 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.633303 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:21:13.665272 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.665234 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-kfs49\"" Oct 30 04:21:13.665438 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.665302 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:21:13.665438 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.665310 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 30 04:21:13.665438 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.665354 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:21:13.665438 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.665430 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:21:13.672752 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.672727 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l"] Oct 30 04:21:13.676228 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.676210 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" Oct 30 04:21:13.694443 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.694418 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l"] Oct 30 04:21:13.721208 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.721138 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e2644fd9-640e-4c3b-b614-c876383484bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:13.721208 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.721191 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/77ca7916-1d2b-4021-a5a0-52ec381c3531-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7d5ccc87d9-pmx4l\" (UID: \"77ca7916-1d2b-4021-a5a0-52ec381c3531\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" Oct 30 04:21:13.721421 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.721222 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/32ce6467-5498-4e99-9fc8-e0dc498b148e-config\") pod \"data-science-perses-0\" (UID: \"32ce6467-5498-4e99-9fc8-e0dc498b148e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:13.721421 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.721248 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/77ca7916-1d2b-4021-a5a0-52ec381c3531-ta-internal\") pod \"data-science-collector-targetallocator-7d5ccc87d9-pmx4l\" (UID: \"77ca7916-1d2b-4021-a5a0-52ec381c3531\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" Oct 30 04:21:13.721421 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.721294 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e2644fd9-640e-4c3b-b614-c876383484bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:13.721421 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.721394 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-589fg\" (UniqueName: \"kubernetes.io/projected/e2644fd9-640e-4c3b-b614-c876383484bc-kube-api-access-589fg\") pod \"data-science-collector-collector-1\" (UID: \"e2644fd9-640e-4c3b-b614-c876383484bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:13.721725 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.721436 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e2644fd9-640e-4c3b-b614-c876383484bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:13.721725 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.721474 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/32ce6467-5498-4e99-9fc8-e0dc498b148e-plugins\") pod \"data-science-perses-0\" (UID: \"32ce6467-5498-4e99-9fc8-e0dc498b148e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:13.721725 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.721529 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwp45\" (UniqueName: \"kubernetes.io/projected/77ca7916-1d2b-4021-a5a0-52ec381c3531-kube-api-access-fwp45\") pod \"data-science-collector-targetallocator-7d5ccc87d9-pmx4l\" (UID: \"77ca7916-1d2b-4021-a5a0-52ec381c3531\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" Oct 30 04:21:13.721725 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.721558 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e2644fd9-640e-4c3b-b614-c876383484bc-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e2644fd9-640e-4c3b-b614-c876383484bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:13.721725 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.721705 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58s29\" (UniqueName: \"kubernetes.io/projected/32ce6467-5498-4e99-9fc8-e0dc498b148e-kube-api-access-58s29\") pod \"data-science-perses-0\" (UID: \"32ce6467-5498-4e99-9fc8-e0dc498b148e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:13.722399 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.722366 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e2644fd9-640e-4c3b-b614-c876383484bc-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e2644fd9-640e-4c3b-b614-c876383484bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:13.723928 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.723911 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e2644fd9-640e-4c3b-b614-c876383484bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:13.724010 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.723976 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e2644fd9-640e-4c3b-b614-c876383484bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:13.724058 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.724014 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e2644fd9-640e-4c3b-b614-c876383484bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:13.746083 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.746057 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-589fg\" (UniqueName: \"kubernetes.io/projected/e2644fd9-640e-4c3b-b614-c876383484bc-kube-api-access-589fg\") pod \"data-science-collector-collector-1\" (UID: \"e2644fd9-640e-4c3b-b614-c876383484bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:13.765195 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.765168 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-7dzbb\"" Oct 30 04:21:13.765291 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.765260 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 30 04:21:13.765336 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.765288 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 30 04:21:13.822467 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.822439 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/77ca7916-1d2b-4021-a5a0-52ec381c3531-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7d5ccc87d9-pmx4l\" (UID: \"77ca7916-1d2b-4021-a5a0-52ec381c3531\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" Oct 30 04:21:13.822612 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.822473 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/32ce6467-5498-4e99-9fc8-e0dc498b148e-config\") pod \"data-science-perses-0\" (UID: \"32ce6467-5498-4e99-9fc8-e0dc498b148e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:13.822612 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.822499 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/77ca7916-1d2b-4021-a5a0-52ec381c3531-ta-internal\") pod \"data-science-collector-targetallocator-7d5ccc87d9-pmx4l\" (UID: \"77ca7916-1d2b-4021-a5a0-52ec381c3531\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" Oct 30 04:21:13.822612 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.822550 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/32ce6467-5498-4e99-9fc8-e0dc498b148e-plugins\") pod \"data-science-perses-0\" (UID: \"32ce6467-5498-4e99-9fc8-e0dc498b148e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:13.822612 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.822596 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwp45\" (UniqueName: \"kubernetes.io/projected/77ca7916-1d2b-4021-a5a0-52ec381c3531-kube-api-access-fwp45\") pod \"data-science-collector-targetallocator-7d5ccc87d9-pmx4l\" (UID: \"77ca7916-1d2b-4021-a5a0-52ec381c3531\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" Oct 30 04:21:13.822877 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.822650 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58s29\" (UniqueName: \"kubernetes.io/projected/32ce6467-5498-4e99-9fc8-e0dc498b148e-kube-api-access-58s29\") pod \"data-science-perses-0\" (UID: \"32ce6467-5498-4e99-9fc8-e0dc498b148e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:13.823229 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.823199 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/32ce6467-5498-4e99-9fc8-e0dc498b148e-plugins\") pod \"data-science-perses-0\" (UID: \"32ce6467-5498-4e99-9fc8-e0dc498b148e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:13.823325 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.823249 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/32ce6467-5498-4e99-9fc8-e0dc498b148e-config\") pod \"data-science-perses-0\" (UID: \"32ce6467-5498-4e99-9fc8-e0dc498b148e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:13.823376 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.823339 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/77ca7916-1d2b-4021-a5a0-52ec381c3531-ta-internal\") pod \"data-science-collector-targetallocator-7d5ccc87d9-pmx4l\" (UID: \"77ca7916-1d2b-4021-a5a0-52ec381c3531\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" Oct 30 04:21:13.825253 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.825230 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/77ca7916-1d2b-4021-a5a0-52ec381c3531-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7d5ccc87d9-pmx4l\" (UID: \"77ca7916-1d2b-4021-a5a0-52ec381c3531\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" Oct 30 04:21:13.848340 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.848313 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwp45\" (UniqueName: \"kubernetes.io/projected/77ca7916-1d2b-4021-a5a0-52ec381c3531-kube-api-access-fwp45\") pod \"data-science-collector-targetallocator-7d5ccc87d9-pmx4l\" (UID: \"77ca7916-1d2b-4021-a5a0-52ec381c3531\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" Oct 30 04:21:13.848443 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.848413 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58s29\" (UniqueName: \"kubernetes.io/projected/32ce6467-5498-4e99-9fc8-e0dc498b148e-kube-api-access-58s29\") pod \"data-science-perses-0\" (UID: \"32ce6467-5498-4e99-9fc8-e0dc498b148e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:13.890790 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.890758 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:13.988647 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:13.988567 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" Oct 30 04:21:14.214063 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:14.214033 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:21:14.222998 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:14.222966 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:21:14.223090 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:14.223022 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:21:14.281786 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:14.281760 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l"] Oct 30 04:21:14.282169 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:21:14.282145 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77ca7916_1d2b_4021_a5a0_52ec381c3531.slice/crio-7d2676f216b37be07867a808e5e758268e7b2269cbee6d730f47a2391690e537 WatchSource:0}: Error finding container 7d2676f216b37be07867a808e5e758268e7b2269cbee6d730f47a2391690e537: Status 404 returned error can't find the container with id 7d2676f216b37be07867a808e5e758268e7b2269cbee6d730f47a2391690e537 Oct 30 04:21:14.284381 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:14.284350 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:21:14.284464 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:14.284402 5882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:21:14.420455 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:14.420419 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e2644fd9-640e-4c3b-b614-c876383484bc","Type":"ContainerStarted","Data":"82a19f56aab2d7561ff2a47369174793ac8eaa69f7932f0cb0f0f82f3043c0e6"} Oct 30 04:21:14.420653 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:14.420509 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e2644fd9-640e-4c3b-b614-c876383484bc","Type":"ContainerStarted","Data":"e438ac23b69c2a4a6d6f5f1cabef174f27c9cedc42c0906adb6ffc913077ce3b"} Oct 30 04:21:14.422149 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:14.422112 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" event={"ID":"77ca7916-1d2b-4021-a5a0-52ec381c3531","Type":"ContainerStarted","Data":"48dd70409a2b56f6e1f9647dc50c7524a3210345f2b831d6648ca45928f3eb2c"} Oct 30 04:21:14.422256 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:14.422150 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" event={"ID":"77ca7916-1d2b-4021-a5a0-52ec381c3531","Type":"ContainerStarted","Data":"7d2676f216b37be07867a808e5e758268e7b2269cbee6d730f47a2391690e537"} Oct 30 04:21:14.422339 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:14.422323 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" Oct 30 04:21:14.423223 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:14.423153 5882 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" podUID="77ca7916-1d2b-4021-a5a0-52ec381c3531" containerName="ta-container" probeResult="failure" output="Get \"http://10.134.0.186:8080/readyz\": dial tcp 10.134.0.186:8080: connect: connection refused" Oct 30 04:21:14.466453 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:14.466408 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.466396488 podStartE2EDuration="1.466396488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:14.465878029 +0000 UTC m=+5986.580193004" watchObservedRunningTime="2026-10-30 04:21:14.466396488 +0000 UTC m=+5986.580711463" Oct 30 04:21:14.511115 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:14.510999 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" podStartSLOduration=1.510983826 podStartE2EDuration="1.510983826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:14.5099368 +0000 UTC m=+5986.624251775" watchObservedRunningTime="2026-10-30 04:21:14.510983826 +0000 UTC m=+5986.625298797" Oct 30 04:21:15.427489 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:15.427457 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" Oct 30 04:21:17.312724 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.312689 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:21:17.316657 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.316633 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.334758 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.334737 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 04:21:17.338755 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.338731 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 04:21:17.349039 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.349002 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-7276v\"" Oct 30 04:21:17.349039 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.349004 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:21:17.352524 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.352501 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbmpb\" (UniqueName: \"kubernetes.io/projected/449bc5dd-afe8-411e-9173-4504d91caff1-kube-api-access-fbmpb\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.353034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.352537 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.353034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.352573 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.353034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.352759 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/449bc5dd-afe8-411e-9173-4504d91caff1-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.353034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.352833 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.353034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.352858 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/449bc5dd-afe8-411e-9173-4504d91caff1-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.353034 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.352878 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/449bc5dd-afe8-411e-9173-4504d91caff1-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.353458 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.353441 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:21:17.385780 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.385753 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:21:17.389534 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.389513 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.413734 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.413701 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:21:17.435379 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.435349 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 04:21:17.435580 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.435438 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-rgp4j\"" Oct 30 04:21:17.435580 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.435518 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 04:21:17.435580 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.435528 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:21:17.435580 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.435544 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 04:21:17.435821 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.435576 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 04:21:17.435821 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.435603 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 04:21:17.435821 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.435607 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 04:21:17.454308 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.454277 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbmpb\" (UniqueName: \"kubernetes.io/projected/449bc5dd-afe8-411e-9173-4504d91caff1-kube-api-access-fbmpb\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.454455 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.454313 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.454455 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.454335 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.454455 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.454381 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/449bc5dd-afe8-411e-9173-4504d91caff1-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.454455 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.454406 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.454455 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.454427 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/449bc5dd-afe8-411e-9173-4504d91caff1-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.454455 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.454453 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/449bc5dd-afe8-411e-9173-4504d91caff1-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.455018 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.454989 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/449bc5dd-afe8-411e-9173-4504d91caff1-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.456904 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.456879 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/449bc5dd-afe8-411e-9173-4504d91caff1-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.457016 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.456929 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.457201 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.457180 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/449bc5dd-afe8-411e-9173-4504d91caff1-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.457328 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.457311 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.457388 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.457374 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.470210 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.470187 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 04:21:17.470335 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.470296 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 04:21:17.470398 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.470341 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 04:21:17.479418 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.479395 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbmpb\" (UniqueName: \"kubernetes.io/projected/449bc5dd-afe8-411e-9173-4504d91caff1-kube-api-access-fbmpb\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.555569 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.555529 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.555569 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.555565 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.555837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.555593 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.555837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.555611 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.555837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.555735 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.555837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.555785 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.555837 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.555824 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.556054 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.555863 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.556054 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.555896 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.556054 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.555921 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.556054 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.555936 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njnwp\" (UniqueName: \"kubernetes.io/projected/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-kube-api-access-njnwp\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.556234 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.556089 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.626158 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.626085 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx"] Oct 30 04:21:17.626302 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.626239 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx" podUID="134b06f4-cbbe-45e4-8ee4-3abc80ff9f19" containerName="thanos-querier" containerID="cri-o://450643912acea109348c2f1cbe0cb6685578902bf45f68bfe8dfea2f5ae14a05" gracePeriod=30 Oct 30 04:21:17.634792 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.634767 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:17.658887 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.656730 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.658887 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.656801 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.658887 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.656843 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.658887 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.656870 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njnwp\" (UniqueName: \"kubernetes.io/projected/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-kube-api-access-njnwp\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.658887 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.656946 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.658887 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.656990 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.658887 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.657012 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.658887 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.657044 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.658887 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.657069 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.658887 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.657098 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.658887 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.657150 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.658887 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.657182 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.658887 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.657318 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.658887 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.658034 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.659895 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.659425 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.660401 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.660358 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.660778 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.660735 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.661175 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.661131 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.661338 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.661308 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.661912 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.661872 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.662053 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.661981 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.662552 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.662530 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.662646 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.662611 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.685876 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.685834 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njnwp\" (UniqueName: \"kubernetes.io/projected/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-kube-api-access-njnwp\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:17.858257 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.858233 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx" Oct 30 04:21:17.944541 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.944510 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:21:17.945356 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:21:17.945332 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449bc5dd_afe8_411e_9173_4504d91caff1.slice/crio-40ca7d5cd65a841521aac711fd48056acb79a8a2b8c0464fbdafcde489c1f7a5 WatchSource:0}: Error finding container 40ca7d5cd65a841521aac711fd48056acb79a8a2b8c0464fbdafcde489c1f7a5: Status 404 returned error can't find the container with id 40ca7d5cd65a841521aac711fd48056acb79a8a2b8c0464fbdafcde489c1f7a5 Oct 30 04:21:17.959951 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.959931 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134b06f4-cbbe-45e4-8ee4-3abc80ff9f19-kube-api-access-txsg9\" (UniqueName: \"kubernetes.io/projected/134b06f4-cbbe-45e4-8ee4-3abc80ff9f19-kube-api-access-txsg9\") pod \"134b06f4-cbbe-45e4-8ee4-3abc80ff9f19\" (UID: \"134b06f4-cbbe-45e4-8ee4-3abc80ff9f19\") " Oct 30 04:21:17.961708 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:17.961687 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134b06f4-cbbe-45e4-8ee4-3abc80ff9f19-kube-api-access-txsg9" pod "134b06f4-cbbe-45e4-8ee4-3abc80ff9f19" (UID: "134b06f4-cbbe-45e4-8ee4-3abc80ff9f19"). InnerVolumeSpecName "kube-api-access-txsg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:18.061655 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.061603 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"32ce6467-5498-4e99-9fc8-e0dc498b148e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:18.061798 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.061698 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txsg9\" (UniqueName: \"kubernetes.io/projected/134b06f4-cbbe-45e4-8ee4-3abc80ff9f19-kube-api-access-txsg9\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:18.093858 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.093828 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"32ce6467-5498-4e99-9fc8-e0dc498b148e\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:18.162443 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.162407 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"32ce6467-5498-4e99-9fc8-e0dc498b148e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:18.162638 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.162543 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"32ce6467-5498-4e99-9fc8-e0dc498b148e\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:18.178801 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.178775 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"32ce6467-5498-4e99-9fc8-e0dc498b148e\") DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:18.282143 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.282105 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"32ce6467-5498-4e99-9fc8-e0dc498b148e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2c422b71c1e0b8fdf34659f2cef7846e06b4353ed88027dc99ade5d7fc29e9d6/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:18.316662 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.316627 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"data-science-perses-0\" (UID: \"32ce6467-5498-4e99-9fc8-e0dc498b148e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:18.425535 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.425505 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:18.443471 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.443442 5882 generic.go:356] "Generic (PLEG): container finished" podID="134b06f4-cbbe-45e4-8ee4-3abc80ff9f19" containerID="450643912acea109348c2f1cbe0cb6685578902bf45f68bfe8dfea2f5ae14a05" exitCode=0 Oct 30 04:21:18.443575 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.443512 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx" Oct 30 04:21:18.443575 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.443523 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx" event={"ID":"134b06f4-cbbe-45e4-8ee4-3abc80ff9f19","Type":"ContainerDied","Data":"450643912acea109348c2f1cbe0cb6685578902bf45f68bfe8dfea2f5ae14a05"} Oct 30 04:21:18.443575 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.443573 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx" event={"ID":"134b06f4-cbbe-45e4-8ee4-3abc80ff9f19","Type":"ContainerDied","Data":"48ed242be269e936c2e93a4c11ff6260fd8dc9c2c5867a4636afbae412756e13"} Oct 30 04:21:18.443752 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.443593 5882 scope.go:122] "RemoveContainer" containerID="450643912acea109348c2f1cbe0cb6685578902bf45f68bfe8dfea2f5ae14a05" Oct 30 04:21:18.445332 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.445310 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"449bc5dd-afe8-411e-9173-4504d91caff1","Type":"ContainerStarted","Data":"d1f9bf44a31982ad218a9141e88c7af8b433734c2c7197faa9bd06cbbc751870"} Oct 30 04:21:18.445415 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.445396 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"449bc5dd-afe8-411e-9173-4504d91caff1","Type":"ContainerStarted","Data":"40ca7d5cd65a841521aac711fd48056acb79a8a2b8c0464fbdafcde489c1f7a5"} Oct 30 04:21:18.452101 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.452082 5882 scope.go:122] "RemoveContainer" containerID="450643912acea109348c2f1cbe0cb6685578902bf45f68bfe8dfea2f5ae14a05" Oct 30 04:21:18.452393 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:18.452373 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"450643912acea109348c2f1cbe0cb6685578902bf45f68bfe8dfea2f5ae14a05\": container with ID starting with 450643912acea109348c2f1cbe0cb6685578902bf45f68bfe8dfea2f5ae14a05 not found: ID does not exist" containerID="450643912acea109348c2f1cbe0cb6685578902bf45f68bfe8dfea2f5ae14a05" Oct 30 04:21:18.452449 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.452405 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450643912acea109348c2f1cbe0cb6685578902bf45f68bfe8dfea2f5ae14a05"} err="failed to get container status \"450643912acea109348c2f1cbe0cb6685578902bf45f68bfe8dfea2f5ae14a05\": rpc error: code = NotFound desc = could not find container \"450643912acea109348c2f1cbe0cb6685578902bf45f68bfe8dfea2f5ae14a05\": container with ID starting with 450643912acea109348c2f1cbe0cb6685578902bf45f68bfe8dfea2f5ae14a05 not found: ID does not exist" Oct 30 04:21:18.549859 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.549723 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx"] Oct 30 04:21:18.551875 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.551847 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-w6lkx"] Oct 30 04:21:18.728387 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:18.728354 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:21:18.728718 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:21:18.728689 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32ce6467_5498_4e99_9fc8_e0dc498b148e.slice/crio-f166bf6f6dacbc9a3afc5140a14994c2beb96b82b8f6efa7dbf0d0edcd7b6976 WatchSource:0}: Error finding container f166bf6f6dacbc9a3afc5140a14994c2beb96b82b8f6efa7dbf0d0edcd7b6976: Status 404 returned error can't find the container with id f166bf6f6dacbc9a3afc5140a14994c2beb96b82b8f6efa7dbf0d0edcd7b6976 Oct 30 04:21:19.453753 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:19.453587 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"32ce6467-5498-4e99-9fc8-e0dc498b148e","Type":"ContainerStarted","Data":"ece93668890242ebeb72a13b3b0336abfa137fa76f95b7c4cb75866b7c858665"} Oct 30 04:21:19.453753 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:19.453658 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"32ce6467-5498-4e99-9fc8-e0dc498b148e","Type":"ContainerStarted","Data":"f166bf6f6dacbc9a3afc5140a14994c2beb96b82b8f6efa7dbf0d0edcd7b6976"} Oct 30 04:21:19.453753 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:19.453724 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:19.497701 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:19.497653 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=6.497640875 podStartE2EDuration="6.497640875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:19.497437724 +0000 UTC m=+5991.611752700" watchObservedRunningTime="2026-10-30 04:21:19.497640875 +0000 UTC m=+5991.611955843" Oct 30 04:21:20.012112 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:20.012082 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="134b06f4-cbbe-45e4-8ee4-3abc80ff9f19" path="/var/lib/kubelet/pods/134b06f4-cbbe-45e4-8ee4-3abc80ff9f19/volumes" Oct 30 04:21:22.471815 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:22.471785 5882 generic.go:356] "Generic (PLEG): container finished" podID="449bc5dd-afe8-411e-9173-4504d91caff1" containerID="d1f9bf44a31982ad218a9141e88c7af8b433734c2c7197faa9bd06cbbc751870" exitCode=0 Oct 30 04:21:22.472205 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:22.471844 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"449bc5dd-afe8-411e-9173-4504d91caff1","Type":"ContainerDied","Data":"d1f9bf44a31982ad218a9141e88c7af8b433734c2c7197faa9bd06cbbc751870"} Oct 30 04:21:23.478229 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:23.478195 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"449bc5dd-afe8-411e-9173-4504d91caff1","Type":"ContainerStarted","Data":"9c6bcf7dcb25f2dc7cae57e163f8d46dfceb628f05309b3641776dd6aeed85a2"} Oct 30 04:21:23.478597 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:23.478252 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"449bc5dd-afe8-411e-9173-4504d91caff1","Type":"ContainerStarted","Data":"571c27e617ba2a88ac04550f2d37fcd92c49d86518eafbac1702a67a89e1e436"} Oct 30 04:21:23.478597 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:23.478428 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:23.556855 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:23.556811 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.353093713 podStartE2EDuration="6.556798043s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.20370433s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:23.542451619 +0000 UTC m=+5995.656766593" watchObservedRunningTime="2026-10-30 04:21:23.556798043 +0000 UTC m=+5995.671113018" Oct 30 04:21:23.944439 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:23.944398 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:21:24.016506 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:24.016471 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:24.048792 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:24.048761 5882 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:24.117697 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:24.117668 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:24.117848 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:24.117773 5882 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:24.133631 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:24.133590 5882 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") DevicePath \"csi-927d18ef2eb99137c68fbe5e0bb8ea629bb5219ed56cdede72da7c0f045be95e\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:24.237809 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:24.237779 5882 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a535375d22e374ba333968465471c7604a06f9d95a513910fe4bb083b629e986/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:24.266800 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:24.266773 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:24.311153 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:24.311123 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:24.634134 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:24.634104 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:21:24.635211 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:21:24.635182 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b7d7e3_de9d_4c3d_9a72_92a2589f8cd6.slice/crio-cca4ce1506c1cfb8094d76498af7908147c2631ea9db77ba13d84d630a5d4e04 WatchSource:0}: Error finding container cca4ce1506c1cfb8094d76498af7908147c2631ea9db77ba13d84d630a5d4e04: Status 404 returned error can't find the container with id cca4ce1506c1cfb8094d76498af7908147c2631ea9db77ba13d84d630a5d4e04 Oct 30 04:21:25.488685 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:25.488574 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6","Type":"ContainerStarted","Data":"024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8"} Oct 30 04:21:25.488852 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:25.488691 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6","Type":"ContainerStarted","Data":"cca4ce1506c1cfb8094d76498af7908147c2631ea9db77ba13d84d630a5d4e04"} Oct 30 04:21:28.632648 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:28.618580 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 04:21:28.633356 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:28.633335 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j9v4r_7f749cb9-f4b4-4db4-a580-2877108e718c/ovn-acl-logging/0.log" Oct 30 04:21:29.483901 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:29.483874 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:29.508105 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:29.508070 5882 generic.go:356] "Generic (PLEG): container finished" podID="22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" containerID="024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8" exitCode=0 Oct 30 04:21:29.508265 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:29.508151 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6","Type":"ContainerDied","Data":"024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8"} Oct 30 04:21:30.462574 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:30.462494 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:30.517929 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:30.517894 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6","Type":"ContainerStarted","Data":"65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806"} Oct 30 04:21:30.518091 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:30.517978 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6","Type":"ContainerStarted","Data":"c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b"} Oct 30 04:21:30.518091 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:30.518018 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6","Type":"ContainerStarted","Data":"211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765"} Oct 30 04:21:30.573180 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:30.573119 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=10.08551726 podStartE2EDuration="13.573100646s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.487583386s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:30.571457364 +0000 UTC m=+6002.685772339" watchObservedRunningTime="2026-10-30 04:21:30.573100646 +0000 UTC m=+6002.687415619" Oct 30 04:21:34.311400 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:34.311372 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:39.312279 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:39.312249 5882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:39.317484 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:39.317463 5882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:39.564998 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:39.564917 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:40.916866 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:40.916832 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:21:40.917282 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:40.917119 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="449bc5dd-afe8-411e-9173-4504d91caff1" containerName="alertmanager" containerID="cri-o://571c27e617ba2a88ac04550f2d37fcd92c49d86518eafbac1702a67a89e1e436" gracePeriod=120 Oct 30 04:21:40.917282 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:40.917182 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="449bc5dd-afe8-411e-9173-4504d91caff1" containerName="config-reloader" containerID="cri-o://9c6bcf7dcb25f2dc7cae57e163f8d46dfceb628f05309b3641776dd6aeed85a2" gracePeriod=120 Oct 30 04:21:40.929816 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:40.929785 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:21:41.233658 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.233552 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:21:41.233885 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.233852 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="e2644fd9-640e-4c3b-b614-c876383484bc" containerName="otc-container" containerID="cri-o://82a19f56aab2d7561ff2a47369174793ac8eaa69f7932f0cb0f0f82f3043c0e6" gracePeriod=30 Oct 30 04:21:41.253099 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.253064 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l"] Oct 30 04:21:41.253316 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.253289 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" podUID="77ca7916-1d2b-4021-a5a0-52ec381c3531" containerName="ta-container" containerID="cri-o://48dd70409a2b56f6e1f9647dc50c7524a3210345f2b831d6648ca45928f3eb2c" gracePeriod=30 Oct 30 04:21:41.360392 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.360360 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:21:41.360579 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.360551 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="32ce6467-5498-4e99-9fc8-e0dc498b148e" containerName="perses" containerID="cri-o://ece93668890242ebeb72a13b3b0336abfa137fa76f95b7c4cb75866b7c858665" gracePeriod=30 Oct 30 04:21:41.502052 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.502011 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" Oct 30 04:21:41.582994 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.582963 5882 generic.go:356] "Generic (PLEG): container finished" podID="32ce6467-5498-4e99-9fc8-e0dc498b148e" containerID="ece93668890242ebeb72a13b3b0336abfa137fa76f95b7c4cb75866b7c858665" exitCode=0 Oct 30 04:21:41.583179 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.583034 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"32ce6467-5498-4e99-9fc8-e0dc498b148e","Type":"ContainerDied","Data":"ece93668890242ebeb72a13b3b0336abfa137fa76f95b7c4cb75866b7c858665"} Oct 30 04:21:41.585494 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.585465 5882 generic.go:356] "Generic (PLEG): container finished" podID="449bc5dd-afe8-411e-9173-4504d91caff1" containerID="9c6bcf7dcb25f2dc7cae57e163f8d46dfceb628f05309b3641776dd6aeed85a2" exitCode=0 Oct 30 04:21:41.585647 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.585531 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"449bc5dd-afe8-411e-9173-4504d91caff1","Type":"ContainerDied","Data":"9c6bcf7dcb25f2dc7cae57e163f8d46dfceb628f05309b3641776dd6aeed85a2"} Oct 30 04:21:41.586863 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.586840 5882 generic.go:356] "Generic (PLEG): container finished" podID="77ca7916-1d2b-4021-a5a0-52ec381c3531" containerID="48dd70409a2b56f6e1f9647dc50c7524a3210345f2b831d6648ca45928f3eb2c" exitCode=0 Oct 30 04:21:41.586999 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.586897 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" Oct 30 04:21:41.586999 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.586916 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" event={"ID":"77ca7916-1d2b-4021-a5a0-52ec381c3531","Type":"ContainerDied","Data":"48dd70409a2b56f6e1f9647dc50c7524a3210345f2b831d6648ca45928f3eb2c"} Oct 30 04:21:41.586999 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.586947 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l" event={"ID":"77ca7916-1d2b-4021-a5a0-52ec381c3531","Type":"ContainerDied","Data":"7d2676f216b37be07867a808e5e758268e7b2269cbee6d730f47a2391690e537"} Oct 30 04:21:41.586999 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.586961 5882 scope.go:122] "RemoveContainer" containerID="48dd70409a2b56f6e1f9647dc50c7524a3210345f2b831d6648ca45928f3eb2c" Oct 30 04:21:41.587223 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.587147 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" containerName="prometheus" containerID="cri-o://211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765" gracePeriod=600 Oct 30 04:21:41.587277 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.587202 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" containerName="config-reloader" containerID="cri-o://c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b" gracePeriod=600 Oct 30 04:21:41.587331 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.587156 5882 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" containerName="thanos-sidecar" containerID="cri-o://65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806" gracePeriod=600 Oct 30 04:21:41.597464 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.597440 5882 scope.go:122] "RemoveContainer" containerID="48dd70409a2b56f6e1f9647dc50c7524a3210345f2b831d6648ca45928f3eb2c" Oct 30 04:21:41.597928 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:41.597898 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48dd70409a2b56f6e1f9647dc50c7524a3210345f2b831d6648ca45928f3eb2c\": container with ID starting with 48dd70409a2b56f6e1f9647dc50c7524a3210345f2b831d6648ca45928f3eb2c not found: ID does not exist" containerID="48dd70409a2b56f6e1f9647dc50c7524a3210345f2b831d6648ca45928f3eb2c" Oct 30 04:21:41.598013 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.597942 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48dd70409a2b56f6e1f9647dc50c7524a3210345f2b831d6648ca45928f3eb2c"} err="failed to get container status \"48dd70409a2b56f6e1f9647dc50c7524a3210345f2b831d6648ca45928f3eb2c\": rpc error: code = NotFound desc = could not find container \"48dd70409a2b56f6e1f9647dc50c7524a3210345f2b831d6648ca45928f3eb2c\": container with ID starting with 48dd70409a2b56f6e1f9647dc50c7524a3210345f2b831d6648ca45928f3eb2c not found: ID does not exist" Oct 30 04:21:41.691851 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.691817 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77ca7916-1d2b-4021-a5a0-52ec381c3531-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/77ca7916-1d2b-4021-a5a0-52ec381c3531-data-science-collector-ta-server-cert\") pod \"77ca7916-1d2b-4021-a5a0-52ec381c3531\" (UID: \"77ca7916-1d2b-4021-a5a0-52ec381c3531\") " Oct 30 04:21:41.692013 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.691992 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ca7916-1d2b-4021-a5a0-52ec381c3531-kube-api-access-fwp45\" (UniqueName: \"kubernetes.io/projected/77ca7916-1d2b-4021-a5a0-52ec381c3531-kube-api-access-fwp45\") pod \"77ca7916-1d2b-4021-a5a0-52ec381c3531\" (UID: \"77ca7916-1d2b-4021-a5a0-52ec381c3531\") " Oct 30 04:21:41.692104 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.692032 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ca7916-1d2b-4021-a5a0-52ec381c3531-ta-internal\" (UniqueName: \"kubernetes.io/configmap/77ca7916-1d2b-4021-a5a0-52ec381c3531-ta-internal\") pod \"77ca7916-1d2b-4021-a5a0-52ec381c3531\" (UID: \"77ca7916-1d2b-4021-a5a0-52ec381c3531\") " Oct 30 04:21:41.692566 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.692536 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ca7916-1d2b-4021-a5a0-52ec381c3531-ta-internal" pod "77ca7916-1d2b-4021-a5a0-52ec381c3531" (UID: "77ca7916-1d2b-4021-a5a0-52ec381c3531"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:21:41.694384 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.694355 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77ca7916-1d2b-4021-a5a0-52ec381c3531-data-science-collector-ta-server-cert" pod "77ca7916-1d2b-4021-a5a0-52ec381c3531" (UID: "77ca7916-1d2b-4021-a5a0-52ec381c3531"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:41.694696 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.694674 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ca7916-1d2b-4021-a5a0-52ec381c3531-kube-api-access-fwp45" pod "77ca7916-1d2b-4021-a5a0-52ec381c3531" (UID: "77ca7916-1d2b-4021-a5a0-52ec381c3531"). InnerVolumeSpecName "kube-api-access-fwp45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:41.701827 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.701712 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:41.793281 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.793252 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ce6467-5498-4e99-9fc8-e0dc498b148e-kube-api-access-58s29\" (UniqueName: \"kubernetes.io/projected/32ce6467-5498-4e99-9fc8-e0dc498b148e-kube-api-access-58s29\") pod \"32ce6467-5498-4e99-9fc8-e0dc498b148e\" (UID: \"32ce6467-5498-4e99-9fc8-e0dc498b148e\") " Oct 30 04:21:41.793396 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.793345 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ce6467-5498-4e99-9fc8-e0dc498b148e-config\" (UniqueName: \"kubernetes.io/configmap/32ce6467-5498-4e99-9fc8-e0dc498b148e-config\") pod \"32ce6467-5498-4e99-9fc8-e0dc498b148e\" (UID: \"32ce6467-5498-4e99-9fc8-e0dc498b148e\") " Oct 30 04:21:41.793396 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.793383 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ce6467-5498-4e99-9fc8-e0dc498b148e-plugins\" (UniqueName: \"kubernetes.io/empty-dir/32ce6467-5498-4e99-9fc8-e0dc498b148e-plugins\") pod \"32ce6467-5498-4e99-9fc8-e0dc498b148e\" (UID: \"32ce6467-5498-4e99-9fc8-e0dc498b148e\") " Oct 30 04:21:41.793554 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.793500 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") pod \"32ce6467-5498-4e99-9fc8-e0dc498b148e\" (UID: \"32ce6467-5498-4e99-9fc8-e0dc498b148e\") " Oct 30 04:21:41.793899 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.793835 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ce6467-5498-4e99-9fc8-e0dc498b148e-config" pod "32ce6467-5498-4e99-9fc8-e0dc498b148e" (UID: "32ce6467-5498-4e99-9fc8-e0dc498b148e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:21:41.793899 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.793855 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwp45\" (UniqueName: \"kubernetes.io/projected/77ca7916-1d2b-4021-a5a0-52ec381c3531-kube-api-access-fwp45\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:41.793899 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.793878 5882 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/77ca7916-1d2b-4021-a5a0-52ec381c3531-ta-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:41.793899 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.793892 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/77ca7916-1d2b-4021-a5a0-52ec381c3531-data-science-collector-ta-server-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:41.797886 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.797858 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ce6467-5498-4e99-9fc8-e0dc498b148e-kube-api-access-58s29" pod "32ce6467-5498-4e99-9fc8-e0dc498b148e" (UID: "32ce6467-5498-4e99-9fc8-e0dc498b148e"). InnerVolumeSpecName "kube-api-access-58s29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:41.799863 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.799839 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e" pod "32ce6467-5498-4e99-9fc8-e0dc498b148e" (UID: "32ce6467-5498-4e99-9fc8-e0dc498b148e"). InnerVolumeSpecName "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:21:41.891171 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.890912 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:41.894356 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.894329 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/32ce6467-5498-4e99-9fc8-e0dc498b148e-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:41.894573 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.894368 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:21:41.894573 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.894380 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58s29\" (UniqueName: \"kubernetes.io/projected/32ce6467-5498-4e99-9fc8-e0dc498b148e-kube-api-access-58s29\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:41.910768 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.910731 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:21:41.973053 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.973016 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ce6467-5498-4e99-9fc8-e0dc498b148e-plugins" pod "32ce6467-5498-4e99-9fc8-e0dc498b148e" (UID: "32ce6467-5498-4e99-9fc8-e0dc498b148e"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:21:41.980236 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.980207 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l"] Oct 30 04:21:41.984425 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.984394 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7d5ccc87d9-pmx4l"] Oct 30 04:21:41.995554 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.995531 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/449bc5dd-afe8-411e-9173-4504d91caff1-tls-assets\" (UniqueName: \"kubernetes.io/projected/449bc5dd-afe8-411e-9173-4504d91caff1-tls-assets\") pod \"449bc5dd-afe8-411e-9173-4504d91caff1\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " Oct 30 04:21:41.995700 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.995578 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-cluster-tls-config\") pod \"449bc5dd-afe8-411e-9173-4504d91caff1\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " Oct 30 04:21:41.995700 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.995638 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-config-volume\" (UniqueName: \"kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-config-volume\") pod \"449bc5dd-afe8-411e-9173-4504d91caff1\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " Oct 30 04:21:41.995700 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.995682 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/449bc5dd-afe8-411e-9173-4504d91caff1-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/449bc5dd-afe8-411e-9173-4504d91caff1-alertmanager-data-science-monitoringstack-db\") pod \"449bc5dd-afe8-411e-9173-4504d91caff1\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " Oct 30 04:21:41.995879 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.995717 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/449bc5dd-afe8-411e-9173-4504d91caff1-config-out\" (UniqueName: \"kubernetes.io/empty-dir/449bc5dd-afe8-411e-9173-4504d91caff1-config-out\") pod \"449bc5dd-afe8-411e-9173-4504d91caff1\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " Oct 30 04:21:41.995879 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.995757 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/449bc5dd-afe8-411e-9173-4504d91caff1-kube-api-access-fbmpb\" (UniqueName: \"kubernetes.io/projected/449bc5dd-afe8-411e-9173-4504d91caff1-kube-api-access-fbmpb\") pod \"449bc5dd-afe8-411e-9173-4504d91caff1\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " Oct 30 04:21:41.995879 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.995830 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-web-config\" (UniqueName: \"kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-web-config\") pod \"449bc5dd-afe8-411e-9173-4504d91caff1\" (UID: \"449bc5dd-afe8-411e-9173-4504d91caff1\") " Oct 30 04:21:41.996219 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.996172 5882 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/32ce6467-5498-4e99-9fc8-e0dc498b148e-plugins\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:41.996219 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.996198 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-0dc626da-2029-40cc-ae2f-995ac1789fcb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-099852842eb43556e\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-ea3aee7dc4f5317c992a357d39aee402155ff76ffc9c0741b337b91a7708b407\"" Oct 30 04:21:41.996479 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.996345 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449bc5dd-afe8-411e-9173-4504d91caff1-alertmanager-data-science-monitoringstack-db" pod "449bc5dd-afe8-411e-9173-4504d91caff1" (UID: "449bc5dd-afe8-411e-9173-4504d91caff1"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:21:41.998200 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.998168 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-config-volume" pod "449bc5dd-afe8-411e-9173-4504d91caff1" (UID: "449bc5dd-afe8-411e-9173-4504d91caff1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:41.998939 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.998913 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449bc5dd-afe8-411e-9173-4504d91caff1-config-out" pod "449bc5dd-afe8-411e-9173-4504d91caff1" (UID: "449bc5dd-afe8-411e-9173-4504d91caff1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:21:41.999841 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.999815 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449bc5dd-afe8-411e-9173-4504d91caff1-kube-api-access-fbmpb" pod "449bc5dd-afe8-411e-9173-4504d91caff1" (UID: "449bc5dd-afe8-411e-9173-4504d91caff1"). InnerVolumeSpecName "kube-api-access-fbmpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:41.999953 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:41.999844 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449bc5dd-afe8-411e-9173-4504d91caff1-tls-assets" pod "449bc5dd-afe8-411e-9173-4504d91caff1" (UID: "449bc5dd-afe8-411e-9173-4504d91caff1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:42.004056 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.004017 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-cluster-tls-config" pod "449bc5dd-afe8-411e-9173-4504d91caff1" (UID: "449bc5dd-afe8-411e-9173-4504d91caff1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:42.014043 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.014020 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ca7916-1d2b-4021-a5a0-52ec381c3531" path="/var/lib/kubelet/pods/77ca7916-1d2b-4021-a5a0-52ec381c3531/volumes" Oct 30 04:21:42.018492 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.018464 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-web-config" pod "449bc5dd-afe8-411e-9173-4504d91caff1" (UID: "449bc5dd-afe8-411e-9173-4504d91caff1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:42.049029 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.048997 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:42.051035 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.051018 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:42.096939 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.096912 5882 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/449bc5dd-afe8-411e-9173-4504d91caff1-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.096939 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.096937 5882 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/449bc5dd-afe8-411e-9173-4504d91caff1-config-out\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.097128 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.096948 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbmpb\" (UniqueName: \"kubernetes.io/projected/449bc5dd-afe8-411e-9173-4504d91caff1-kube-api-access-fbmpb\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.097128 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.096959 5882 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-web-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.097128 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.096967 5882 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/449bc5dd-afe8-411e-9173-4504d91caff1-tls-assets\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.097128 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.096975 5882 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-cluster-tls-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.097128 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.096983 5882 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/449bc5dd-afe8-411e-9173-4504d91caff1-config-volume\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.197878 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.197847 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-thanos-prometheus-http-client-file\") pod \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " Oct 30 04:21:42.198075 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.197896 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-monitoring-tls-certs\") pod \"e2644fd9-640e-4c3b-b614-c876383484bc\" (UID: \"e2644fd9-640e-4c3b-b614-c876383484bc\") " Oct 30 04:21:42.198075 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.197930 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2644fd9-640e-4c3b-b614-c876383484bc-kube-api-access-589fg\" (UniqueName: \"kubernetes.io/projected/e2644fd9-640e-4c3b-b614-c876383484bc-kube-api-access-589fg\") pod \"e2644fd9-640e-4c3b-b614-c876383484bc\" (UID: \"e2644fd9-640e-4c3b-b614-c876383484bc\") " Oct 30 04:21:42.198075 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.197953 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-tls-certs\") pod \"e2644fd9-640e-4c3b-b614-c876383484bc\" (UID: \"e2644fd9-640e-4c3b-b614-c876383484bc\") " Oct 30 04:21:42.198075 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.197986 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " Oct 30 04:21:42.198308 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.198078 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " Oct 30 04:21:42.198308 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.198145 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " Oct 30 04:21:42.198308 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.198177 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " Oct 30 04:21:42.198308 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.198213 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-kube-api-access-njnwp\" (UniqueName: \"kubernetes.io/projected/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-kube-api-access-njnwp\") pod \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " Oct 30 04:21:42.198308 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.198251 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config\") pod \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " Oct 30 04:21:42.199283 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.199248 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-0" pod "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" (UID: "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:21:42.199905 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.199879 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-data-science-collector-ta-client-cert\") pod \"e2644fd9-640e-4c3b-b614-c876383484bc\" (UID: \"e2644fd9-640e-4c3b-b614-c876383484bc\") " Oct 30 04:21:42.200015 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.199956 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2644fd9-640e-4c3b-b614-c876383484bc-otc-internal\" (UniqueName: \"kubernetes.io/configmap/e2644fd9-640e-4c3b-b614-c876383484bc-otc-internal\") pod \"e2644fd9-640e-4c3b-b614-c876383484bc\" (UID: \"e2644fd9-640e-4c3b-b614-c876383484bc\") " Oct 30 04:21:42.200015 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.199996 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " Oct 30 04:21:42.200170 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.200035 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-config-out\" (UniqueName: \"kubernetes.io/empty-dir/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-config-out\") pod \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " Oct 30 04:21:42.200170 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.200094 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-secret-prometheus-web-tls-ca\") pod \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " Oct 30 04:21:42.200170 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.200123 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-tls-assets\" (UniqueName: \"kubernetes.io/projected/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-tls-assets\") pod \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " Oct 30 04:21:42.200170 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.200157 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " Oct 30 04:21:42.208033 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.200193 5882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-config\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-config\") pod \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\" (UID: \"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6\") " Oct 30 04:21:42.208033 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.200551 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2644fd9-640e-4c3b-b614-c876383484bc-otc-internal" pod "e2644fd9-640e-4c3b-b614-c876383484bc" (UID: "e2644fd9-640e-4c3b-b614-c876383484bc"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:21:42.208033 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.200633 5882 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e2644fd9-640e-4c3b-b614-c876383484bc-otc-internal\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.208033 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.200660 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.208033 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.201217 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-1" pod "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" (UID: "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:21:42.208033 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.201995 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-2" pod "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" (UID: "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:21:42.208033 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.204403 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-data-science-collector-ta-client-cert" pod "e2644fd9-640e-4c3b-b614-c876383484bc" (UID: "e2644fd9-640e-4c3b-b614-c876383484bc"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:42.208033 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.204692 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-config" pod "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" (UID: "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:42.208033 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.204853 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-tls-assets" pod "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" (UID: "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:42.208033 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.205192 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-config-out" pod "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" (UID: "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:21:42.208033 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.205456 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-secret-prometheus-web-tls-ca" pod "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" (UID: "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:42.208033 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.205792 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-kube-api-access-njnwp" pod "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" (UID: "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6"). InnerVolumeSpecName "kube-api-access-njnwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:42.208033 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.206860 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" (UID: "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:42.208033 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.206926 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-thanos-prometheus-http-client-file" pod "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" (UID: "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:42.208033 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.207703 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2644fd9-640e-4c3b-b614-c876383484bc-kube-api-access-589fg" pod "e2644fd9-640e-4c3b-b614-c876383484bc" (UID: "e2644fd9-640e-4c3b-b614-c876383484bc"). InnerVolumeSpecName "kube-api-access-589fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:42.208861 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.208521 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-monitoring-tls-certs" pod "e2644fd9-640e-4c3b-b614-c876383484bc" (UID: "e2644fd9-640e-4c3b-b614-c876383484bc"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:42.208861 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.208570 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" (UID: "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:42.209235 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.209078 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-tls-certs" pod "e2644fd9-640e-4c3b-b614-c876383484bc" (UID: "e2644fd9-640e-4c3b-b614-c876383484bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:42.216281 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.216256 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4" pod "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" (UID: "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6"). InnerVolumeSpecName "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:21:42.225121 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.225087 5882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config" pod "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" (UID: "22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:42.301834 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.301752 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njnwp\" (UniqueName: \"kubernetes.io/projected/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-kube-api-access-njnwp\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.301834 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.301784 5882 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.301834 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.301798 5882 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.301834 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.301812 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.301834 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.301825 5882 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-config-out\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.302218 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.301839 5882 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-secret-prometheus-web-tls-ca\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.302218 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.301853 5882 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-tls-assets\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.302218 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.301866 5882 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.302218 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.301877 5882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-config\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.302218 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.301889 5882 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.302218 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.301903 5882 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-monitoring-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.302218 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.301914 5882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-589fg\" (UniqueName: \"kubernetes.io/projected/e2644fd9-640e-4c3b-b614-c876383484bc-kube-api-access-589fg\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.302218 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.301928 5882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2644fd9-640e-4c3b-b614-c876383484bc-tls-certs\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.302218 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.301965 5882 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-141-200.ec2.internal\" " Oct 30 04:21:42.302218 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.301980 5882 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.302218 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.301996 5882 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"\"" Oct 30 04:21:42.321280 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.321254 5882 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4") on node "ip-10-0-141-200.ec2.internal" Oct 30 04:21:42.402652 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.402601 5882 reconciler_common.go:299] "Volume detached for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-141-200.ec2.internal\" DevicePath \"csi-927d18ef2eb99137c68fbe5e0bb8ea629bb5219ed56cdede72da7c0f045be95e\"" Oct 30 04:21:42.592982 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.592897 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 30 04:21:42.592982 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.592924 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"32ce6467-5498-4e99-9fc8-e0dc498b148e","Type":"ContainerDied","Data":"f166bf6f6dacbc9a3afc5140a14994c2beb96b82b8f6efa7dbf0d0edcd7b6976"} Oct 30 04:21:42.593267 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.592979 5882 scope.go:122] "RemoveContainer" containerID="ece93668890242ebeb72a13b3b0336abfa137fa76f95b7c4cb75866b7c858665" Oct 30 04:21:42.595587 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.595561 5882 generic.go:356] "Generic (PLEG): container finished" podID="22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" containerID="65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806" exitCode=0 Oct 30 04:21:42.595725 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.595590 5882 generic.go:356] "Generic (PLEG): container finished" podID="22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" containerID="c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b" exitCode=0 Oct 30 04:21:42.595725 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.595588 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6","Type":"ContainerDied","Data":"65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806"} Oct 30 04:21:42.595725 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.595600 5882 generic.go:356] "Generic (PLEG): container finished" podID="22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" containerID="211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765" exitCode=0 Oct 30 04:21:42.595725 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.595655 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:21:42.595725 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.595703 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6","Type":"ContainerDied","Data":"c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b"} Oct 30 04:21:42.595964 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.595745 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6","Type":"ContainerDied","Data":"211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765"} Oct 30 04:21:42.595964 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.595785 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6","Type":"ContainerDied","Data":"cca4ce1506c1cfb8094d76498af7908147c2631ea9db77ba13d84d630a5d4e04"} Oct 30 04:21:42.599320 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.599295 5882 generic.go:356] "Generic (PLEG): container finished" podID="449bc5dd-afe8-411e-9173-4504d91caff1" containerID="571c27e617ba2a88ac04550f2d37fcd92c49d86518eafbac1702a67a89e1e436" exitCode=0 Oct 30 04:21:42.599414 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.599345 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:21:42.599414 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.599367 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"449bc5dd-afe8-411e-9173-4504d91caff1","Type":"ContainerDied","Data":"571c27e617ba2a88ac04550f2d37fcd92c49d86518eafbac1702a67a89e1e436"} Oct 30 04:21:42.599542 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.599419 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"449bc5dd-afe8-411e-9173-4504d91caff1","Type":"ContainerDied","Data":"40ca7d5cd65a841521aac711fd48056acb79a8a2b8c0464fbdafcde489c1f7a5"} Oct 30 04:21:42.600880 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.600857 5882 generic.go:356] "Generic (PLEG): container finished" podID="e2644fd9-640e-4c3b-b614-c876383484bc" containerID="82a19f56aab2d7561ff2a47369174793ac8eaa69f7932f0cb0f0f82f3043c0e6" exitCode=0 Oct 30 04:21:42.600979 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.600894 5882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:42.600979 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.600943 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e2644fd9-640e-4c3b-b614-c876383484bc","Type":"ContainerDied","Data":"82a19f56aab2d7561ff2a47369174793ac8eaa69f7932f0cb0f0f82f3043c0e6"} Oct 30 04:21:42.601096 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.600983 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e2644fd9-640e-4c3b-b614-c876383484bc","Type":"ContainerDied","Data":"e438ac23b69c2a4a6d6f5f1cabef174f27c9cedc42c0906adb6ffc913077ce3b"} Oct 30 04:21:42.602727 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.602702 5882 scope.go:122] "RemoveContainer" containerID="65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806" Oct 30 04:21:42.611451 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.611431 5882 scope.go:122] "RemoveContainer" containerID="c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b" Oct 30 04:21:42.619384 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.619366 5882 scope.go:122] "RemoveContainer" containerID="211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765" Oct 30 04:21:42.627059 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.627037 5882 scope.go:122] "RemoveContainer" containerID="024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8" Oct 30 04:21:42.634602 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.634578 5882 scope.go:122] "RemoveContainer" containerID="65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806" Oct 30 04:21:42.634895 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:42.634864 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806\": container with ID starting with 65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806 not found: ID does not exist" containerID="65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806" Oct 30 04:21:42.634952 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.634897 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806"} err="failed to get container status \"65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806\": rpc error: code = NotFound desc = could not find container \"65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806\": container with ID starting with 65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806 not found: ID does not exist" Oct 30 04:21:42.634952 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.634914 5882 scope.go:122] "RemoveContainer" containerID="c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b" Oct 30 04:21:42.635151 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:42.635136 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b\": container with ID starting with c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b not found: ID does not exist" containerID="c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b" Oct 30 04:21:42.635194 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.635157 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b"} err="failed to get container status \"c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b\": rpc error: code = NotFound desc = could not find container \"c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b\": container with ID starting with c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b not found: ID does not exist" Oct 30 04:21:42.635194 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.635171 5882 scope.go:122] "RemoveContainer" containerID="211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765" Oct 30 04:21:42.635371 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:42.635356 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765\": container with ID starting with 211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765 not found: ID does not exist" containerID="211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765" Oct 30 04:21:42.635415 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.635376 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765"} err="failed to get container status \"211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765\": rpc error: code = NotFound desc = could not find container \"211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765\": container with ID starting with 211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765 not found: ID does not exist" Oct 30 04:21:42.635415 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.635389 5882 scope.go:122] "RemoveContainer" containerID="024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8" Oct 30 04:21:42.635594 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:42.635574 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8\": container with ID starting with 024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8 not found: ID does not exist" containerID="024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8" Oct 30 04:21:42.635685 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.635602 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8"} err="failed to get container status \"024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8\": rpc error: code = NotFound desc = could not find container \"024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8\": container with ID starting with 024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8 not found: ID does not exist" Oct 30 04:21:42.635685 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.635638 5882 scope.go:122] "RemoveContainer" containerID="65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806" Oct 30 04:21:42.635851 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.635834 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806"} err="failed to get container status \"65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806\": rpc error: code = NotFound desc = could not find container \"65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806\": container with ID starting with 65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806 not found: ID does not exist" Oct 30 04:21:42.635906 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.635851 5882 scope.go:122] "RemoveContainer" containerID="c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b" Oct 30 04:21:42.636028 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.636013 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b"} err="failed to get container status \"c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b\": rpc error: code = NotFound desc = could not find container \"c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b\": container with ID starting with c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b not found: ID does not exist" Oct 30 04:21:42.636068 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.636030 5882 scope.go:122] "RemoveContainer" containerID="211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765" Oct 30 04:21:42.636225 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.636207 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765"} err="failed to get container status \"211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765\": rpc error: code = NotFound desc = could not find container \"211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765\": container with ID starting with 211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765 not found: ID does not exist" Oct 30 04:21:42.636266 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.636225 5882 scope.go:122] "RemoveContainer" containerID="024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8" Oct 30 04:21:42.636398 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.636382 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8"} err="failed to get container status \"024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8\": rpc error: code = NotFound desc = could not find container \"024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8\": container with ID starting with 024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8 not found: ID does not exist" Oct 30 04:21:42.636443 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.636398 5882 scope.go:122] "RemoveContainer" containerID="65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806" Oct 30 04:21:42.636582 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.636567 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806"} err="failed to get container status \"65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806\": rpc error: code = NotFound desc = could not find container \"65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806\": container with ID starting with 65880644f63836374010da31ff6e982abb574b9142e9ae6c71bfd76d02755806 not found: ID does not exist" Oct 30 04:21:42.636643 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.636582 5882 scope.go:122] "RemoveContainer" containerID="c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b" Oct 30 04:21:42.636781 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.636766 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b"} err="failed to get container status \"c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b\": rpc error: code = NotFound desc = could not find container \"c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b\": container with ID starting with c8ace3ae85a9f250405f8cad0eb29e18b23de8f798e90599ec3039566184414b not found: ID does not exist" Oct 30 04:21:42.636829 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.636781 5882 scope.go:122] "RemoveContainer" containerID="211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765" Oct 30 04:21:42.636953 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.636938 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765"} err="failed to get container status \"211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765\": rpc error: code = NotFound desc = could not find container \"211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765\": container with ID starting with 211c38c5a54eba20f8fadefe46567c4f65c3de6e629ff21e13cd83c3f6c85765 not found: ID does not exist" Oct 30 04:21:42.637001 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.636953 5882 scope.go:122] "RemoveContainer" containerID="024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8" Oct 30 04:21:42.637117 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.637101 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8"} err="failed to get container status \"024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8\": rpc error: code = NotFound desc = could not find container \"024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8\": container with ID starting with 024dbe4e64ea2a4acd9d9276b0d2df4b4126fb441b8fa5fdb2b24ed1790b06f8 not found: ID does not exist" Oct 30 04:21:42.637163 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.637116 5882 scope.go:122] "RemoveContainer" containerID="9c6bcf7dcb25f2dc7cae57e163f8d46dfceb628f05309b3641776dd6aeed85a2" Oct 30 04:21:42.644015 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.644000 5882 scope.go:122] "RemoveContainer" containerID="571c27e617ba2a88ac04550f2d37fcd92c49d86518eafbac1702a67a89e1e436" Oct 30 04:21:42.651364 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.651349 5882 scope.go:122] "RemoveContainer" containerID="d1f9bf44a31982ad218a9141e88c7af8b433734c2c7197faa9bd06cbbc751870" Oct 30 04:21:42.652733 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.652713 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:21:42.656411 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.656392 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:21:42.659100 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.659081 5882 scope.go:122] "RemoveContainer" containerID="9c6bcf7dcb25f2dc7cae57e163f8d46dfceb628f05309b3641776dd6aeed85a2" Oct 30 04:21:42.659402 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:42.659381 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c6bcf7dcb25f2dc7cae57e163f8d46dfceb628f05309b3641776dd6aeed85a2\": container with ID starting with 9c6bcf7dcb25f2dc7cae57e163f8d46dfceb628f05309b3641776dd6aeed85a2 not found: ID does not exist" containerID="9c6bcf7dcb25f2dc7cae57e163f8d46dfceb628f05309b3641776dd6aeed85a2" Oct 30 04:21:42.659475 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.659407 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c6bcf7dcb25f2dc7cae57e163f8d46dfceb628f05309b3641776dd6aeed85a2"} err="failed to get container status \"9c6bcf7dcb25f2dc7cae57e163f8d46dfceb628f05309b3641776dd6aeed85a2\": rpc error: code = NotFound desc = could not find container \"9c6bcf7dcb25f2dc7cae57e163f8d46dfceb628f05309b3641776dd6aeed85a2\": container with ID starting with 9c6bcf7dcb25f2dc7cae57e163f8d46dfceb628f05309b3641776dd6aeed85a2 not found: ID does not exist" Oct 30 04:21:42.659475 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.659423 5882 scope.go:122] "RemoveContainer" containerID="571c27e617ba2a88ac04550f2d37fcd92c49d86518eafbac1702a67a89e1e436" Oct 30 04:21:42.659801 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:42.659778 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"571c27e617ba2a88ac04550f2d37fcd92c49d86518eafbac1702a67a89e1e436\": container with ID starting with 571c27e617ba2a88ac04550f2d37fcd92c49d86518eafbac1702a67a89e1e436 not found: ID does not exist" containerID="571c27e617ba2a88ac04550f2d37fcd92c49d86518eafbac1702a67a89e1e436" Oct 30 04:21:42.659880 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.659811 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571c27e617ba2a88ac04550f2d37fcd92c49d86518eafbac1702a67a89e1e436"} err="failed to get container status \"571c27e617ba2a88ac04550f2d37fcd92c49d86518eafbac1702a67a89e1e436\": rpc error: code = NotFound desc = could not find container \"571c27e617ba2a88ac04550f2d37fcd92c49d86518eafbac1702a67a89e1e436\": container with ID starting with 571c27e617ba2a88ac04550f2d37fcd92c49d86518eafbac1702a67a89e1e436 not found: ID does not exist" Oct 30 04:21:42.659880 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.659832 5882 scope.go:122] "RemoveContainer" containerID="d1f9bf44a31982ad218a9141e88c7af8b433734c2c7197faa9bd06cbbc751870" Oct 30 04:21:42.660099 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:42.660084 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1f9bf44a31982ad218a9141e88c7af8b433734c2c7197faa9bd06cbbc751870\": container with ID starting with d1f9bf44a31982ad218a9141e88c7af8b433734c2c7197faa9bd06cbbc751870 not found: ID does not exist" containerID="d1f9bf44a31982ad218a9141e88c7af8b433734c2c7197faa9bd06cbbc751870" Oct 30 04:21:42.660172 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.660105 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f9bf44a31982ad218a9141e88c7af8b433734c2c7197faa9bd06cbbc751870"} err="failed to get container status \"d1f9bf44a31982ad218a9141e88c7af8b433734c2c7197faa9bd06cbbc751870\": rpc error: code = NotFound desc = could not find container \"d1f9bf44a31982ad218a9141e88c7af8b433734c2c7197faa9bd06cbbc751870\": container with ID starting with d1f9bf44a31982ad218a9141e88c7af8b433734c2c7197faa9bd06cbbc751870 not found: ID does not exist" Oct 30 04:21:42.660172 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.660137 5882 scope.go:122] "RemoveContainer" containerID="82a19f56aab2d7561ff2a47369174793ac8eaa69f7932f0cb0f0f82f3043c0e6" Oct 30 04:21:42.667760 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.667742 5882 scope.go:122] "RemoveContainer" containerID="82a19f56aab2d7561ff2a47369174793ac8eaa69f7932f0cb0f0f82f3043c0e6" Oct 30 04:21:42.668013 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:42.667996 5882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82a19f56aab2d7561ff2a47369174793ac8eaa69f7932f0cb0f0f82f3043c0e6\": container with ID starting with 82a19f56aab2d7561ff2a47369174793ac8eaa69f7932f0cb0f0f82f3043c0e6 not found: ID does not exist" containerID="82a19f56aab2d7561ff2a47369174793ac8eaa69f7932f0cb0f0f82f3043c0e6" Oct 30 04:21:42.668065 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.668018 5882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82a19f56aab2d7561ff2a47369174793ac8eaa69f7932f0cb0f0f82f3043c0e6"} err="failed to get container status \"82a19f56aab2d7561ff2a47369174793ac8eaa69f7932f0cb0f0f82f3043c0e6\": rpc error: code = NotFound desc = could not find container \"82a19f56aab2d7561ff2a47369174793ac8eaa69f7932f0cb0f0f82f3043c0e6\": container with ID starting with 82a19f56aab2d7561ff2a47369174793ac8eaa69f7932f0cb0f0f82f3043c0e6 not found: ID does not exist" Oct 30 04:21:42.725207 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.725170 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:21:42.729051 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.729022 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 30 04:21:42.767449 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.767419 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:21:42.773038 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.773014 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:21:42.810238 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.810214 5882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:21:42.814333 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:42.814311 5882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:21:44.009439 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:44.009402 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" path="/var/lib/kubelet/pods/22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6/volumes" Oct 30 04:21:44.009964 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:44.009949 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ce6467-5498-4e99-9fc8-e0dc498b148e" path="/var/lib/kubelet/pods/32ce6467-5498-4e99-9fc8-e0dc498b148e/volumes" Oct 30 04:21:44.010564 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:44.010550 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="449bc5dd-afe8-411e-9173-4504d91caff1" path="/var/lib/kubelet/pods/449bc5dd-afe8-411e-9173-4504d91caff1/volumes" Oct 30 04:21:44.011023 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:44.011009 5882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2644fd9-640e-4c3b-b614-c876383484bc" path="/var/lib/kubelet/pods/e2644fd9-640e-4c3b-b614-c876383484bc/volumes" Oct 30 04:21:53.942293 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:21:53.942252 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:21:55.793057 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793026 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zrscj/must-gather-75p7w"] Oct 30 04:21:55.793442 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793364 5882 cpu_manager.go:414] "containerMap: removing container" podUID="e2644fd9-640e-4c3b-b614-c876383484bc" containerName="otc-container" Oct 30 04:21:55.793442 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793375 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2644fd9-640e-4c3b-b614-c876383484bc" containerName="otc-container" Oct 30 04:21:55.793442 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793394 5882 cpu_manager.go:414] "containerMap: removing container" podUID="449bc5dd-afe8-411e-9173-4504d91caff1" containerName="init-config-reloader" Oct 30 04:21:55.793442 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793399 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="449bc5dd-afe8-411e-9173-4504d91caff1" containerName="init-config-reloader" Oct 30 04:21:55.793442 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793406 5882 cpu_manager.go:414] "containerMap: removing container" podUID="449bc5dd-afe8-411e-9173-4504d91caff1" containerName="alertmanager" Oct 30 04:21:55.793442 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793411 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="449bc5dd-afe8-411e-9173-4504d91caff1" containerName="alertmanager" Oct 30 04:21:55.793442 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793420 5882 cpu_manager.go:414] "containerMap: removing container" podUID="32ce6467-5498-4e99-9fc8-e0dc498b148e" containerName="perses" Oct 30 04:21:55.793442 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793424 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ce6467-5498-4e99-9fc8-e0dc498b148e" containerName="perses" Oct 30 04:21:55.793442 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793431 5882 cpu_manager.go:414] "containerMap: removing container" podUID="22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" containerName="init-config-reloader" Oct 30 04:21:55.793442 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793435 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" containerName="init-config-reloader" Oct 30 04:21:55.793442 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793443 5882 cpu_manager.go:414] "containerMap: removing container" podUID="449bc5dd-afe8-411e-9173-4504d91caff1" containerName="config-reloader" Oct 30 04:21:55.793807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793448 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="449bc5dd-afe8-411e-9173-4504d91caff1" containerName="config-reloader" Oct 30 04:21:55.793807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793456 5882 cpu_manager.go:414] "containerMap: removing container" podUID="77ca7916-1d2b-4021-a5a0-52ec381c3531" containerName="ta-container" Oct 30 04:21:55.793807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793461 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ca7916-1d2b-4021-a5a0-52ec381c3531" containerName="ta-container" Oct 30 04:21:55.793807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793472 5882 cpu_manager.go:414] "containerMap: removing container" podUID="22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" containerName="config-reloader" Oct 30 04:21:55.793807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793478 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" containerName="config-reloader" Oct 30 04:21:55.793807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793485 5882 cpu_manager.go:414] "containerMap: removing container" podUID="22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" containerName="prometheus" Oct 30 04:21:55.793807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793490 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" containerName="prometheus" Oct 30 04:21:55.793807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793497 5882 cpu_manager.go:414] "containerMap: removing container" podUID="22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" containerName="thanos-sidecar" Oct 30 04:21:55.793807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793502 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" containerName="thanos-sidecar" Oct 30 04:21:55.793807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793509 5882 cpu_manager.go:414] "containerMap: removing container" podUID="134b06f4-cbbe-45e4-8ee4-3abc80ff9f19" containerName="thanos-querier" Oct 30 04:21:55.793807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793514 5882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="134b06f4-cbbe-45e4-8ee4-3abc80ff9f19" containerName="thanos-querier" Oct 30 04:21:55.793807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793563 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" containerName="thanos-sidecar" Oct 30 04:21:55.793807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793572 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="134b06f4-cbbe-45e4-8ee4-3abc80ff9f19" containerName="thanos-querier" Oct 30 04:21:55.793807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793579 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2644fd9-640e-4c3b-b614-c876383484bc" containerName="otc-container" Oct 30 04:21:55.793807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793586 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="449bc5dd-afe8-411e-9173-4504d91caff1" containerName="alertmanager" Oct 30 04:21:55.793807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793592 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ce6467-5498-4e99-9fc8-e0dc498b148e" containerName="perses" Oct 30 04:21:55.793807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793597 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="449bc5dd-afe8-411e-9173-4504d91caff1" containerName="config-reloader" Oct 30 04:21:55.793807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793602 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ca7916-1d2b-4021-a5a0-52ec381c3531" containerName="ta-container" Oct 30 04:21:55.793807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793608 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" containerName="config-reloader" Oct 30 04:21:55.793807 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.793629 5882 memory_manager.go:365] "RemoveStaleState removing state" podUID="22b7d7e3-de9d-4c3d-9a72-92a2589f8cd6" containerName="prometheus" Oct 30 04:21:55.796667 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.796650 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zrscj/must-gather-75p7w" Oct 30 04:21:55.813468 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.813445 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zrscj/must-gather-75p7w"] Oct 30 04:21:55.827095 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.827068 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zrscj\"/\"kube-root-ca.crt\"" Oct 30 04:21:55.827235 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.827145 5882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zrscj\"/\"openshift-service-ca.crt\"" Oct 30 04:21:55.827235 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.827203 5882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zrscj\"/\"default-dockercfg-c8v22\"" Oct 30 04:21:55.910112 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.910074 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/94a1fb07-b95b-4f0b-aec9-5e45cfee6b82-must-gather-output\") pod \"must-gather-75p7w\" (UID: \"94a1fb07-b95b-4f0b-aec9-5e45cfee6b82\") " pod="openshift-must-gather-zrscj/must-gather-75p7w" Oct 30 04:21:55.910258 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:55.910154 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6rww\" (UniqueName: \"kubernetes.io/projected/94a1fb07-b95b-4f0b-aec9-5e45cfee6b82-kube-api-access-g6rww\") pod \"must-gather-75p7w\" (UID: \"94a1fb07-b95b-4f0b-aec9-5e45cfee6b82\") " pod="openshift-must-gather-zrscj/must-gather-75p7w" Oct 30 04:21:56.011407 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:56.011376 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rww\" (UniqueName: \"kubernetes.io/projected/94a1fb07-b95b-4f0b-aec9-5e45cfee6b82-kube-api-access-g6rww\") pod \"must-gather-75p7w\" (UID: \"94a1fb07-b95b-4f0b-aec9-5e45cfee6b82\") " pod="openshift-must-gather-zrscj/must-gather-75p7w" Oct 30 04:21:56.011595 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:56.011436 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/94a1fb07-b95b-4f0b-aec9-5e45cfee6b82-must-gather-output\") pod \"must-gather-75p7w\" (UID: \"94a1fb07-b95b-4f0b-aec9-5e45cfee6b82\") " pod="openshift-must-gather-zrscj/must-gather-75p7w" Oct 30 04:21:56.011764 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:56.011745 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/94a1fb07-b95b-4f0b-aec9-5e45cfee6b82-must-gather-output\") pod \"must-gather-75p7w\" (UID: \"94a1fb07-b95b-4f0b-aec9-5e45cfee6b82\") " pod="openshift-must-gather-zrscj/must-gather-75p7w" Oct 30 04:21:56.036095 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:56.036067 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6rww\" (UniqueName: \"kubernetes.io/projected/94a1fb07-b95b-4f0b-aec9-5e45cfee6b82-kube-api-access-g6rww\") pod \"must-gather-75p7w\" (UID: \"94a1fb07-b95b-4f0b-aec9-5e45cfee6b82\") " pod="openshift-must-gather-zrscj/must-gather-75p7w" Oct 30 04:21:56.123064 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:56.122970 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zrscj/must-gather-75p7w" Oct 30 04:21:56.415137 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:56.415108 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zrscj/must-gather-75p7w"] Oct 30 04:21:56.415266 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:21:56.415168 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94a1fb07_b95b_4f0b_aec9_5e45cfee6b82.slice/crio-cde1a0d9ef23b2f1899daf7747135b5b6d932bf9809905fed3e97e54320aaf93 WatchSource:0}: Error finding container cde1a0d9ef23b2f1899daf7747135b5b6d932bf9809905fed3e97e54320aaf93: Status 404 returned error can't find the container with id cde1a0d9ef23b2f1899daf7747135b5b6d932bf9809905fed3e97e54320aaf93 Oct 30 04:21:56.664438 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:56.664351 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zrscj/must-gather-75p7w" event={"ID":"94a1fb07-b95b-4f0b-aec9-5e45cfee6b82","Type":"ContainerStarted","Data":"cde1a0d9ef23b2f1899daf7747135b5b6d932bf9809905fed3e97e54320aaf93"} Oct 30 04:21:57.672389 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:57.672352 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zrscj/must-gather-75p7w" event={"ID":"94a1fb07-b95b-4f0b-aec9-5e45cfee6b82","Type":"ContainerStarted","Data":"e1dfe88c57569d98a1510d9fab52afd512e252e84d079b47d9ace1ac596b2913"} Oct 30 04:21:57.672851 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:57.672403 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zrscj/must-gather-75p7w" event={"ID":"94a1fb07-b95b-4f0b-aec9-5e45cfee6b82","Type":"ContainerStarted","Data":"99af6a96227b5f24f930190d59f2e87901afdbe6d5bbea0cc01e8614ee80e0fb"} Oct 30 04:21:57.721588 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:21:57.721518 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zrscj/must-gather-75p7w" podStartSLOduration=1.909837203 podStartE2EDuration="2.721502158s" totalImagesPullingTime="811.664955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:57.71858885 +0000 UTC m=+6029.832903826" watchObservedRunningTime="2026-10-30 04:21:57.721502158 +0000 UTC m=+6029.835817132" Oct 30 04:22:00.035402 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:00.035367 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cd6rj_107bc8a5-c01b-4b78-aa29-765aab336604/global-pull-secret-syncer/0.log" Oct 30 04:22:01.314635 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:01.307651 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xkkvm_6d9c24e9-4f7d-4a79-9459-a3e2cf1129bb/konnectivity-agent/0.log" Oct 30 04:22:01.670582 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:01.670500 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-200.ec2.internal_09b727cb32d7ac800f6bb9bd152bad1f/haproxy/0.log" Oct 30 04:22:06.170639 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:06.167273 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7171f659-3a2a-4a26-bd1d-2047f058d00c/alertmanager/0.log" Oct 30 04:22:06.295321 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:06.293584 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7171f659-3a2a-4a26-bd1d-2047f058d00c/config-reloader/0.log" Oct 30 04:22:06.418589 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:06.418499 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7171f659-3a2a-4a26-bd1d-2047f058d00c/kube-rbac-proxy-web/0.log" Oct 30 04:22:06.561379 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:06.561346 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7171f659-3a2a-4a26-bd1d-2047f058d00c/kube-rbac-proxy/0.log" Oct 30 04:22:06.685797 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:06.685770 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7171f659-3a2a-4a26-bd1d-2047f058d00c/kube-rbac-proxy-metric/0.log" Oct 30 04:22:06.810640 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:06.810600 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7171f659-3a2a-4a26-bd1d-2047f058d00c/prom-label-proxy/0.log" Oct 30 04:22:06.936732 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:06.936589 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7171f659-3a2a-4a26-bd1d-2047f058d00c/init-config-reloader/0.log" Oct 30 04:22:07.208442 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:07.208360 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-s6gck_e06d9d5b-be5f-46aa-a633-8ce9c9c37589/kube-state-metrics/0.log" Oct 30 04:22:07.334106 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:07.333998 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-s6gck_e06d9d5b-be5f-46aa-a633-8ce9c9c37589/kube-rbac-proxy-main/0.log" Oct 30 04:22:07.450647 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:07.450407 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-s6gck_e06d9d5b-be5f-46aa-a633-8ce9c9c37589/kube-rbac-proxy-self/0.log" Oct 30 04:22:07.574341 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:07.574310 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-dc789c958-vff7h_fd320963-1dae-4be2-9352-15a810104d69/metrics-server/0.log" Oct 30 04:22:07.705366 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:07.705337 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-shbmn_c42eb03a-a0d4-4509-8d16-ae82e1ac3c26/monitoring-plugin/0.log" Oct 30 04:22:08.210908 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:08.210871 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c5f6v_7cecaabc-af83-4bf9-b3e5-d446b1302f12/node-exporter/0.log" Oct 30 04:22:08.338253 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:08.338221 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c5f6v_7cecaabc-af83-4bf9-b3e5-d446b1302f12/kube-rbac-proxy/0.log" Oct 30 04:22:08.460252 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:08.460225 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c5f6v_7cecaabc-af83-4bf9-b3e5-d446b1302f12/init-textfile/0.log" Oct 30 04:22:08.963188 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:08.963116 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-z5vhl_0c71f4fe-cc89-4856-accb-5e6c7c9452e2/kube-rbac-proxy-main/0.log" Oct 30 04:22:09.084124 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.084091 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-z5vhl_0c71f4fe-cc89-4856-accb-5e6c7c9452e2/kube-rbac-proxy-self/0.log" Oct 30 04:22:09.206734 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.206697 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-z5vhl_0c71f4fe-cc89-4856-accb-5e6c7c9452e2/openshift-state-metrics/0.log" Oct 30 04:22:09.345549 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.345522 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c79c37b-c9d5-41e1-b13a-4b39bfd46178/prometheus/0.log" Oct 30 04:22:09.470633 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.470166 5882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g"] Oct 30 04:22:09.473584 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.473552 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c79c37b-c9d5-41e1-b13a-4b39bfd46178/config-reloader/0.log" Oct 30 04:22:09.477154 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.477131 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" Oct 30 04:22:09.493411 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.492036 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g"] Oct 30 04:22:09.507262 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.507224 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18c4c60e-e058-415b-92e2-72dcc9427250-lib-modules\") pod \"perf-node-gather-daemonset-m2q4g\" (UID: \"18c4c60e-e058-415b-92e2-72dcc9427250\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" Oct 30 04:22:09.507262 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.507260 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/18c4c60e-e058-415b-92e2-72dcc9427250-proc\") pod \"perf-node-gather-daemonset-m2q4g\" (UID: \"18c4c60e-e058-415b-92e2-72dcc9427250\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" Oct 30 04:22:09.507499 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.507350 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18c4c60e-e058-415b-92e2-72dcc9427250-sys\") pod \"perf-node-gather-daemonset-m2q4g\" (UID: \"18c4c60e-e058-415b-92e2-72dcc9427250\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" Oct 30 04:22:09.507499 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.507385 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxcdj\" (UniqueName: \"kubernetes.io/projected/18c4c60e-e058-415b-92e2-72dcc9427250-kube-api-access-lxcdj\") pod \"perf-node-gather-daemonset-m2q4g\" (UID: \"18c4c60e-e058-415b-92e2-72dcc9427250\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" Oct 30 04:22:09.507499 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.507426 5882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/18c4c60e-e058-415b-92e2-72dcc9427250-podres\") pod \"perf-node-gather-daemonset-m2q4g\" (UID: \"18c4c60e-e058-415b-92e2-72dcc9427250\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" Oct 30 04:22:09.587543 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.587513 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c79c37b-c9d5-41e1-b13a-4b39bfd46178/thanos-sidecar/0.log" Oct 30 04:22:09.608547 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.608483 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/18c4c60e-e058-415b-92e2-72dcc9427250-podres\") pod \"perf-node-gather-daemonset-m2q4g\" (UID: \"18c4c60e-e058-415b-92e2-72dcc9427250\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" Oct 30 04:22:09.608547 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.608539 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18c4c60e-e058-415b-92e2-72dcc9427250-lib-modules\") pod \"perf-node-gather-daemonset-m2q4g\" (UID: \"18c4c60e-e058-415b-92e2-72dcc9427250\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" Oct 30 04:22:09.608785 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.608561 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/18c4c60e-e058-415b-92e2-72dcc9427250-proc\") pod \"perf-node-gather-daemonset-m2q4g\" (UID: \"18c4c60e-e058-415b-92e2-72dcc9427250\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" Oct 30 04:22:09.608785 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.608587 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18c4c60e-e058-415b-92e2-72dcc9427250-sys\") pod \"perf-node-gather-daemonset-m2q4g\" (UID: \"18c4c60e-e058-415b-92e2-72dcc9427250\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" Oct 30 04:22:09.608785 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.608608 5882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxcdj\" (UniqueName: \"kubernetes.io/projected/18c4c60e-e058-415b-92e2-72dcc9427250-kube-api-access-lxcdj\") pod \"perf-node-gather-daemonset-m2q4g\" (UID: \"18c4c60e-e058-415b-92e2-72dcc9427250\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" Oct 30 04:22:09.608785 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.608632 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/18c4c60e-e058-415b-92e2-72dcc9427250-podres\") pod \"perf-node-gather-daemonset-m2q4g\" (UID: \"18c4c60e-e058-415b-92e2-72dcc9427250\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" Oct 30 04:22:09.608785 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.608688 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/18c4c60e-e058-415b-92e2-72dcc9427250-proc\") pod \"perf-node-gather-daemonset-m2q4g\" (UID: \"18c4c60e-e058-415b-92e2-72dcc9427250\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" Oct 30 04:22:09.608785 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.608699 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18c4c60e-e058-415b-92e2-72dcc9427250-lib-modules\") pod \"perf-node-gather-daemonset-m2q4g\" (UID: \"18c4c60e-e058-415b-92e2-72dcc9427250\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" Oct 30 04:22:09.608785 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.608717 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18c4c60e-e058-415b-92e2-72dcc9427250-sys\") pod \"perf-node-gather-daemonset-m2q4g\" (UID: \"18c4c60e-e058-415b-92e2-72dcc9427250\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" Oct 30 04:22:09.636538 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.636499 5882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxcdj\" (UniqueName: \"kubernetes.io/projected/18c4c60e-e058-415b-92e2-72dcc9427250-kube-api-access-lxcdj\") pod \"perf-node-gather-daemonset-m2q4g\" (UID: \"18c4c60e-e058-415b-92e2-72dcc9427250\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" Oct 30 04:22:09.712162 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.712125 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c79c37b-c9d5-41e1-b13a-4b39bfd46178/kube-rbac-proxy-web/0.log" Oct 30 04:22:09.802636 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.801766 5882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" Oct 30 04:22:09.831717 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.831673 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c79c37b-c9d5-41e1-b13a-4b39bfd46178/kube-rbac-proxy/0.log" Oct 30 04:22:09.953254 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:09.953228 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c79c37b-c9d5-41e1-b13a-4b39bfd46178/kube-rbac-proxy-thanos/0.log" Oct 30 04:22:10.076797 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:10.076759 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7c79c37b-c9d5-41e1-b13a-4b39bfd46178/init-config-reloader/0.log" Oct 30 04:22:10.109708 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:10.109662 5882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g"] Oct 30 04:22:10.110927 ip-10-0-141-200 kubenswrapper[5882]: W1030 04:22:10.110897 5882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod18c4c60e_e058_415b_92e2_72dcc9427250.slice/crio-1c4551031e3b93430597a50d8d08b81af57554d73b63057218ce9dcf232c4d48 WatchSource:0}: Error finding container 1c4551031e3b93430597a50d8d08b81af57554d73b63057218ce9dcf232c4d48: Status 404 returned error can't find the container with id 1c4551031e3b93430597a50d8d08b81af57554d73b63057218ce9dcf232c4d48 Oct 30 04:22:10.572545 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:10.572519 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84cd9b5f86-vtnlf_c426339e-5bc3-45ec-a994-7b205c2cdc44/telemeter-client/0.log" Oct 30 04:22:10.690175 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:10.690148 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84cd9b5f86-vtnlf_c426339e-5bc3-45ec-a994-7b205c2cdc44/reload/0.log" Oct 30 04:22:10.772935 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:10.772904 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" event={"ID":"18c4c60e-e058-415b-92e2-72dcc9427250","Type":"ContainerStarted","Data":"bb2ff1ceb6ac86e372a13f34c2ee1f02dc1eec91f897295fedda3b43ff8429f6"} Oct 30 04:22:10.773125 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:10.772956 5882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" event={"ID":"18c4c60e-e058-415b-92e2-72dcc9427250","Type":"ContainerStarted","Data":"1c4551031e3b93430597a50d8d08b81af57554d73b63057218ce9dcf232c4d48"} Oct 30 04:22:10.773125 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:10.773026 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" Oct 30 04:22:10.806524 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:10.806478 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84cd9b5f86-vtnlf_c426339e-5bc3-45ec-a994-7b205c2cdc44/kube-rbac-proxy/0.log" Oct 30 04:22:10.817404 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:10.817344 5882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" podStartSLOduration=1.817326805 podStartE2EDuration="1.817326805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:22:10.816553594 +0000 UTC m=+6042.930868589" watchObservedRunningTime="2026-10-30 04:22:10.817326805 +0000 UTC m=+6042.931641778" Oct 30 04:22:10.928305 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:10.928219 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67fdc77d96-6slgz_2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/thanos-query/0.log" Oct 30 04:22:11.044245 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:11.044214 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67fdc77d96-6slgz_2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/kube-rbac-proxy-web/0.log" Oct 30 04:22:11.160035 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:11.160009 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67fdc77d96-6slgz_2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/kube-rbac-proxy/0.log" Oct 30 04:22:11.275860 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:11.275823 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67fdc77d96-6slgz_2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/prom-label-proxy/0.log" Oct 30 04:22:11.392358 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:11.392333 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67fdc77d96-6slgz_2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/kube-rbac-proxy-rules/0.log" Oct 30 04:22:11.513842 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:11.513812 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67fdc77d96-6slgz_2ec808c2-5a3a-46f1-b8ea-95d24ee6cdb0/kube-rbac-proxy-metrics/0.log" Oct 30 04:22:16.787735 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:16.787708 5882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-m2q4g" Oct 30 04:22:17.626344 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:17.626314 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2xmlx_b2f64d21-ae78-4fdd-b2e5-83c7629e563a/dns/0.log" Oct 30 04:22:17.743301 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:17.742307 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2xmlx_b2f64d21-ae78-4fdd-b2e5-83c7629e563a/kube-rbac-proxy/0.log" Oct 30 04:22:18.507374 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:18.507346 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-86gjv_8b33f658-c455-4aa4-ab62-7b5e3de65be5/dns-node-resolver/0.log" Oct 30 04:22:19.949904 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:19.949878 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s5lq6_e754f0c3-6ac7-4e3e-ab0e-c343b673abc1/node-ca/0.log" Oct 30 04:22:22.689079 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:22.689040 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q9hsv_be9a9474-03c1-489e-ab01-0e8e83d79daf/serve-healthcheck-canary/0.log" Oct 30 04:22:23.942625 ip-10-0-141-200 kubenswrapper[5882]: E1030 04:22:23.942566 5882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tfqlk" containerName="extractor" Oct 30 04:22:24.485133 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:24.485105 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tfqlk_7e93a8c5-35f7-459b-a8f1-f282130116cb/kube-rbac-proxy/0.log" Oct 30 04:22:24.600469 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:24.600441 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tfqlk_7e93a8c5-35f7-459b-a8f1-f282130116cb/exporter/0.log" Oct 30 04:22:24.716765 ip-10-0-141-200 kubenswrapper[5882]: I1030 04:22:24.716741 5882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tfqlk_7e93a8c5-35f7-459b-a8f1-f282130116cb/extractor/0.log"