Oct 30 02:39:57.049508 ip-10-0-139-19 systemd[1]: Starting Kubernetes Kubelet... Oct 30 02:39:57.510366 ip-10-0-139-19 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 02:39:57.510366 ip-10-0-139-19 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 02:39:57.510366 ip-10-0-139-19 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 02:39:57.510366 ip-10-0-139-19 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 02:39:57.515744 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515666 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:39:57.515744 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515745 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:39:57.515820 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515752 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:39:57.515820 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515756 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:39:57.515820 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515759 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:39:57.515820 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515762 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:39:57.515820 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515765 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:39:57.515820 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515768 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:39:57.515820 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515771 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:39:57.515820 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515774 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:39:57.515820 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515776 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:39:57.515820 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515779 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:39:57.515820 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515782 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:39:57.515820 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515785 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:39:57.515820 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515787 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:39:57.515820 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515790 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:39:57.515820 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515792 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:39:57.515820 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515795 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:39:57.515820 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515798 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:39:57.515820 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515801 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:39:57.515820 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515803 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:39:57.515820 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515806 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:39:57.516297 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515809 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:39:57.516297 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515812 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:39:57.516297 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515815 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:39:57.516297 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515817 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:39:57.516297 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515820 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:39:57.516297 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515822 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:39:57.516297 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515826 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:39:57.516297 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515828 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:39:57.516297 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515831 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:39:57.516297 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515834 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:39:57.516297 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515836 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:39:57.516297 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515839 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:39:57.516297 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515842 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:39:57.516297 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515844 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:39:57.516297 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515847 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:39:57.516297 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515849 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:39:57.516297 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515853 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:39:57.516297 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515855 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:39:57.516297 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515858 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515860 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515863 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515866 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515868 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515871 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515873 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515876 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515878 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515881 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515883 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515886 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515888 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515891 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515894 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515897 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515900 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515903 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515905 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515908 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515910 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:39:57.516756 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515913 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:39:57.517259 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515915 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:39:57.517259 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515918 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:39:57.517259 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515921 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:39:57.517259 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515923 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:39:57.517259 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515926 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:39:57.517259 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515928 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:39:57.517259 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515931 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:39:57.517259 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515933 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:39:57.517259 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515937 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:39:57.517259 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515940 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:39:57.517259 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515942 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:39:57.517259 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515945 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:39:57.517259 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515947 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:39:57.517259 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515950 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:39:57.517259 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515952 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:39:57.517259 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515955 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:39:57.517259 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515957 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:39:57.517259 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515961 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:39:57.517259 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515963 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:39:57.517708 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515966 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:39:57.517708 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515968 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:39:57.517708 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515971 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:39:57.517708 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515973 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:39:57.517708 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515978 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:39:57.517708 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515982 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:39:57.517708 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515986 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:39:57.517708 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515989 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:39:57.517708 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515992 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:39:57.517708 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515995 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:39:57.517708 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.515998 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:39:57.517708 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516001 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:39:57.517708 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516003 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:39:57.517708 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516006 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:39:57.517708 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516008 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:39:57.517708 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516011 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:39:57.517708 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516013 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:39:57.517708 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516017 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:39:57.517708 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516020 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:39:57.518161 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516022 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:39:57.518161 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516025 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:39:57.518161 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516027 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:39:57.518161 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516030 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:39:57.518161 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516033 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:39:57.518161 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516035 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:39:57.518161 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516886 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:39:57.518161 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516893 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:39:57.518161 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516896 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:39:57.518161 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516900 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:39:57.518161 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516903 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:39:57.518161 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516906 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:39:57.518161 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516909 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:39:57.518161 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516912 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:39:57.518161 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516914 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:39:57.518161 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516917 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:39:57.518161 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516920 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:39:57.518161 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516924 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:39:57.518161 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516930 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:39:57.518647 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516934 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:39:57.518647 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516937 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:39:57.518647 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516940 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:39:57.518647 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516943 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:39:57.518647 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516945 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:39:57.518647 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516948 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:39:57.518647 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516951 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:39:57.518647 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516954 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:39:57.518647 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516957 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:39:57.518647 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516961 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:39:57.518647 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516964 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:39:57.518647 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516967 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:39:57.518647 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516970 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:39:57.518647 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516972 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:39:57.518647 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516975 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:39:57.518647 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516977 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:39:57.518647 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516980 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:39:57.518647 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516982 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:39:57.518647 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516986 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:39:57.519093 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516989 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:39:57.519093 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516992 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:39:57.519093 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516995 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:39:57.519093 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.516997 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:39:57.519093 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517000 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:39:57.519093 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517003 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:39:57.519093 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517006 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:39:57.519093 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517008 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:39:57.519093 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517011 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:39:57.519093 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517014 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:39:57.519093 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517017 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:39:57.519093 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517019 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:39:57.519093 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517022 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:39:57.519093 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517025 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:39:57.519093 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517028 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:39:57.519093 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517030 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:39:57.519093 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517033 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:39:57.519093 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517035 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:39:57.519093 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517037 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:39:57.519093 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517040 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:39:57.519595 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517043 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:39:57.519595 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517045 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:39:57.519595 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517048 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:39:57.519595 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517050 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:39:57.519595 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517052 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:39:57.519595 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517055 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:39:57.519595 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517057 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:39:57.519595 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517060 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:39:57.519595 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517062 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:39:57.519595 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517065 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:39:57.519595 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517068 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:39:57.519595 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517070 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:39:57.519595 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517073 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:39:57.519595 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517076 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:39:57.519595 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517078 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:39:57.519595 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517081 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:39:57.519595 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517083 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:39:57.519595 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517085 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:39:57.519595 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517088 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:39:57.519595 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517091 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:39:57.520063 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517093 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:39:57.520063 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517096 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:39:57.520063 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517098 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:39:57.520063 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517101 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:39:57.520063 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517104 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:39:57.520063 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517106 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:39:57.520063 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517109 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:39:57.520063 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517111 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:39:57.520063 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517114 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:39:57.520063 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517116 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:39:57.520063 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517118 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:39:57.520063 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517121 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:39:57.520063 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517123 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:39:57.520063 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517126 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:39:57.520063 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517128 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:39:57.520063 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517145 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:39:57.520063 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517148 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:39:57.520063 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517152 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:39:57.520063 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517154 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:39:57.520063 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517157 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:39:57.520824 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517160 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:39:57.520824 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517163 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:39:57.520824 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517166 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:39:57.520824 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517168 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:39:57.520824 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517171 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:39:57.520824 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517174 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:39:57.520824 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517177 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:39:57.520824 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517180 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:39:57.520824 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517182 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:39:57.520824 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517185 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:39:57.520824 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517187 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:39:57.520824 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517190 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:39:57.520824 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517192 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:39:57.520824 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517194 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:39:57.520824 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517684 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:39:57.520824 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517688 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:39:57.520824 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517692 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:39:57.520824 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517695 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:39:57.520824 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517698 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:39:57.520824 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517700 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:39:57.521492 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517703 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:39:57.521492 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517705 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:39:57.521492 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517708 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:39:57.521492 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517710 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:39:57.521492 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517713 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:39:57.521492 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517716 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:39:57.521492 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517718 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:39:57.521492 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517721 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:39:57.521492 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517724 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:39:57.521492 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517727 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:39:57.521492 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517730 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:39:57.521492 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517732 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:39:57.521492 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517735 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:39:57.521492 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517738 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:39:57.521492 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517740 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:39:57.521492 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517743 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:39:57.521492 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517746 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:39:57.521492 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517749 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:39:57.521492 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517751 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:39:57.521934 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517754 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:39:57.521934 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517757 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:39:57.521934 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517759 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:39:57.521934 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517761 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:39:57.521934 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517764 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:39:57.521934 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517767 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:39:57.521934 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517769 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:39:57.521934 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517772 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:39:57.521934 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517775 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:39:57.521934 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517777 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:39:57.521934 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517780 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:39:57.521934 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517782 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:39:57.521934 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517785 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:39:57.521934 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517787 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:39:57.521934 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517790 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:39:57.521934 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517792 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:39:57.521934 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517795 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:39:57.521934 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517798 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:39:57.521934 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517801 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:39:57.521934 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517803 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:39:57.522442 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517806 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:39:57.522442 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517808 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:39:57.522442 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517811 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:39:57.522442 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517814 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:39:57.522442 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517819 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:39:57.522442 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517821 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:39:57.522442 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517826 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:39:57.522442 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517829 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:39:57.522442 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517831 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:39:57.522442 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517834 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:39:57.522442 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517836 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:39:57.522442 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517839 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:39:57.522442 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517842 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:39:57.522442 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517844 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:39:57.522442 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517847 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:39:57.522442 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517849 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:39:57.522442 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517852 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:39:57.522442 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517854 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:39:57.522442 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517857 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:39:57.522886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517859 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:39:57.522886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517863 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:39:57.522886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517866 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:39:57.522886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517869 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:39:57.522886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517871 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:39:57.522886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517874 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:39:57.522886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517876 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:39:57.522886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517878 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:39:57.522886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517881 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:39:57.522886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517884 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:39:57.522886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517886 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:39:57.522886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517889 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:39:57.522886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517891 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:39:57.522886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517894 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:39:57.522886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517896 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:39:57.522886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517899 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:39:57.522886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517901 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:39:57.522886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517903 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:39:57.522886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517906 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:39:57.522886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517908 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:39:57.523368 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517911 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:39:57.523368 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517913 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:39:57.523368 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517915 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:39:57.523368 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517918 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:39:57.523368 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517920 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:39:57.523368 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517923 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:39:57.523368 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517926 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:39:57.523368 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517928 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:39:57.523368 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517930 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:39:57.523368 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517933 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:39:57.523368 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517935 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:39:57.523368 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517938 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:39:57.523368 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517940 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:39:57.523368 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517943 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:39:57.523368 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517946 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:39:57.523368 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517948 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:39:57.523368 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517950 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:39:57.523368 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517953 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:39:57.523368 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517955 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517958 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517962 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.517965 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518086 2576 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518097 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518103 2576 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518108 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518112 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518116 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518121 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518125 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518129 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518143 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518147 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518150 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518153 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518156 2576 flags.go:64] FLAG: --cgroup-root="" Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518159 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518162 2576 flags.go:64] FLAG: --client-ca-file="" Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518166 2576 flags.go:64] FLAG: --cloud-provider="external" Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518170 2576 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518174 2576 flags.go:64] FLAG: --cluster-domain="" Oct 30 02:39:57.523886 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518177 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518180 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518183 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518187 2576 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518191 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518194 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518197 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518201 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518204 2576 flags.go:64] FLAG: --contention-profiling="false" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518207 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518210 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518214 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518217 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518222 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518225 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518228 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518231 2576 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518234 2576 flags.go:64] FLAG: --enable-server="true" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518237 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518241 2576 flags.go:64] FLAG: --event-burst="100" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518244 2576 flags.go:64] FLAG: --event-qps="50" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518247 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518250 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518253 2576 flags.go:64] FLAG: --eviction-hard="" Oct 30 02:39:57.524424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518257 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518261 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518264 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518267 2576 flags.go:64] FLAG: --eviction-soft="" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518270 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518273 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518285 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518289 2576 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518292 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518295 2576 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518297 2576 flags.go:64] FLAG: --feature-gates="" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518306 2576 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518312 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518316 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518319 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518322 2576 flags.go:64] FLAG: --healthz-port="10248" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518325 2576 flags.go:64] FLAG: --help="false" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518329 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-139-19.ec2.internal" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518332 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518337 2576 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518340 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518343 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518347 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518350 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 02:39:57.525047 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518353 2576 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518356 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518359 2576 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518362 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518365 2576 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518368 2576 flags.go:64] FLAG: --kube-reserved="" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518371 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518374 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518377 2576 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518380 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518383 2576 flags.go:64] FLAG: --lock-file="" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518385 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518389 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518392 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518398 2576 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518401 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518404 2576 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518407 2576 flags.go:64] FLAG: --logging-format="text" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518410 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518414 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518418 2576 flags.go:64] FLAG: --manifest-url="" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518421 2576 flags.go:64] FLAG: --manifest-url-header="" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518426 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518428 2576 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518432 2576 flags.go:64] FLAG: --max-pods="110" Oct 30 02:39:57.525780 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518435 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518439 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518443 2576 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518446 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518450 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518453 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518456 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518463 2576 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518466 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518469 2576 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518473 2576 flags.go:64] FLAG: --pod-cidr="" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518476 2576 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518479 2576 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518482 2576 flags.go:64] FLAG: --pods-per-core="0" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518485 2576 flags.go:64] FLAG: --port="10250" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518488 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518491 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c3dd801830cd7ae3" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518495 2576 flags.go:64] FLAG: --qos-reserved="" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518498 2576 flags.go:64] FLAG: --read-only-port="10255" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518501 2576 flags.go:64] FLAG: --register-node="true" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518504 2576 flags.go:64] FLAG: --register-with-taints="" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518508 2576 flags.go:64] FLAG: --registry-burst="10" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518511 2576 flags.go:64] FLAG: --registry-qps="5" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518515 2576 flags.go:64] FLAG: --reserved-cpus="" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518517 2576 flags.go:64] FLAG: --reserved-memory="" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518521 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 02:39:57.526865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518525 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518529 2576 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518532 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518535 2576 flags.go:64] FLAG: --runonce="false" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518538 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518541 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518544 2576 flags.go:64] FLAG: --seccomp-default="false" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518547 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518551 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518555 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518558 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518561 2576 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518564 2576 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518567 2576 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518570 2576 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518573 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518576 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518579 2576 flags.go:64] FLAG: --system-cgroups="" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518582 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518588 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518591 2576 flags.go:64] FLAG: --tls-cert-file="" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518593 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518597 2576 flags.go:64] FLAG: --tls-min-version="" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518600 2576 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518603 2576 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 02:39:57.527577 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518606 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518608 2576 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518611 2576 flags.go:64] FLAG: --v="2" Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518616 2576 flags.go:64] FLAG: --version="false" Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518620 2576 flags.go:64] FLAG: --vmodule="" Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518625 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.518629 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518785 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518792 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518795 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518798 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518801 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518804 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518807 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518810 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518814 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518817 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518820 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518823 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518826 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518829 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:39:57.528391 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518831 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:39:57.528936 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518834 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:39:57.528936 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518836 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:39:57.528936 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518839 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:39:57.528936 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518842 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:39:57.528936 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518844 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:39:57.528936 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518847 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:39:57.528936 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518849 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:39:57.528936 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518852 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:39:57.528936 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518855 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:39:57.528936 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518857 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:39:57.528936 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518860 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:39:57.528936 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518862 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:39:57.528936 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518865 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:39:57.528936 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518868 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:39:57.528936 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518870 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:39:57.528936 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518873 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:39:57.528936 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518876 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:39:57.528936 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518879 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:39:57.528936 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518883 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:39:57.529433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518885 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:39:57.529433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518888 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:39:57.529433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518891 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:39:57.529433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518893 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:39:57.529433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518896 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:39:57.529433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518898 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:39:57.529433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518902 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:39:57.529433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518905 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:39:57.529433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518908 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:39:57.529433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518910 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:39:57.529433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518913 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:39:57.529433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518916 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:39:57.529433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518918 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:39:57.529433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518921 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:39:57.529433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518923 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:39:57.529433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518926 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:39:57.529433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518929 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:39:57.529433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518931 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:39:57.529433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518934 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:39:57.529433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518936 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:39:57.529953 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518939 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:39:57.529953 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518941 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:39:57.529953 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518944 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:39:57.529953 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518947 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:39:57.529953 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518949 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:39:57.529953 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518953 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:39:57.529953 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518957 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:39:57.529953 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518959 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:39:57.529953 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518962 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:39:57.529953 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518965 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:39:57.529953 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518968 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:39:57.529953 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518972 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:39:57.529953 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518975 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:39:57.529953 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518979 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:39:57.529953 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518982 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:39:57.529953 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518986 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:39:57.529953 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518988 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:39:57.529953 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518991 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:39:57.529953 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518995 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:39:57.530479 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.518997 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:39:57.530479 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519000 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:39:57.530479 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519002 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:39:57.530479 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519005 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:39:57.530479 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519007 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:39:57.530479 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519010 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:39:57.530479 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519012 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:39:57.530479 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519015 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:39:57.530479 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519017 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:39:57.530479 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519020 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:39:57.530479 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519023 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:39:57.530479 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519025 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:39:57.530479 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519028 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:39:57.530479 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519030 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:39:57.530479 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519033 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:39:57.530479 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519035 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:39:57.530479 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519038 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:39:57.530479 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519040 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:39:57.530479 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519043 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:39:57.530479 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519045 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:39:57.531000 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519048 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:39:57.531000 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519050 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:39:57.531000 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519053 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:39:57.531000 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519055 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:39:57.531000 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519060 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:39:57.531000 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519062 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:39:57.531000 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519065 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:39:57.531000 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519067 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:39:57.531000 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519070 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:39:57.531000 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519072 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:39:57.531000 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519075 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:39:57.531000 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519079 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:39:57.531000 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519082 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:39:57.531000 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.519109 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 02:39:57.531000 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519161 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:39:57.531000 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519166 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:39:57.531485 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519168 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:39:57.531485 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519171 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:39:57.531485 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519173 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:39:57.531485 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519176 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:39:57.531485 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519179 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:39:57.531485 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519182 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:39:57.531485 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519185 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:39:57.531485 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519187 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:39:57.531485 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519190 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:39:57.531485 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519193 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:39:57.531485 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519195 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:39:57.531485 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519198 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:39:57.531485 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519200 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:39:57.531485 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519203 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:39:57.531485 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519205 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:39:57.531485 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519208 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:39:57.531485 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519210 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:39:57.531485 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519213 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:39:57.531485 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519215 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:39:57.531941 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519218 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:39:57.531941 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519222 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:39:57.531941 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519225 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:39:57.531941 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519228 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:39:57.531941 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519231 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:39:57.531941 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519233 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:39:57.531941 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519236 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:39:57.531941 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519238 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:39:57.531941 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519241 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:39:57.531941 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519245 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:39:57.531941 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519248 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:39:57.531941 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519250 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:39:57.531941 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519253 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:39:57.531941 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519256 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:39:57.531941 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519258 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:39:57.531941 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519261 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:39:57.531941 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519263 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:39:57.531941 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519266 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:39:57.531941 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519268 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:39:57.531941 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519271 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:39:57.532434 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519273 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:39:57.532434 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519276 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:39:57.532434 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519279 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:39:57.532434 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519281 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:39:57.532434 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519284 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:39:57.532434 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519286 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:39:57.532434 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519289 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:39:57.532434 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519291 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:39:57.532434 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519294 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:39:57.532434 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519296 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:39:57.532434 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519299 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:39:57.532434 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519301 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:39:57.532434 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519304 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:39:57.532434 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519307 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:39:57.532434 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519311 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:39:57.532434 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519315 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:39:57.532434 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519318 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:39:57.532434 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519321 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:39:57.532434 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519323 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:39:57.532884 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519327 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:39:57.532884 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519329 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:39:57.532884 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519333 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:39:57.532884 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519338 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:39:57.532884 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519342 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:39:57.532884 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519345 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:39:57.532884 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519348 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:39:57.532884 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519351 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:39:57.532884 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519354 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:39:57.532884 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519357 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:39:57.532884 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519359 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:39:57.532884 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519362 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:39:57.532884 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519364 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:39:57.532884 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519367 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:39:57.532884 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519370 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:39:57.532884 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519373 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:39:57.532884 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519375 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:39:57.532884 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519377 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:39:57.532884 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519380 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:39:57.532884 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519383 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:39:57.533362 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519385 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:39:57.533362 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519388 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:39:57.533362 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519391 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:39:57.533362 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519393 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:39:57.533362 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519396 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:39:57.533362 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519398 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:39:57.533362 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519403 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:39:57.533362 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519405 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:39:57.533362 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519408 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:39:57.533362 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519411 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:39:57.533362 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519413 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:39:57.533362 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519416 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:39:57.533362 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519418 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:39:57.533362 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519421 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:39:57.533362 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519425 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:39:57.533362 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519428 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:39:57.533362 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519430 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:39:57.533362 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519433 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:39:57.533362 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519435 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:39:57.533852 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519438 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:39:57.533852 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519442 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:39:57.533852 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519445 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:39:57.533852 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519447 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:39:57.533852 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519450 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:39:57.533852 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519452 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:39:57.533852 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.519455 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:39:57.533852 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.528308 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 02:39:57.533852 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.528324 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 02:39:57.533852 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.529149 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 02:39:57.533852 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.529156 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 02:39:57.533852 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529219 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:39:57.533852 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529224 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:39:57.533852 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529228 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:39:57.533852 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529231 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:39:57.533852 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529235 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:39:57.533852 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529238 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:39:57.533852 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529241 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:39:57.533852 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529244 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:39:57.533852 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529246 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:39:57.534367 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529249 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:39:57.534367 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529252 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:39:57.534367 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529255 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:39:57.534367 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529258 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:39:57.534367 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529260 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:39:57.534367 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529263 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:39:57.534367 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529266 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:39:57.534367 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529268 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:39:57.534367 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529272 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:39:57.534367 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529274 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:39:57.534367 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529277 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:39:57.534367 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529280 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:39:57.534367 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529283 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:39:57.534367 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529285 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:39:57.534367 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529288 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:39:57.534367 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529291 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:39:57.534367 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529294 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:39:57.534367 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529296 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:39:57.534367 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529299 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:39:57.534367 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529303 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:39:57.534835 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529306 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:39:57.534835 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529309 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:39:57.534835 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529312 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:39:57.534835 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529314 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:39:57.534835 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529317 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:39:57.534835 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529319 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:39:57.534835 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529322 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:39:57.534835 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529325 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:39:57.534835 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529328 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:39:57.534835 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529331 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:39:57.534835 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529333 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:39:57.534835 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529336 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:39:57.534835 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529339 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:39:57.534835 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529341 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:39:57.534835 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529344 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:39:57.534835 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529347 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:39:57.534835 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529349 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:39:57.534835 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529352 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:39:57.534835 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529355 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:39:57.534835 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529357 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:39:57.535325 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529360 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:39:57.535325 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529362 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:39:57.535325 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529365 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:39:57.535325 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529368 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:39:57.535325 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529371 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:39:57.535325 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529374 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:39:57.535325 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529377 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:39:57.535325 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529379 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:39:57.535325 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529382 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:39:57.535325 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529384 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:39:57.535325 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529387 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:39:57.535325 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529391 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:39:57.535325 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529394 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:39:57.535325 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529398 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:39:57.535325 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529403 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:39:57.535325 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529407 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:39:57.535325 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529410 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:39:57.535325 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529413 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:39:57.535325 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529416 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:39:57.535781 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529419 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:39:57.535781 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529421 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:39:57.535781 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529424 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:39:57.535781 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529427 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:39:57.535781 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529429 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:39:57.535781 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529432 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:39:57.535781 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529435 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:39:57.535781 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529438 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:39:57.535781 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529440 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:39:57.535781 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529443 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:39:57.535781 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529446 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:39:57.535781 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529449 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:39:57.535781 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529452 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:39:57.535781 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529454 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:39:57.535781 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529457 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:39:57.535781 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529460 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:39:57.535781 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529463 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:39:57.535781 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529465 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:39:57.535781 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529468 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:39:57.535781 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529471 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:39:57.536307 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529473 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:39:57.536307 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529476 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:39:57.536307 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529478 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:39:57.536307 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529481 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:39:57.536307 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529485 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:39:57.536307 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529487 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:39:57.536307 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529490 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:39:57.536307 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529493 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:39:57.536307 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529495 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:39:57.536307 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529498 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:39:57.536307 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529501 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:39:57.536307 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529503 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:39:57.536307 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529506 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:39:57.536307 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529510 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:39:57.536307 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529513 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:39:57.536307 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529516 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:39:57.536307 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529519 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:39:57.536307 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529522 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:39:57.536728 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.529551 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 02:39:57.536728 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529698 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:39:57.536728 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529704 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:39:57.536728 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529707 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:39:57.536728 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529710 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:39:57.536728 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529713 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:39:57.536728 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529716 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:39:57.536728 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529719 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:39:57.536728 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529722 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:39:57.536728 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529724 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:39:57.536728 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529727 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:39:57.536728 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529730 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:39:57.536728 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529733 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:39:57.536728 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529735 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:39:57.536728 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529738 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:39:57.537113 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529741 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:39:57.537113 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529743 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:39:57.537113 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529746 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:39:57.537113 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529749 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:39:57.537113 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529752 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:39:57.537113 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529755 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:39:57.537113 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529758 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:39:57.537113 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529760 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:39:57.537113 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529763 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:39:57.537113 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529766 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:39:57.537113 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529768 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:39:57.537113 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529771 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:39:57.537113 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529773 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:39:57.537113 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529776 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:39:57.537113 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529778 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:39:57.537113 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529781 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:39:57.537113 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529783 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:39:57.537113 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529786 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:39:57.537113 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529788 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:39:57.537113 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529791 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:39:57.537621 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529794 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:39:57.537621 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529797 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:39:57.537621 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529799 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:39:57.537621 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529802 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:39:57.537621 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529806 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:39:57.537621 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529808 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:39:57.537621 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529811 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:39:57.537621 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529814 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:39:57.537621 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529817 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:39:57.537621 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529819 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:39:57.537621 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529822 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:39:57.537621 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529824 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:39:57.537621 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529827 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:39:57.537621 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529829 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:39:57.537621 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529832 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:39:57.537621 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529835 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:39:57.537621 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529838 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:39:57.537621 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529841 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:39:57.537621 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529843 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:39:57.537621 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529846 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:39:57.538080 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529848 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:39:57.538080 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529851 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:39:57.538080 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529854 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:39:57.538080 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529857 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:39:57.538080 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529859 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:39:57.538080 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529862 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:39:57.538080 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529865 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:39:57.538080 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529867 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:39:57.538080 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529871 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:39:57.538080 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529875 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:39:57.538080 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529877 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:39:57.538080 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529880 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:39:57.538080 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529883 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:39:57.538080 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529886 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:39:57.538080 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529889 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:39:57.538080 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529891 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:39:57.538080 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529894 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:39:57.538080 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529897 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:39:57.538080 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529900 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:39:57.538550 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529902 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:39:57.538550 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529905 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:39:57.538550 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529907 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:39:57.538550 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529910 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:39:57.538550 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529913 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:39:57.538550 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529915 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:39:57.538550 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529918 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:39:57.538550 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529921 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:39:57.538550 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529923 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:39:57.538550 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529927 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:39:57.538550 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529930 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:39:57.538550 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529933 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:39:57.538550 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529936 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:39:57.538550 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529939 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:39:57.538550 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529941 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:39:57.538550 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529944 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:39:57.538550 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529946 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:39:57.538550 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529949 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:39:57.538550 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529952 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:39:57.538550 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529954 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:39:57.539019 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529957 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:39:57.539019 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529960 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:39:57.539019 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529963 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:39:57.539019 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529965 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:39:57.539019 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529968 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:39:57.539019 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529971 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:39:57.539019 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529974 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:39:57.539019 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529976 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:39:57.539019 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529979 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:39:57.539019 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529981 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:39:57.539019 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529984 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:39:57.539019 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529988 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:39:57.539019 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.529990 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:39:57.539019 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.530014 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 02:39:57.539019 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530046 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:39:57.539019 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530050 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:39:57.539414 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530054 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:39:57.539414 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530057 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:39:57.539414 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530059 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:39:57.539414 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530062 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:39:57.539414 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530065 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:39:57.539414 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530068 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:39:57.539414 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530070 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:39:57.539414 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530073 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:39:57.539414 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530075 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:39:57.539414 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530078 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:39:57.539414 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530081 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:39:57.539414 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530083 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:39:57.539414 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530086 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:39:57.539414 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530089 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:39:57.539414 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530093 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:39:57.539414 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530096 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:39:57.539414 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530098 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:39:57.539414 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530101 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:39:57.539414 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530104 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:39:57.539414 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530106 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:39:57.539947 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530109 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:39:57.539947 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530112 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:39:57.539947 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530114 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:39:57.539947 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530117 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:39:57.539947 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530120 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:39:57.539947 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530122 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:39:57.539947 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530125 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:39:57.539947 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530127 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:39:57.539947 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530151 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:39:57.539947 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530155 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:39:57.539947 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530158 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:39:57.539947 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530160 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:39:57.539947 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530163 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:39:57.539947 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530166 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:39:57.539947 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530169 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:39:57.539947 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530173 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:39:57.539947 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530176 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:39:57.539947 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530179 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:39:57.539947 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530182 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:39:57.540438 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530184 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:39:57.540438 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530187 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:39:57.540438 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530190 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:39:57.540438 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530193 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:39:57.540438 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530195 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:39:57.540438 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530200 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:39:57.540438 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530202 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:39:57.540438 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530205 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:39:57.540438 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530207 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:39:57.540438 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530210 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:39:57.540438 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530213 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:39:57.540438 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530216 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:39:57.540438 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530218 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:39:57.540438 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530221 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:39:57.540438 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530224 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:39:57.540438 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530226 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:39:57.540438 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530229 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:39:57.540438 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530231 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:39:57.540438 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530234 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:39:57.540883 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530237 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:39:57.540883 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530239 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:39:57.540883 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530242 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:39:57.540883 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530246 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:39:57.540883 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530248 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:39:57.540883 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530251 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:39:57.540883 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530253 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:39:57.540883 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530256 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:39:57.540883 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530259 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:39:57.540883 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530263 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:39:57.540883 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530265 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:39:57.540883 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530268 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:39:57.540883 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530271 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:39:57.540883 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530274 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:39:57.540883 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530277 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:39:57.540883 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530279 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:39:57.540883 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530282 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:39:57.540883 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530285 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:39:57.540883 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530289 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:39:57.540883 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530292 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:39:57.541433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530295 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:39:57.541433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530298 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:39:57.541433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530301 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:39:57.541433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530303 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:39:57.541433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530306 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:39:57.541433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530308 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:39:57.541433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530311 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:39:57.541433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530313 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:39:57.541433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530316 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:39:57.541433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530319 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:39:57.541433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530321 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:39:57.541433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530324 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:39:57.541433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530326 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:39:57.541433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530329 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:39:57.541433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530332 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:39:57.541433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530334 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:39:57.541433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530338 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:39:57.541433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530341 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:39:57.541433 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530343 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:39:57.541951 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530346 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:39:57.541951 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530348 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:39:57.541951 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530354 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:39:57.541951 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530358 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:39:57.541951 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530361 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:39:57.541951 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530364 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:39:57.541951 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:39:57.530367 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:39:57.541951 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.530422 2576 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 02:39:57.541951 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.532927 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 02:39:57.541951 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.533946 2576 server.go:1014] "Starting client certificate rotation" Oct 30 02:39:57.541951 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.534051 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 02:39:57.541951 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.534097 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 02:39:57.541951 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.536242 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 02:39:57.541951 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.537473 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 02:39:57.548269 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.548250 2576 log.go:25] "Validated CRI v1 runtime API" Oct 30 02:39:57.551568 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.551552 2576 log.go:25] "Validated CRI v1 image API" Oct 30 02:39:57.552522 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.552490 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 02:39:57.558241 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.558219 2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9210d514-1422-44a9-b70e-0566f3bc391b:/dev/nvme0n1p4 b5ffe6e2-0bff-420a-85e3-881607245aee:/dev/nvme0n1p3] Oct 30 02:39:57.558322 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.558239 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 02:39:57.565317 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.565190 2576 manager.go:217] Machine: {Timestamp:2026-10-30 02:39:57.563751177 +0000 UTC m=+0.380577179 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104921 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23303e67edfa373af0977749bcfa97 SystemUUID:ec23303e-67ed-fa37-3af0-977749bcfa97 BootID:97a968b1-eefe-4f25-8bb8-8b2265ab78bb Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fc:b8:b8:5d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fc:b8:b8:5d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:ca:96:02:af:33 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 02:39:57.565317 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.565307 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 02:39:57.565497 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.565424 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 02:39:57.567597 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.567562 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 02:39:57.567738 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.567598 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-19.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 02:39:57.567790 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.567741 2576 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 02:39:57.567790 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.567749 2576 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 02:39:57.567790 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.567774 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 02:39:57.569406 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.569395 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 02:39:57.570639 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.570629 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 02:39:57.575286 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.575267 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 02:39:57.575415 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.575405 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 02:39:57.576239 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.576229 2576 kubelet.go:493] "Attempting to sync node with API server" Oct 30 02:39:57.576290 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.576243 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 02:39:57.576290 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.576254 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 02:39:57.576290 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.576264 2576 kubelet.go:405] "Adding apiserver pod source" Oct 30 02:39:57.576290 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.576273 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 02:39:57.576725 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.576702 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 02:39:57.579343 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.579317 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 02:39:57.587036 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.587021 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 02:39:57.588004 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.587990 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 02:39:57.588046 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.588022 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 02:39:57.588240 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.588224 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 02:39:57.588283 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.588254 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 02:39:57.588283 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.588267 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 02:39:57.588283 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.588279 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 02:39:57.588367 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.588292 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 02:39:57.588367 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.588301 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 02:39:57.589084 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.589075 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 02:39:57.589118 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.589085 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 02:39:57.589118 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.589093 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 02:39:57.589118 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.589099 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 02:39:57.589118 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.589112 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 02:39:57.589285 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.589122 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 02:39:57.590693 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.590672 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 02:39:57.590693 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.590686 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 02:39:57.594943 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.594925 2576 server.go:1263] "Started kubelet" Oct 30 02:39:57.595026 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.594983 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 02:39:57.595092 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.595044 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 02:39:57.595153 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.595123 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 02:39:57.595813 ip-10-0-139-19 systemd[1]: Started Kubernetes Kubelet. Oct 30 02:39:57.597029 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.596697 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 02:39:57.598224 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.598197 2576 server.go:317] "Adding debug handlers to kubelet server" Oct 30 02:39:57.602802 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:57.602781 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 02:39:57.606903 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.606887 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 02:39:57.607453 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.607324 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 02:39:57.607584 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:57.607563 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-19.ec2.internal\" not found" Oct 30 02:39:57.608528 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.608423 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 02:39:57.608528 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.608450 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 02:39:57.608528 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.608467 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 02:39:57.608733 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.608530 2576 factory.go:153] Registering CRI-O factory Oct 30 02:39:57.608733 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.608546 2576 factory.go:223] Registration of the crio container factory successfully Oct 30 02:39:57.608733 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.608567 2576 reconstruct.go:97] "Volume reconstruction finished" Oct 30 02:39:57.608733 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.608575 2576 reconciler.go:29] "Reconciler: start to sync state" Oct 30 02:39:57.608733 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.608612 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 02:39:57.608733 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.608622 2576 factory.go:55] Registering systemd factory Oct 30 02:39:57.608733 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.608630 2576 factory.go:223] Registration of the systemd container factory successfully Oct 30 02:39:57.608733 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.608648 2576 factory.go:103] Registering Raw factory Oct 30 02:39:57.608733 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.608662 2576 manager.go:1199] Started watching for new ooms in manager Oct 30 02:39:57.609903 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.609881 2576 manager.go:319] Starting recovery of all containers Oct 30 02:39:57.618492 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.618441 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 02:39:57.624232 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.624208 2576 manager.go:324] Recovery completed Oct 30 02:39:57.628522 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.628501 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 02:39:57.628630 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.628618 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:39:57.630411 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.630384 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:39:57.630470 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.630434 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:39:57.630470 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.630446 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:39:57.630966 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.630948 2576 cpu_manager.go:226] "Starting" policy="none" Oct 30 02:39:57.630966 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.630961 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:39:57.631090 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.630962 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 02:39:57.631090 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.630999 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 02:39:57.631595 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.631580 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:39:57.631669 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.631604 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:39:57.631669 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.631617 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:39:57.631813 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:57.631791 2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 02:39:57.631813 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.631794 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 02:39:57.632911 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:57.631793 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-19.ec2.internal.18e32e4ff6653d08 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-19.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-19.ec2.internal,},FirstTimestamp:2026-10-30 02:39:57.594713352 +0000 UTC m=+0.411539354,LastTimestamp:2026-10-30 02:39:57.594713352 +0000 UTC m=+0.411539354,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-19.ec2.internal,}" Oct 30 02:39:57.633241 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.633231 2576 policy_none.go:50] "Start" Oct 30 02:39:57.633279 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.633249 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 02:39:57.633279 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.633260 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 02:39:57.635309 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.635294 2576 policy_none.go:44] "Start" Oct 30 02:39:57.651948 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:57.651907 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 02:39:57.652071 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:57.651969 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 02:39:57.652071 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:57.651972 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-19.ec2.internal.18e32e4ff885f9e1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-19.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-19.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-19.ec2.internal,},FirstTimestamp:2026-10-30 02:39:57.630413281 +0000 UTC m=+0.447239286,LastTimestamp:2026-10-30 02:39:57.630413281 +0000 UTC m=+0.447239286,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-19.ec2.internal,}" Oct 30 02:39:57.655652 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.655628 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4hcr8" Oct 30 02:39:57.695102 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.669809 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4hcr8" Oct 30 02:39:57.695102 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:57.678568 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 02:39:57.695102 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:57.678550 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-19.ec2.internal.18e32e4ff8866234 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-19.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-19.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-19.ec2.internal,},FirstTimestamp:2026-10-30 02:39:57.630439988 +0000 UTC m=+0.447265989,LastTimestamp:2026-10-30 02:39:57.630439988 +0000 UTC m=+0.447265989,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-19.ec2.internal,}" Oct 30 02:39:57.695102 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.682831 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 02:39:57.695102 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.685595 2576 manager.go:341] "Starting Device Plugin manager" Oct 30 02:39:57.695102 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:57.685630 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 02:39:57.695102 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.685640 2576 server.go:87] "Starting device plugin registration server" Oct 30 02:39:57.695102 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.685935 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 02:39:57.695102 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.685949 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 02:39:57.695102 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.686109 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 02:39:57.695102 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.686220 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 02:39:57.695102 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.686227 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 02:39:57.695102 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:57.686642 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 02:39:57.695102 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:57.686689 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-19.ec2.internal\" not found" Oct 30 02:39:57.695102 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.687440 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 02:39:57.695102 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.687474 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 02:39:57.695102 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.687496 2576 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 02:39:57.695102 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:57.687538 2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 02:39:57.701660 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.701641 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:39:57.786590 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.786503 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:39:57.788913 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.787906 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal"] Oct 30 02:39:57.788913 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.787997 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:39:57.789427 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.789396 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:39:57.789427 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.789430 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:39:57.789569 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.789440 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:39:57.789569 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.789461 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-19.ec2.internal" Oct 30 02:39:57.789757 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.789743 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:39:57.789813 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.789767 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:39:57.789813 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.789776 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:39:57.791968 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.791956 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:39:57.792119 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.792104 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal" Oct 30 02:39:57.792170 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.792149 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:39:57.792610 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.792593 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:39:57.792681 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.792615 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:39:57.792681 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.792626 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:39:57.792681 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.792632 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:39:57.792681 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.792651 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:39:57.792681 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.792659 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:39:57.794750 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.794737 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" Oct 30 02:39:57.794795 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.794760 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:39:57.795384 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.795368 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:39:57.795432 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.795394 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:39:57.795432 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.795403 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:39:57.809092 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.809077 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-19.ec2.internal" Oct 30 02:39:57.809170 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:57.809098 2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-19.ec2.internal\": node \"ip-10-0-139-19.ec2.internal\" not found" Oct 30 02:39:57.809688 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.809671 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf4d60387e18d9cce88d9d9a653a225b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal\" (UID: \"cf4d60387e18d9cce88d9d9a653a225b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" Oct 30 02:39:57.809723 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.809695 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36dcf6a72ae2b4d471dead03fce936be-config\") pod \"kube-apiserver-proxy-ip-10-0-139-19.ec2.internal\" (UID: \"36dcf6a72ae2b4d471dead03fce936be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal" Oct 30 02:39:57.809723 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.809711 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf4d60387e18d9cce88d9d9a653a225b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal\" (UID: \"cf4d60387e18d9cce88d9d9a653a225b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" Oct 30 02:39:57.824007 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:57.823984 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-19.ec2.internal\" not found" node="ip-10-0-139-19.ec2.internal" Oct 30 02:39:57.831189 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:57.831171 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-19.ec2.internal\" not found" node="ip-10-0-139-19.ec2.internal" Oct 30 02:39:57.847182 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:57.847163 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-19.ec2.internal\" not found" Oct 30 02:39:57.910581 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.910404 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf4d60387e18d9cce88d9d9a653a225b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal\" (UID: \"cf4d60387e18d9cce88d9d9a653a225b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" Oct 30 02:39:57.910581 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.910581 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf4d60387e18d9cce88d9d9a653a225b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal\" (UID: \"cf4d60387e18d9cce88d9d9a653a225b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" Oct 30 02:39:57.910771 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.910597 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36dcf6a72ae2b4d471dead03fce936be-config\") pod \"kube-apiserver-proxy-ip-10-0-139-19.ec2.internal\" (UID: \"36dcf6a72ae2b4d471dead03fce936be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal" Oct 30 02:39:57.910771 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.910522 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf4d60387e18d9cce88d9d9a653a225b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal\" (UID: \"cf4d60387e18d9cce88d9d9a653a225b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" Oct 30 02:39:57.910771 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.910661 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf4d60387e18d9cce88d9d9a653a225b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal\" (UID: \"cf4d60387e18d9cce88d9d9a653a225b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" Oct 30 02:39:57.910771 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:57.910638 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36dcf6a72ae2b4d471dead03fce936be-config\") pod \"kube-apiserver-proxy-ip-10-0-139-19.ec2.internal\" (UID: \"36dcf6a72ae2b4d471dead03fce936be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal" Oct 30 02:39:57.948095 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:57.948071 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-19.ec2.internal\" not found" Oct 30 02:39:58.048487 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:58.048402 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-19.ec2.internal\" not found" Oct 30 02:39:59.518854 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:59.518822 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal" Oct 30 02:39:59.526175 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:59.526153 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" Oct 30 02:39:59.538225 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:59.538206 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-19.ec2.internal\" not found" Oct 30 02:39:59.638816 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:59.638766 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-19.ec2.internal\" not found" Oct 30 02:39:59.739339 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:59.739282 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-19.ec2.internal\" not found" Oct 30 02:39:59.840007 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:59.839973 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-19.ec2.internal\" not found" Oct 30 02:39:59.923510 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:59.923480 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 02:39:59.923971 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:39:59.923650 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 02:39:59.940801 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:39:59.940775 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-19.ec2.internal\" not found" Oct 30 02:40:00.041415 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:00.041337 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-19.ec2.internal\" not found" Oct 30 02:40:00.060685 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:00.060660 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:40:00.065813 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:00.065775 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:34:59 +0000 UTC" deadline="2028-04-16 21:59:31.821598224 +0000 UTC" Oct 30 02:40:00.065813 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:00.065812 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12835h19m31.755788531s" Oct 30 02:40:00.088338 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:40:00.088302 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36dcf6a72ae2b4d471dead03fce936be.slice/crio-821a01fbf56f6767e3d2f96c33227981bf771479c90ab3360e5e0d5908d9a59a WatchSource:0}: Error finding container 821a01fbf56f6767e3d2f96c33227981bf771479c90ab3360e5e0d5908d9a59a: Status 404 returned error can't find the container with id 821a01fbf56f6767e3d2f96c33227981bf771479c90ab3360e5e0d5908d9a59a Oct 30 02:40:00.088812 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:40:00.088781 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4d60387e18d9cce88d9d9a653a225b.slice/crio-205ec985195e5559c3c07665ac1ecf35cf3dcfcc72314eeb7961592015d0b1bc WatchSource:0}: Error finding container 205ec985195e5559c3c07665ac1ecf35cf3dcfcc72314eeb7961592015d0b1bc: Status 404 returned error can't find the container with id 205ec985195e5559c3c07665ac1ecf35cf3dcfcc72314eeb7961592015d0b1bc Oct 30 02:40:00.093058 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:00.093039 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:40:00.141925 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:00.141888 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-19.ec2.internal\" not found" Oct 30 02:40:00.202283 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:00.202260 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:40:00.203512 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:00.203494 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:40:00.203571 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:00.203529 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:40:00.203571 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:00.203540 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:40:00.242775 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:00.242732 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-19.ec2.internal\" not found" Oct 30 02:40:00.343403 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:00.343325 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-19.ec2.internal\" not found" Oct 30 02:40:00.443913 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:00.443883 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-19.ec2.internal\" not found" Oct 30 02:40:00.483327 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:00.483300 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:40:00.543987 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:00.543955 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-19.ec2.internal\" not found" Oct 30 02:40:00.644610 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:00.644535 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-19.ec2.internal\" not found" Oct 30 02:40:00.662191 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:00.662089 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:40:00.698731 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:00.698563 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" Oct 30 02:40:00.722652 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:00.722507 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 02:40:00.723733 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:00.723710 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal" Oct 30 02:40:00.742543 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:00.742516 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 02:40:00.967368 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:00.967304 2576 apiserver.go:52] "Watching apiserver" Oct 30 02:40:00.997942 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:00.997334 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 02:40:01.003289 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.002543 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 02:40:01.003289 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.002930 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-8gjpf","kube-system/global-pull-secret-syncer-m9zrt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf","openshift-cluster-node-tuning-operator/tuned-76vs8","kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal","openshift-image-registry/node-ca-zg4qh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal","openshift-multus/multus-gk587","openshift-network-diagnostics/network-check-target-7rx9c","openshift-ovn-kubernetes/ovnkube-node-b995w","kube-system/konnectivity-agent-n96rg","openshift-dns/node-resolver-hbr7c","openshift-multus/multus-additional-cni-plugins-j5jm9","openshift-multus/network-metrics-daemon-9t4cc"] Oct 30 02:40:01.005643 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.005618 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.008123 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.007859 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n96rg" Oct 30 02:40:01.010763 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.010245 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.012510 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.012483 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.014979 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.014942 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:01.015077 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:01.015010 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t4cc" podUID="e17aabad-c1ca-490d-904d-ddd40b5fd741" Oct 30 02:40:01.017603 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.017585 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zg4qh" Oct 30 02:40:01.019013 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.018995 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 02:40:01.019235 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.019218 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 02:40:01.019235 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.019227 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 02:40:01.019355 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.019294 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 02:40:01.019355 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.019233 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 02:40:01.019355 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.019333 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xkr2x\"" Oct 30 02:40:01.019959 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.019942 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:01.020044 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:01.020021 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rx9c" podUID="ee2a4d7e-545a-4946-a44d-cc895441c477" Oct 30 02:40:01.020518 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.020488 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 02:40:01.022654 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.022633 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.023768 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.023738 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 02:40:01.023881 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.023868 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 02:40:01.023963 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.023928 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 02:40:01.024025 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.023950 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 02:40:01.024025 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.023951 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 02:40:01.024025 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.023950 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nmhn4\"" Oct 30 02:40:01.024025 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.024021 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cb5f2\"" Oct 30 02:40:01.024827 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.024812 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:01.024920 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:01.024900 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9zrt" podUID="ef5e843c-9ceb-44f3-bfc9-c56be367804e" Oct 30 02:40:01.025208 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.025188 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 02:40:01.025296 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.025274 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q9zqs\"" Oct 30 02:40:01.025353 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.025292 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:40:01.027432 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.027117 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8gjpf" Oct 30 02:40:01.029492 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.029473 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gk587" Oct 30 02:40:01.031567 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.031550 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 02:40:01.032057 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.032038 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hbr7c" Oct 30 02:40:01.038972 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.038954 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 02:40:01.039713 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.039693 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kn8pn" Oct 30 02:40:01.048675 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.048658 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kn8pn" Oct 30 02:40:01.085464 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.085385 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal" event={"ID":"36dcf6a72ae2b4d471dead03fce936be","Type":"ContainerStarted","Data":"821a01fbf56f6767e3d2f96c33227981bf771479c90ab3360e5e0d5908d9a59a"} Oct 30 02:40:01.086921 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.086898 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" event={"ID":"cf4d60387e18d9cce88d9d9a653a225b","Type":"ContainerStarted","Data":"205ec985195e5559c3c07665ac1ecf35cf3dcfcc72314eeb7961592015d0b1bc"} Oct 30 02:40:01.100187 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.100163 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 02:40:01.110573 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.110544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-var-lib-cni-multus\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.110677 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.110584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/183230bf-332c-465f-8525-2094db22e38a-multus-daemon-config\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.110677 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.110609 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.110677 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.110631 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-host\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.110677 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.110653 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/164920df-32a0-48ab-830a-171a05caab08-host-slash\") pod \"iptables-alerter-8gjpf\" (UID: \"164920df-32a0-48ab-830a-171a05caab08\") " pod="openshift-network-operator/iptables-alerter-8gjpf" Oct 30 02:40:01.110677 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.110673 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh7dx\" (UniqueName: \"kubernetes.io/projected/164920df-32a0-48ab-830a-171a05caab08-kube-api-access-gh7dx\") pod \"iptables-alerter-8gjpf\" (UID: \"164920df-32a0-48ab-830a-171a05caab08\") " pod="openshift-network-operator/iptables-alerter-8gjpf" Oct 30 02:40:01.110959 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.110694 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-run-netns\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.110959 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.110717 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-cni-binary-copy\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.110959 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.110742 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-modprobe-d\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.110959 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.110783 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-lib-modules\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.110959 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.110810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-var-lib-cni-bin\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.110959 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.110838 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-slash\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.110959 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.110861 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-run-ovn-kubernetes\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.110959 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.110886 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-cni-netd\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.110959 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.110908 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3c34ce6-91f3-42e4-ba49-a905f477571e-host\") pod \"node-ca-zg4qh\" (UID: \"a3c34ce6-91f3-42e4-ba49-a905f477571e\") " pod="openshift-image-registry/node-ca-zg4qh" Oct 30 02:40:01.110959 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.110929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-node-log\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.110959 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.110952 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-var-lib-kubelet\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.111467 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.110978 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-etc-kubernetes\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.111467 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111002 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-cni-bin\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.111467 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111025 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f17a0e8-ce89-4a88-98f2-c7782260534d-agent-certs\") pod \"konnectivity-agent-n96rg\" (UID: \"8f17a0e8-ce89-4a88-98f2-c7782260534d\") " pod="kube-system/konnectivity-agent-n96rg" Oct 30 02:40:01.111467 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111050 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-cnibin\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.111467 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-kubernetes\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.111467 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111096 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvtmv\" (UniqueName: \"kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv\") pod \"network-check-target-7rx9c\" (UID: \"ee2a4d7e-545a-4946-a44d-cc895441c477\") " pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:01.111467 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-multus-socket-dir-parent\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.111467 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111143 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e9def4d-df87-404d-90eb-1b21b023fbd8-tmp-dir\") pod \"node-resolver-hbr7c\" (UID: \"5e9def4d-df87-404d-90eb-1b21b023fbd8\") " pod="openshift-dns/node-resolver-hbr7c" Oct 30 02:40:01.111467 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111167 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-run-ovn\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.111467 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-system-cni-dir\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.111467 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111215 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-sysconfig\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.111467 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111238 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-tuned\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.111467 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111264 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef5e843c-9ceb-44f3-bfc9-c56be367804e-kubelet-config\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:01.111467 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-system-cni-dir\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.111467 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111312 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-cnibin\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.111467 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111336 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-run-multus-certs\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.111467 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111361 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5e9def4d-df87-404d-90eb-1b21b023fbd8-hosts-file\") pod \"node-resolver-hbr7c\" (UID: \"5e9def4d-df87-404d-90eb-1b21b023fbd8\") " pod="openshift-dns/node-resolver-hbr7c" Oct 30 02:40:01.112256 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111391 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2b91cd0-34ba-453f-8088-13e814a3a367-tmp\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.112256 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111416 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:01.112256 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111441 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:01.112256 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111467 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/183230bf-332c-465f-8525-2094db22e38a-cni-binary-copy\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.112256 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111490 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-var-lib-kubelet\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.112256 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111514 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.112256 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-etc-selinux\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.112256 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111566 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.112256 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111594 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjbgs\" (UniqueName: \"kubernetes.io/projected/183230bf-332c-465f-8525-2094db22e38a-kube-api-access-tjbgs\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.112256 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-etc-openvswitch\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.112256 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111644 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-sys-fs\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.112256 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.112256 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111692 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-run-netns\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.112256 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kzk2\" (UniqueName: \"kubernetes.io/projected/e17aabad-c1ca-490d-904d-ddd40b5fd741-kube-api-access-2kzk2\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:01.112256 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111741 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-multus-cni-dir\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.112256 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111781 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-systemd-units\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.113009 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111807 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-log-socket\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.113009 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111830 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-socket-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.113009 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111856 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-device-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.113009 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111883 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw82d\" (UniqueName: \"kubernetes.io/projected/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-kube-api-access-vw82d\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.113009 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111908 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-sysctl-conf\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.113009 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111941 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-systemd\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.113009 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6llj\" (UniqueName: \"kubernetes.io/projected/5e9def4d-df87-404d-90eb-1b21b023fbd8-kube-api-access-m6llj\") pod \"node-resolver-hbr7c\" (UID: \"5e9def4d-df87-404d-90eb-1b21b023fbd8\") " pod="openshift-dns/node-resolver-hbr7c" Oct 30 02:40:01.113009 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.111997 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1bac52de-7a27-4fce-828b-9eedcfd60de5-env-overrides\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.113009 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112024 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djdvk\" (UniqueName: \"kubernetes.io/projected/06222d65-ed37-48ff-9b12-19271e1f9662-kube-api-access-djdvk\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.113009 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112051 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.113009 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112077 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wkjm\" (UniqueName: \"kubernetes.io/projected/a3c34ce6-91f3-42e4-ba49-a905f477571e-kube-api-access-4wkjm\") pod \"node-ca-zg4qh\" (UID: \"a3c34ce6-91f3-42e4-ba49-a905f477571e\") " pod="openshift-image-registry/node-ca-zg4qh" Oct 30 02:40:01.113009 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-var-lib-openvswitch\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.113009 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rckxh\" (UniqueName: \"kubernetes.io/projected/1bac52de-7a27-4fce-828b-9eedcfd60de5-kube-api-access-rckxh\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.113009 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112153 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-registration-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.113009 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112178 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-sysctl-d\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.113009 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112203 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-run\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.113767 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112231 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-hostroot\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.113767 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-multus-conf-dir\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.113767 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-run-systemd\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.113767 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112311 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f17a0e8-ce89-4a88-98f2-c7782260534d-konnectivity-ca\") pod \"konnectivity-agent-n96rg\" (UID: \"8f17a0e8-ce89-4a88-98f2-c7782260534d\") " pod="kube-system/konnectivity-agent-n96rg" Oct 30 02:40:01.113767 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bjwl\" (UniqueName: \"kubernetes.io/projected/c2b91cd0-34ba-453f-8088-13e814a3a367-kube-api-access-4bjwl\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.113767 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112361 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1bac52de-7a27-4fce-828b-9eedcfd60de5-ovnkube-config\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.113767 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1bac52de-7a27-4fce-828b-9eedcfd60de5-ovnkube-script-lib\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.113767 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112413 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-sys\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.113767 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3c34ce6-91f3-42e4-ba49-a905f477571e-serviceca\") pod \"node-ca-zg4qh\" (UID: \"a3c34ce6-91f3-42e4-ba49-a905f477571e\") " pod="openshift-image-registry/node-ca-zg4qh" Oct 30 02:40:01.113767 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112463 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-kubelet\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.113767 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-run-openvswitch\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.113767 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112513 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bac52de-7a27-4fce-828b-9eedcfd60de5-ovn-node-metrics-cert\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.113767 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef5e843c-9ceb-44f3-bfc9-c56be367804e-dbus\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:01.113767 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112564 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/164920df-32a0-48ab-830a-171a05caab08-iptables-alerter-script\") pod \"iptables-alerter-8gjpf\" (UID: \"164920df-32a0-48ab-830a-171a05caab08\") " pod="openshift-network-operator/iptables-alerter-8gjpf" Oct 30 02:40:01.113767 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.112589 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-run-k8s-cni-cncf-io\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.120090 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.120057 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 02:40:01.120182 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.120127 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 02:40:01.120442 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.120427 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dfk55\"" Oct 30 02:40:01.120544 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.120469 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 02:40:01.120605 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.120553 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 02:40:01.120658 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.120613 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 02:40:01.120728 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.120713 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jvpkf\"" Oct 30 02:40:01.120811 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.120772 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 02:40:01.120863 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.120852 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 02:40:01.120959 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.120941 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 02:40:01.121059 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.121027 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 02:40:01.121110 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.121085 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 02:40:01.121155 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.121141 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 02:40:01.121276 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.121264 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2trkp\"" Oct 30 02:40:01.121323 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.121314 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-74jzq\"" Oct 30 02:40:01.121365 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.121343 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 02:40:01.121365 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.121364 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:40:01.121431 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.121405 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nd26x\"" Oct 30 02:40:01.213341 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213304 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjbgs\" (UniqueName: \"kubernetes.io/projected/183230bf-332c-465f-8525-2094db22e38a-kube-api-access-tjbgs\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.213525 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213349 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-etc-openvswitch\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.213525 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213373 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-sys-fs\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.213525 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213399 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.213525 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-run-netns\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.213525 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213449 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kzk2\" (UniqueName: \"kubernetes.io/projected/e17aabad-c1ca-490d-904d-ddd40b5fd741-kube-api-access-2kzk2\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:01.213525 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213479 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-multus-cni-dir\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.213525 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213502 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-systemd-units\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.213525 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-log-socket\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.213835 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-socket-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.213835 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213569 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-device-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.213835 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213592 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw82d\" (UniqueName: \"kubernetes.io/projected/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-kube-api-access-vw82d\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.213835 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-sysctl-conf\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.213835 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-systemd\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.213835 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213668 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6llj\" (UniqueName: \"kubernetes.io/projected/5e9def4d-df87-404d-90eb-1b21b023fbd8-kube-api-access-m6llj\") pod \"node-resolver-hbr7c\" (UID: \"5e9def4d-df87-404d-90eb-1b21b023fbd8\") " pod="openshift-dns/node-resolver-hbr7c" Oct 30 02:40:01.213835 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1bac52de-7a27-4fce-828b-9eedcfd60de5-env-overrides\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.213835 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdvk\" (UniqueName: \"kubernetes.io/projected/06222d65-ed37-48ff-9b12-19271e1f9662-kube-api-access-djdvk\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.213835 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213740 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.213835 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213785 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wkjm\" (UniqueName: \"kubernetes.io/projected/a3c34ce6-91f3-42e4-ba49-a905f477571e-kube-api-access-4wkjm\") pod \"node-ca-zg4qh\" (UID: \"a3c34ce6-91f3-42e4-ba49-a905f477571e\") " pod="openshift-image-registry/node-ca-zg4qh" Oct 30 02:40:01.213835 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-var-lib-openvswitch\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.213835 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213819 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-etc-openvswitch\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.213835 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rckxh\" (UniqueName: \"kubernetes.io/projected/1bac52de-7a27-4fce-828b-9eedcfd60de5-kube-api-access-rckxh\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.214301 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213859 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-registration-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.214301 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-sysctl-d\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.214301 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-run\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.214301 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-hostroot\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.214301 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-multus-conf-dir\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.214301 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.213977 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-run-systemd\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.214301 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214000 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f17a0e8-ce89-4a88-98f2-c7782260534d-konnectivity-ca\") pod \"konnectivity-agent-n96rg\" (UID: \"8f17a0e8-ce89-4a88-98f2-c7782260534d\") " pod="kube-system/konnectivity-agent-n96rg" Oct 30 02:40:01.214301 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214024 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bjwl\" (UniqueName: \"kubernetes.io/projected/c2b91cd0-34ba-453f-8088-13e814a3a367-kube-api-access-4bjwl\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.214301 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1bac52de-7a27-4fce-828b-9eedcfd60de5-ovnkube-config\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.214301 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1bac52de-7a27-4fce-828b-9eedcfd60de5-ovnkube-script-lib\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.214301 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-sys\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.214301 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214120 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3c34ce6-91f3-42e4-ba49-a905f477571e-serviceca\") pod \"node-ca-zg4qh\" (UID: \"a3c34ce6-91f3-42e4-ba49-a905f477571e\") " pod="openshift-image-registry/node-ca-zg4qh" Oct 30 02:40:01.214301 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214144 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-kubelet\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.214301 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214167 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-run-openvswitch\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.214301 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bac52de-7a27-4fce-828b-9eedcfd60de5-ovn-node-metrics-cert\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.214301 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214211 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef5e843c-9ceb-44f3-bfc9-c56be367804e-dbus\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:01.214301 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214239 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/164920df-32a0-48ab-830a-171a05caab08-iptables-alerter-script\") pod \"iptables-alerter-8gjpf\" (UID: \"164920df-32a0-48ab-830a-171a05caab08\") " pod="openshift-network-operator/iptables-alerter-8gjpf" Oct 30 02:40:01.214831 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214265 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-run-k8s-cni-cncf-io\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.214831 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214291 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-var-lib-cni-multus\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.214831 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214316 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/183230bf-332c-465f-8525-2094db22e38a-multus-daemon-config\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.214831 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.214831 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-host\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.214831 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214391 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/164920df-32a0-48ab-830a-171a05caab08-host-slash\") pod \"iptables-alerter-8gjpf\" (UID: \"164920df-32a0-48ab-830a-171a05caab08\") " pod="openshift-network-operator/iptables-alerter-8gjpf" Oct 30 02:40:01.214831 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214415 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh7dx\" (UniqueName: \"kubernetes.io/projected/164920df-32a0-48ab-830a-171a05caab08-kube-api-access-gh7dx\") pod \"iptables-alerter-8gjpf\" (UID: \"164920df-32a0-48ab-830a-171a05caab08\") " pod="openshift-network-operator/iptables-alerter-8gjpf" Oct 30 02:40:01.214831 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-run-netns\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.214831 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214447 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.214831 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214467 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-cni-binary-copy\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.214831 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214492 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-modprobe-d\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.214831 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214522 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-run-netns\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.214831 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214530 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-lib-modules\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.214831 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214557 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-var-lib-cni-bin\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.214831 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214563 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-run-systemd\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.214831 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-slash\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.214831 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214611 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-run-ovn-kubernetes\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.215515 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-cni-netd\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.215515 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214651 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3c34ce6-91f3-42e4-ba49-a905f477571e-host\") pod \"node-ca-zg4qh\" (UID: \"a3c34ce6-91f3-42e4-ba49-a905f477571e\") " pod="openshift-image-registry/node-ca-zg4qh" Oct 30 02:40:01.215515 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-node-log\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.215515 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214712 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-node-log\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.215515 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214722 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-var-lib-kubelet\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.215515 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214778 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-systemd-units\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.215515 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214783 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-multus-cni-dir\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.215515 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214847 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-device-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.215515 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214885 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-socket-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.215515 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214926 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-log-socket\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.215515 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214965 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-var-lib-openvswitch\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.215515 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.215089 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-sysctl-conf\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.215515 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.215137 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-systemd\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.215515 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.215247 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f17a0e8-ce89-4a88-98f2-c7782260534d-konnectivity-ca\") pod \"konnectivity-agent-n96rg\" (UID: \"8f17a0e8-ce89-4a88-98f2-c7782260534d\") " pod="kube-system/konnectivity-agent-n96rg" Oct 30 02:40:01.215515 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.215253 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-sys-fs\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.215515 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.215294 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-etc-kubernetes\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.215515 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.215310 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-registration-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.215515 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.215431 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-sysctl-d\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.216243 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.215466 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-run\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.216243 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.215490 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-hostroot\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.216243 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.215731 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1bac52de-7a27-4fce-828b-9eedcfd60de5-env-overrides\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.216243 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.215950 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1bac52de-7a27-4fce-828b-9eedcfd60de5-ovnkube-config\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.216243 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.215993 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/164920df-32a0-48ab-830a-171a05caab08-host-slash\") pod \"iptables-alerter-8gjpf\" (UID: \"164920df-32a0-48ab-830a-171a05caab08\") " pod="openshift-network-operator/iptables-alerter-8gjpf" Oct 30 02:40:01.216243 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216001 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-multus-conf-dir\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.216243 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216001 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1bac52de-7a27-4fce-828b-9eedcfd60de5-ovnkube-script-lib\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.216243 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216029 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-sys\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.216243 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216190 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-run-netns\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.216553 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216262 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-var-lib-kubelet\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.216553 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216315 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-modprobe-d\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.216553 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216407 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-lib-modules\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.216553 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216419 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.216553 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216449 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-var-lib-cni-bin\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.216553 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216494 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-slash\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.216553 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216537 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-run-ovn-kubernetes\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.216844 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216578 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-cni-netd\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.216844 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216616 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3c34ce6-91f3-42e4-ba49-a905f477571e-host\") pod \"node-ca-zg4qh\" (UID: \"a3c34ce6-91f3-42e4-ba49-a905f477571e\") " pod="openshift-image-registry/node-ca-zg4qh" Oct 30 02:40:01.216844 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.214780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-etc-kubernetes\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.216844 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-cni-bin\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.216844 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216679 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f17a0e8-ce89-4a88-98f2-c7782260534d-agent-certs\") pod \"konnectivity-agent-n96rg\" (UID: \"8f17a0e8-ce89-4a88-98f2-c7782260534d\") " pod="kube-system/konnectivity-agent-n96rg" Oct 30 02:40:01.216844 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-cnibin\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.216844 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216728 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-kubernetes\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.216844 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvtmv\" (UniqueName: \"kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv\") pod \"network-check-target-7rx9c\" (UID: \"ee2a4d7e-545a-4946-a44d-cc895441c477\") " pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:01.216844 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216791 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3c34ce6-91f3-42e4-ba49-a905f477571e-serviceca\") pod \"node-ca-zg4qh\" (UID: \"a3c34ce6-91f3-42e4-ba49-a905f477571e\") " pod="openshift-image-registry/node-ca-zg4qh" Oct 30 02:40:01.216844 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-multus-socket-dir-parent\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.217295 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216847 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-run-k8s-cni-cncf-io\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.217295 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216858 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-multus-socket-dir-parent\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.217295 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216879 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e9def4d-df87-404d-90eb-1b21b023fbd8-tmp-dir\") pod \"node-resolver-hbr7c\" (UID: \"5e9def4d-df87-404d-90eb-1b21b023fbd8\") " pod="openshift-dns/node-resolver-hbr7c" Oct 30 02:40:01.217295 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216898 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-cni-bin\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.217295 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-run-ovn\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.217295 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-system-cni-dir\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.217295 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-sysconfig\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.217295 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-tuned\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.217295 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.217005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef5e843c-9ceb-44f3-bfc9-c56be367804e-kubelet-config\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:01.217295 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.217028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-system-cni-dir\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.217295 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.217050 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-cnibin\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.217295 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.217081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-run-multus-certs\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.217295 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.217104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5e9def4d-df87-404d-90eb-1b21b023fbd8-hosts-file\") pod \"node-resolver-hbr7c\" (UID: \"5e9def4d-df87-404d-90eb-1b21b023fbd8\") " pod="openshift-dns/node-resolver-hbr7c" Oct 30 02:40:01.217295 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.217126 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2b91cd0-34ba-453f-8088-13e814a3a367-tmp\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.217295 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.217151 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:01.217295 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.217159 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/164920df-32a0-48ab-830a-171a05caab08-iptables-alerter-script\") pod \"iptables-alerter-8gjpf\" (UID: \"164920df-32a0-48ab-830a-171a05caab08\") " pod="openshift-network-operator/iptables-alerter-8gjpf" Oct 30 02:40:01.217295 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.217177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:01.217295 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.217189 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 02:40:01.219287 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.217214 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-kubelet\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.219287 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.217246 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-run-openvswitch\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.219287 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.217605 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-host\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.219287 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.217657 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-var-lib-cni-multus\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.219287 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.217681 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/183230bf-332c-465f-8525-2094db22e38a-cni-binary-copy\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.219287 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.217725 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.219287 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:01.217827 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:01.219287 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:01.217892 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs podName:e17aabad-c1ca-490d-904d-ddd40b5fd741 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:01.717872457 +0000 UTC m=+3.145082414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs") pod "network-metrics-daemon-9t4cc" (UID: "e17aabad-c1ca-490d-904d-ddd40b5fd741") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:01.219287 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.217910 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5e9def4d-df87-404d-90eb-1b21b023fbd8-hosts-file\") pod \"node-resolver-hbr7c\" (UID: \"5e9def4d-df87-404d-90eb-1b21b023fbd8\") " pod="openshift-dns/node-resolver-hbr7c" Oct 30 02:40:01.219287 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:01.217954 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:01.219287 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.217963 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-system-cni-dir\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.219287 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.218007 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-cnibin\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.219287 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:01.218043 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret podName:ef5e843c-9ceb-44f3-bfc9-c56be367804e nodeName:}" failed. No retries permitted until 2026-10-30 02:40:01.71802665 +0000 UTC m=+3.145236593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret") pod "global-pull-secret-syncer-m9zrt" (UID: "ef5e843c-9ceb-44f3-bfc9-c56be367804e") : object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:01.219287 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.218043 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-run-multus-certs\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.219287 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.216682 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-cni-binary-copy\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.219287 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.218094 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-cnibin\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.219287 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.218151 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef5e843c-9ceb-44f3-bfc9-c56be367804e-dbus\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:01.220132 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.218172 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/183230bf-332c-465f-8525-2094db22e38a-multus-daemon-config\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.220132 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.218206 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-kubernetes\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.220132 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.218252 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-run-ovn\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.220132 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.218256 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef5e843c-9ceb-44f3-bfc9-c56be367804e-kubelet-config\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:01.220132 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.218295 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-system-cni-dir\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.220132 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.218497 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e9def4d-df87-404d-90eb-1b21b023fbd8-tmp-dir\") pod \"node-resolver-hbr7c\" (UID: \"5e9def4d-df87-404d-90eb-1b21b023fbd8\") " pod="openshift-dns/node-resolver-hbr7c" Oct 30 02:40:01.220132 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.218513 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-sysconfig\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.220132 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.217207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/183230bf-332c-465f-8525-2094db22e38a-cni-binary-copy\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.220132 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.218583 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-var-lib-kubelet\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.220132 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.218627 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.220132 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.218684 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.220132 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.218715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-etc-selinux\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.220132 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.218770 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-var-lib-kubelet\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.220132 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.218793 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.220132 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.218845 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-etc-selinux\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.220132 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.218954 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.221218 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.221133 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2b91cd0-34ba-453f-8088-13e814a3a367-tmp\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.221690 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.221667 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bac52de-7a27-4fce-828b-9eedcfd60de5-ovn-node-metrics-cert\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.223282 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.223261 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-tuned\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.223534 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.223507 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f17a0e8-ce89-4a88-98f2-c7782260534d-agent-certs\") pod \"konnectivity-agent-n96rg\" (UID: \"8f17a0e8-ce89-4a88-98f2-c7782260534d\") " pod="kube-system/konnectivity-agent-n96rg" Oct 30 02:40:01.235837 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.235814 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kzk2\" (UniqueName: \"kubernetes.io/projected/e17aabad-c1ca-490d-904d-ddd40b5fd741-kube-api-access-2kzk2\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:01.236542 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.236494 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjbgs\" (UniqueName: \"kubernetes.io/projected/183230bf-332c-465f-8525-2094db22e38a-kube-api-access-tjbgs\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:01.236645 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.236627 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bjwl\" (UniqueName: \"kubernetes.io/projected/c2b91cd0-34ba-453f-8088-13e814a3a367-kube-api-access-4bjwl\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.237236 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:01.237139 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 02:40:01.237236 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:01.237165 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 02:40:01.237236 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:01.237181 2576 projected.go:196] Error preparing data for projected volume kube-api-access-qvtmv for pod openshift-network-diagnostics/network-check-target-7rx9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:01.237236 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:01.237239 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv podName:ee2a4d7e-545a-4946-a44d-cc895441c477 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:01.737222342 +0000 UTC m=+3.164432290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qvtmv" (UniqueName: "kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv") pod "network-check-target-7rx9c" (UID: "ee2a4d7e-545a-4946-a44d-cc895441c477") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:01.237790 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.237723 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djdvk\" (UniqueName: \"kubernetes.io/projected/06222d65-ed37-48ff-9b12-19271e1f9662-kube-api-access-djdvk\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.237790 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.237726 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6llj\" (UniqueName: \"kubernetes.io/projected/5e9def4d-df87-404d-90eb-1b21b023fbd8-kube-api-access-m6llj\") pod \"node-resolver-hbr7c\" (UID: \"5e9def4d-df87-404d-90eb-1b21b023fbd8\") " pod="openshift-dns/node-resolver-hbr7c" Oct 30 02:40:01.238777 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.238738 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wkjm\" (UniqueName: \"kubernetes.io/projected/a3c34ce6-91f3-42e4-ba49-a905f477571e-kube-api-access-4wkjm\") pod \"node-ca-zg4qh\" (UID: \"a3c34ce6-91f3-42e4-ba49-a905f477571e\") " pod="openshift-image-registry/node-ca-zg4qh" Oct 30 02:40:01.238855 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.238738 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh7dx\" (UniqueName: \"kubernetes.io/projected/164920df-32a0-48ab-830a-171a05caab08-kube-api-access-gh7dx\") pod \"iptables-alerter-8gjpf\" (UID: \"164920df-32a0-48ab-830a-171a05caab08\") " pod="openshift-network-operator/iptables-alerter-8gjpf" Oct 30 02:40:01.239301 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.239277 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw82d\" (UniqueName: \"kubernetes.io/projected/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-kube-api-access-vw82d\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.242896 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.242875 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rckxh\" (UniqueName: \"kubernetes.io/projected/1bac52de-7a27-4fce-828b-9eedcfd60de5-kube-api-access-rckxh\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.341833 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.341803 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:01.343941 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.343912 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n96rg" Oct 30 02:40:01.350512 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.350485 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:01.350880 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.350855 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:01.354101 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.354081 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zg4qh" Oct 30 02:40:01.362900 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.362879 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:40:01.373591 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.373571 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8gjpf" Oct 30 02:40:01.375990 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.375970 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:01.376073 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:40:01.376011 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bac52de_7a27_4fce_828b_9eedcfd60de5.slice/crio-e5c5c48eb62d87c86e2f5f5edfc784e6d16c943c7ce87bbaa00e34c55763d4f2 WatchSource:0}: Error finding container e5c5c48eb62d87c86e2f5f5edfc784e6d16c943c7ce87bbaa00e34c55763d4f2: Status 404 returned error can't find the container with id e5c5c48eb62d87c86e2f5f5edfc784e6d16c943c7ce87bbaa00e34c55763d4f2 Oct 30 02:40:01.378404 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.378384 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gk587" Oct 30 02:40:01.381542 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.381520 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hbr7c" Oct 30 02:40:01.382040 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:40:01.382016 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f17a0e8_ce89_4a88_98f2_c7782260534d.slice/crio-2252f4bec55cb55bf6e70ece220dec0aef9bc971cffb1f483b6a7d363f960c62 WatchSource:0}: Error finding container 2252f4bec55cb55bf6e70ece220dec0aef9bc971cffb1f483b6a7d363f960c62: Status 404 returned error can't find the container with id 2252f4bec55cb55bf6e70ece220dec0aef9bc971cffb1f483b6a7d363f960c62 Oct 30 02:40:01.400000 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:40:01.399975 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06222d65_ed37_48ff_9b12_19271e1f9662.slice/crio-631d8a03a67067aed1a5182295c2e02528b887eeeef12b507519128a5056b9e6 WatchSource:0}: Error finding container 631d8a03a67067aed1a5182295c2e02528b887eeeef12b507519128a5056b9e6: Status 404 returned error can't find the container with id 631d8a03a67067aed1a5182295c2e02528b887eeeef12b507519128a5056b9e6 Oct 30 02:40:01.403995 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:40:01.402872 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8053d7_4573_4d8a_b8aa_54ad6543d84f.slice/crio-d381d99baff9993609cbfad7dadb508331dcdab3c39ecb12958af1b9b47231c1 WatchSource:0}: Error finding container d381d99baff9993609cbfad7dadb508331dcdab3c39ecb12958af1b9b47231c1: Status 404 returned error can't find the container with id d381d99baff9993609cbfad7dadb508331dcdab3c39ecb12958af1b9b47231c1 Oct 30 02:40:01.406622 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:40:01.406600 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b91cd0_34ba_453f_8088_13e814a3a367.slice/crio-20effaedab036ff03d98b3d22baba22f29db49b77ab7c605c8f5d1ea04a26e9b WatchSource:0}: Error finding container 20effaedab036ff03d98b3d22baba22f29db49b77ab7c605c8f5d1ea04a26e9b: Status 404 returned error can't find the container with id 20effaedab036ff03d98b3d22baba22f29db49b77ab7c605c8f5d1ea04a26e9b Oct 30 02:40:01.408369 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:40:01.408321 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183230bf_332c_465f_8525_2094db22e38a.slice/crio-e2fbcb710ca85ed9015e60ab5d3b24f22a3f019a47fa9b88ad70d270ec92383a WatchSource:0}: Error finding container e2fbcb710ca85ed9015e60ab5d3b24f22a3f019a47fa9b88ad70d270ec92383a: Status 404 returned error can't find the container with id e2fbcb710ca85ed9015e60ab5d3b24f22a3f019a47fa9b88ad70d270ec92383a Oct 30 02:40:01.409053 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:40:01.409025 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9def4d_df87_404d_90eb_1b21b023fbd8.slice/crio-17a3b3a87deed02ee0d9fac868e411b2563d287e9f7c6b0c2015afc561f02154 WatchSource:0}: Error finding container 17a3b3a87deed02ee0d9fac868e411b2563d287e9f7c6b0c2015afc561f02154: Status 404 returned error can't find the container with id 17a3b3a87deed02ee0d9fac868e411b2563d287e9f7c6b0c2015afc561f02154 Oct 30 02:40:01.722828 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.722610 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:01.722978 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.722834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:01.722978 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:01.722764 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:01.722978 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:01.722927 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret podName:ef5e843c-9ceb-44f3-bfc9-c56be367804e nodeName:}" failed. No retries permitted until 2026-10-30 02:40:02.72291262 +0000 UTC m=+4.150122563 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret") pod "global-pull-secret-syncer-m9zrt" (UID: "ef5e843c-9ceb-44f3-bfc9-c56be367804e") : object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:01.722978 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:01.722951 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:01.723141 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:01.723007 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs podName:e17aabad-c1ca-490d-904d-ddd40b5fd741 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:02.722992389 +0000 UTC m=+4.150202346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs") pod "network-metrics-daemon-9t4cc" (UID: "e17aabad-c1ca-490d-904d-ddd40b5fd741") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:01.824412 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:01.823710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvtmv\" (UniqueName: \"kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv\") pod \"network-check-target-7rx9c\" (UID: \"ee2a4d7e-545a-4946-a44d-cc895441c477\") " pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:01.824412 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:01.823936 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 02:40:01.824412 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:01.823957 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 02:40:01.824412 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:01.823970 2576 projected.go:196] Error preparing data for projected volume kube-api-access-qvtmv for pod openshift-network-diagnostics/network-check-target-7rx9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:01.824412 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:01.824030 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv podName:ee2a4d7e-545a-4946-a44d-cc895441c477 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:02.824011112 +0000 UTC m=+4.251221062 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvtmv" (UniqueName: "kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv") pod "network-check-target-7rx9c" (UID: "ee2a4d7e-545a-4946-a44d-cc895441c477") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:02.051859 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:02.051718 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:35:01 +0000 UTC" deadline="2028-05-04 23:48:21.799880488 +0000 UTC" Oct 30 02:40:02.051859 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:02.051797 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13269h8m19.748088392s" Oct 30 02:40:02.112502 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:02.111686 2576 generic.go:356] "Generic (PLEG): container finished" podID="cf4d60387e18d9cce88d9d9a653a225b" containerID="5dbd685629ba6c174535543bd936fc9ebb44c53654687950c5ebf1a3d2e1312b" exitCode=0 Oct 30 02:40:02.112502 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:02.111804 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" event={"ID":"cf4d60387e18d9cce88d9d9a653a225b","Type":"ContainerDied","Data":"5dbd685629ba6c174535543bd936fc9ebb44c53654687950c5ebf1a3d2e1312b"} Oct 30 02:40:02.124134 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:02.124100 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n96rg" event={"ID":"8f17a0e8-ce89-4a88-98f2-c7782260534d","Type":"ContainerStarted","Data":"2252f4bec55cb55bf6e70ece220dec0aef9bc971cffb1f483b6a7d363f960c62"} Oct 30 02:40:02.134390 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:02.134360 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" event={"ID":"1bac52de-7a27-4fce-828b-9eedcfd60de5","Type":"ContainerStarted","Data":"e5c5c48eb62d87c86e2f5f5edfc784e6d16c943c7ce87bbaa00e34c55763d4f2"} Oct 30 02:40:02.150613 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:02.150529 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zg4qh" event={"ID":"a3c34ce6-91f3-42e4-ba49-a905f477571e","Type":"ContainerStarted","Data":"ed70e3f25a89f353dd0328f78fa5b6fe4bb43ad9a65ec276e26ab3f525cf3bb5"} Oct 30 02:40:02.157620 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:02.157509 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hbr7c" event={"ID":"5e9def4d-df87-404d-90eb-1b21b023fbd8","Type":"ContainerStarted","Data":"17a3b3a87deed02ee0d9fac868e411b2563d287e9f7c6b0c2015afc561f02154"} Oct 30 02:40:02.164697 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:02.164663 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gk587" event={"ID":"183230bf-332c-465f-8525-2094db22e38a","Type":"ContainerStarted","Data":"e2fbcb710ca85ed9015e60ab5d3b24f22a3f019a47fa9b88ad70d270ec92383a"} Oct 30 02:40:02.177572 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:02.177535 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-76vs8" event={"ID":"c2b91cd0-34ba-453f-8088-13e814a3a367","Type":"ContainerStarted","Data":"20effaedab036ff03d98b3d22baba22f29db49b77ab7c605c8f5d1ea04a26e9b"} Oct 30 02:40:02.197008 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:02.196891 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5jm9" event={"ID":"fc8053d7-4573-4d8a-b8aa-54ad6543d84f","Type":"ContainerStarted","Data":"d381d99baff9993609cbfad7dadb508331dcdab3c39ecb12958af1b9b47231c1"} Oct 30 02:40:02.208447 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:02.208409 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" event={"ID":"06222d65-ed37-48ff-9b12-19271e1f9662","Type":"ContainerStarted","Data":"631d8a03a67067aed1a5182295c2e02528b887eeeef12b507519128a5056b9e6"} Oct 30 02:40:02.223206 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:02.223167 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8gjpf" event={"ID":"164920df-32a0-48ab-830a-171a05caab08","Type":"ContainerStarted","Data":"2b75e3b02471030d2f34e5c1c6e7fedcc8bf89131b0d82e8a9a5d6798fca3293"} Oct 30 02:40:02.250341 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:02.250301 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal" event={"ID":"36dcf6a72ae2b4d471dead03fce936be","Type":"ContainerStarted","Data":"681e4122a6d10a058855b024c38aa4bdf2d89e24d7b44a1b5eadc684c4bf0737"} Oct 30 02:40:02.731701 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:02.731662 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:02.731911 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:02.731712 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:02.731986 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:02.731905 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:02.731986 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:02.731948 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:02.732095 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:02.732000 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret podName:ef5e843c-9ceb-44f3-bfc9-c56be367804e nodeName:}" failed. No retries permitted until 2026-10-30 02:40:04.731984633 +0000 UTC m=+6.159194589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret") pod "global-pull-secret-syncer-m9zrt" (UID: "ef5e843c-9ceb-44f3-bfc9-c56be367804e") : object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:02.732095 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:02.732043 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs podName:e17aabad-c1ca-490d-904d-ddd40b5fd741 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:04.732025809 +0000 UTC m=+6.159235767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs") pod "network-metrics-daemon-9t4cc" (UID: "e17aabad-c1ca-490d-904d-ddd40b5fd741") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:02.832725 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:02.832688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvtmv\" (UniqueName: \"kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv\") pod \"network-check-target-7rx9c\" (UID: \"ee2a4d7e-545a-4946-a44d-cc895441c477\") " pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:02.832913 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:02.832887 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 02:40:02.832913 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:02.832909 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 02:40:02.833020 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:02.832922 2576 projected.go:196] Error preparing data for projected volume kube-api-access-qvtmv for pod openshift-network-diagnostics/network-check-target-7rx9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:02.833020 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:02.832976 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv podName:ee2a4d7e-545a-4946-a44d-cc895441c477 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:04.832959054 +0000 UTC m=+6.260169014 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvtmv" (UniqueName: "kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv") pod "network-check-target-7rx9c" (UID: "ee2a4d7e-545a-4946-a44d-cc895441c477") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:03.052105 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:03.052013 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:35:01 +0000 UTC" deadline="2028-05-13 06:35:32.010283653 +0000 UTC" Oct 30 02:40:03.052105 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:03.052046 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13467h55m28.958240796s" Oct 30 02:40:03.081344 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:03.081315 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:03.081499 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:03.081447 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9zrt" podUID="ef5e843c-9ceb-44f3-bfc9-c56be367804e" Oct 30 02:40:03.081866 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:03.081844 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:03.081965 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:03.081945 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t4cc" podUID="e17aabad-c1ca-490d-904d-ddd40b5fd741" Oct 30 02:40:03.082033 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:03.082011 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:03.082090 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:03.082079 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rx9c" podUID="ee2a4d7e-545a-4946-a44d-cc895441c477" Oct 30 02:40:03.262842 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:03.262806 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" event={"ID":"cf4d60387e18d9cce88d9d9a653a225b","Type":"ContainerStarted","Data":"15757182b73257031d3a7bd3bc767128143e22bcb6022d796a8779f8e99b61f6"} Oct 30 02:40:03.302480 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:03.302373 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal" podStartSLOduration=3.302357061 podStartE2EDuration="3.302357061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:02.287883883 +0000 UTC m=+3.715093849" watchObservedRunningTime="2026-10-30 02:40:03.302357061 +0000 UTC m=+4.729567068" Oct 30 02:40:04.269971 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:04.269935 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5jm9" event={"ID":"fc8053d7-4573-4d8a-b8aa-54ad6543d84f","Type":"ContainerStarted","Data":"e522105e3ca4076b6eb499877c9a61ac3a62a598a74ba60155be9a092f48eaa9"} Oct 30 02:40:04.337697 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:04.336953 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" podStartSLOduration=4.3369342490000005 podStartE2EDuration="4.336934249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:03.30283111 +0000 UTC m=+4.730041076" watchObservedRunningTime="2026-10-30 02:40:04.336934249 +0000 UTC m=+5.764144215" Oct 30 02:40:04.749723 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:04.748963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:04.749723 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:04.749010 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:04.749723 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:04.749174 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:04.749723 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:04.749237 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs podName:e17aabad-c1ca-490d-904d-ddd40b5fd741 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:08.749220107 +0000 UTC m=+10.176430065 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs") pod "network-metrics-daemon-9t4cc" (UID: "e17aabad-c1ca-490d-904d-ddd40b5fd741") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:04.749723 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:04.749635 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:04.749723 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:04.749683 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret podName:ef5e843c-9ceb-44f3-bfc9-c56be367804e nodeName:}" failed. No retries permitted until 2026-10-30 02:40:08.749668684 +0000 UTC m=+10.176878645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret") pod "global-pull-secret-syncer-m9zrt" (UID: "ef5e843c-9ceb-44f3-bfc9-c56be367804e") : object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:04.850904 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:04.850211 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvtmv\" (UniqueName: \"kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv\") pod \"network-check-target-7rx9c\" (UID: \"ee2a4d7e-545a-4946-a44d-cc895441c477\") " pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:04.850904 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:04.850441 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 02:40:04.850904 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:04.850461 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 02:40:04.850904 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:04.850474 2576 projected.go:196] Error preparing data for projected volume kube-api-access-qvtmv for pod openshift-network-diagnostics/network-check-target-7rx9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:04.850904 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:04.850528 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv podName:ee2a4d7e-545a-4946-a44d-cc895441c477 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:08.850511625 +0000 UTC m=+10.277721582 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvtmv" (UniqueName: "kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv") pod "network-check-target-7rx9c" (UID: "ee2a4d7e-545a-4946-a44d-cc895441c477") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:05.080041 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:05.079967 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:05.080209 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:05.080091 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rx9c" podUID="ee2a4d7e-545a-4946-a44d-cc895441c477" Oct 30 02:40:05.080477 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:05.080456 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:05.080589 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:05.080550 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9zrt" podUID="ef5e843c-9ceb-44f3-bfc9-c56be367804e" Oct 30 02:40:05.080666 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:05.080649 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:05.080798 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:05.080776 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t4cc" podUID="e17aabad-c1ca-490d-904d-ddd40b5fd741" Oct 30 02:40:05.291300 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:05.290294 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc8053d7-4573-4d8a-b8aa-54ad6543d84f" containerID="e522105e3ca4076b6eb499877c9a61ac3a62a598a74ba60155be9a092f48eaa9" exitCode=0 Oct 30 02:40:05.291300 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:05.290354 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5jm9" event={"ID":"fc8053d7-4573-4d8a-b8aa-54ad6543d84f","Type":"ContainerDied","Data":"e522105e3ca4076b6eb499877c9a61ac3a62a598a74ba60155be9a092f48eaa9"} Oct 30 02:40:06.298436 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:06.298086 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n96rg" event={"ID":"8f17a0e8-ce89-4a88-98f2-c7782260534d","Type":"ContainerStarted","Data":"d50d6e1feb28a53b695cbba4ce156d3eeca10a4acc2956793b5935522dd7485b"} Oct 30 02:40:06.335884 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:06.334977 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n96rg" podStartSLOduration=3.512769737 podStartE2EDuration="7.334956849s" totalImagesPullingTime="3.822187112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:06.334617397 +0000 UTC m=+7.761827362" watchObservedRunningTime="2026-10-30 02:40:06.334956849 +0000 UTC m=+7.762166816" Oct 30 02:40:07.079746 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:07.077490 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:07.079746 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:07.077514 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:07.079746 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:07.077605 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rx9c" podUID="ee2a4d7e-545a-4946-a44d-cc895441c477" Oct 30 02:40:07.079746 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:07.077690 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9zrt" podUID="ef5e843c-9ceb-44f3-bfc9-c56be367804e" Oct 30 02:40:07.079746 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:07.077733 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:07.079746 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:07.077850 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t4cc" podUID="e17aabad-c1ca-490d-904d-ddd40b5fd741" Oct 30 02:40:07.308218 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:07.308186 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" event={"ID":"06222d65-ed37-48ff-9b12-19271e1f9662","Type":"ContainerStarted","Data":"4badead749539c13cdfec3ec5c8c0732b4728b56db49423f03e91fc24f22080d"} Oct 30 02:40:07.315926 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:07.315900 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zg4qh" event={"ID":"a3c34ce6-91f3-42e4-ba49-a905f477571e","Type":"ContainerStarted","Data":"fcfcac22a0b75baaa212abdf8b2db249e79fcf4e6d3caee6bca277dd2443cab3"} Oct 30 02:40:07.352295 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:07.352235 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zg4qh" podStartSLOduration=3.457159752 podStartE2EDuration="8.352214962s" totalImagesPullingTime="4.89505521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:07.351637473 +0000 UTC m=+8.778847451" watchObservedRunningTime="2026-10-30 02:40:07.352214962 +0000 UTC m=+8.779424928" Oct 30 02:40:08.785692 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:08.785652 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:08.786093 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:08.785706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:08.786093 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:08.785875 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:08.786093 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:08.785936 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs podName:e17aabad-c1ca-490d-904d-ddd40b5fd741 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:16.785917525 +0000 UTC m=+18.213127481 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs") pod "network-metrics-daemon-9t4cc" (UID: "e17aabad-c1ca-490d-904d-ddd40b5fd741") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:08.786093 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:08.785985 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:08.786093 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:08.786013 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret podName:ef5e843c-9ceb-44f3-bfc9-c56be367804e nodeName:}" failed. No retries permitted until 2026-10-30 02:40:16.786003529 +0000 UTC m=+18.213213473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret") pod "global-pull-secret-syncer-m9zrt" (UID: "ef5e843c-9ceb-44f3-bfc9-c56be367804e") : object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:08.869231 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:08.869205 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 02:40:08.888425 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:08.886738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvtmv\" (UniqueName: \"kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv\") pod \"network-check-target-7rx9c\" (UID: \"ee2a4d7e-545a-4946-a44d-cc895441c477\") " pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:08.888425 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:08.886911 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 02:40:08.888425 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:08.886932 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 02:40:08.888425 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:08.886945 2576 projected.go:196] Error preparing data for projected volume kube-api-access-qvtmv for pod openshift-network-diagnostics/network-check-target-7rx9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:08.888425 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:08.887002 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv podName:ee2a4d7e-545a-4946-a44d-cc895441c477 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:16.886984443 +0000 UTC m=+18.314194403 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvtmv" (UniqueName: "kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv") pod "network-check-target-7rx9c" (UID: "ee2a4d7e-545a-4946-a44d-cc895441c477") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:09.081994 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:09.081899 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:09.081994 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:09.081937 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:09.082996 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:09.082823 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rx9c" podUID="ee2a4d7e-545a-4946-a44d-cc895441c477" Oct 30 02:40:09.082996 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:09.082853 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:09.082996 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:09.082899 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:40:08.869226064Z","UUID":"2995a708-8f10-4790-a3ea-58de33d71139","Handler":null,"Name":"","Endpoint":""} Oct 30 02:40:09.083186 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:09.083008 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9zrt" podUID="ef5e843c-9ceb-44f3-bfc9-c56be367804e" Oct 30 02:40:09.083186 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:09.083156 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t4cc" podUID="e17aabad-c1ca-490d-904d-ddd40b5fd741" Oct 30 02:40:09.086711 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:09.086066 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 02:40:09.086711 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:09.086099 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 02:40:09.322773 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:09.322005 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hbr7c" event={"ID":"5e9def4d-df87-404d-90eb-1b21b023fbd8","Type":"ContainerStarted","Data":"994cc187567c8d840b52e61951a5c0060d98d08f28842aeda481ac48146bd298"} Oct 30 02:40:09.325003 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:09.324533 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gk587" event={"ID":"183230bf-332c-465f-8525-2094db22e38a","Type":"ContainerStarted","Data":"1db528b60f6e5111caf75ab544191ebc035baaa304eb9edac2324a5bae2c886d"} Oct 30 02:40:09.327581 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:09.327486 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" event={"ID":"06222d65-ed37-48ff-9b12-19271e1f9662","Type":"ContainerStarted","Data":"cc77fb84ff7639d4045f56608298fc364cad7a1445211a3e41a78f734e1a016f"} Oct 30 02:40:09.359246 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:09.358981 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-hbr7c" podStartSLOduration=2.692010568 podStartE2EDuration="10.358963861s" totalImagesPullingTime="7.666953293s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:09.358119048 +0000 UTC m=+10.785329014" watchObservedRunningTime="2026-10-30 02:40:09.358963861 +0000 UTC m=+10.786173829" Oct 30 02:40:09.396614 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:09.396555 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-gk587" podStartSLOduration=3.001794361 podStartE2EDuration="10.39653883s" totalImagesPullingTime="7.394744469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:09.396244188 +0000 UTC m=+10.823454155" watchObservedRunningTime="2026-10-30 02:40:09.39653883 +0000 UTC m=+10.823748797" Oct 30 02:40:10.332134 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:10.332092 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc8053d7-4573-4d8a-b8aa-54ad6543d84f" containerID="6ca2d65db303a6e699f88cede89813705750b12d2d600a70e451e4222ad66b8b" exitCode=0 Oct 30 02:40:10.332618 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:10.332190 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5jm9" event={"ID":"fc8053d7-4573-4d8a-b8aa-54ad6543d84f","Type":"ContainerDied","Data":"6ca2d65db303a6e699f88cede89813705750b12d2d600a70e451e4222ad66b8b"} Oct 30 02:40:10.335483 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:10.335452 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" event={"ID":"06222d65-ed37-48ff-9b12-19271e1f9662","Type":"ContainerStarted","Data":"ab261952cacf7b6275d8f99847d8eb2b17803ce4bdef20b53c4de8ea7523fd86"} Oct 30 02:40:11.051173 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:11.051135 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n96rg" Oct 30 02:40:11.051830 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:11.051807 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n96rg" Oct 30 02:40:11.078721 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:11.078521 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:11.078721 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:11.078622 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rx9c" podUID="ee2a4d7e-545a-4946-a44d-cc895441c477" Oct 30 02:40:11.078721 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:11.078685 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:11.078905 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:11.078833 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9zrt" podUID="ef5e843c-9ceb-44f3-bfc9-c56be367804e" Oct 30 02:40:11.078905 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:11.078876 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:11.078970 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:11.078945 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t4cc" podUID="e17aabad-c1ca-490d-904d-ddd40b5fd741" Oct 30 02:40:11.088361 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:11.088309 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" podStartSLOduration=4.026037094 podStartE2EDuration="12.088291374s" totalImagesPullingTime="8.06225428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:39:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:10.418415908 +0000 UTC m=+11.845625872" watchObservedRunningTime="2026-10-30 02:40:11.088291374 +0000 UTC m=+12.515501341" Oct 30 02:40:11.339206 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:11.339171 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-76vs8" event={"ID":"c2b91cd0-34ba-453f-8088-13e814a3a367","Type":"ContainerStarted","Data":"bf572b4a1e46ae0489a62a239e84b0cf73776920b31aae50cc008d3cf37dbb9d"} Oct 30 02:40:11.341458 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:11.341434 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc8053d7-4573-4d8a-b8aa-54ad6543d84f" containerID="1f951b9f87d1877a340950e65b596d305adeb1f27630386162f42a194584d01b" exitCode=0 Oct 30 02:40:11.341570 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:11.341526 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5jm9" event={"ID":"fc8053d7-4573-4d8a-b8aa-54ad6543d84f","Type":"ContainerDied","Data":"1f951b9f87d1877a340950e65b596d305adeb1f27630386162f42a194584d01b"} Oct 30 02:40:11.343284 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:11.343263 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8gjpf" event={"ID":"164920df-32a0-48ab-830a-171a05caab08","Type":"ContainerStarted","Data":"ec717b9ef762fff1628970df024e9362cb2f6bcd1d4013d5184bf642821a9386"} Oct 30 02:40:11.377946 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:11.377891 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-76vs8" podStartSLOduration=2.835065244 podStartE2EDuration="12.377876846s" totalImagesPullingTime="9.542811602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:11.377253136 +0000 UTC m=+12.804463101" watchObservedRunningTime="2026-10-30 02:40:11.377876846 +0000 UTC m=+12.805086810" Oct 30 02:40:11.454978 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:11.454930 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8gjpf" podStartSLOduration=4.95586701 podStartE2EDuration="12.454913234s" totalImagesPullingTime="7.499046224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:11.454717778 +0000 UTC m=+12.881927744" watchObservedRunningTime="2026-10-30 02:40:11.454913234 +0000 UTC m=+12.882123198" Oct 30 02:40:12.348296 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:12.348262 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5jm9" event={"ID":"fc8053d7-4573-4d8a-b8aa-54ad6543d84f","Type":"ContainerStarted","Data":"b3d6f3561058d24be9f22338e9de67e3c78c0cba211f2dd49d126f12db06fcf9"} Oct 30 02:40:13.078289 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:13.078242 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:13.078450 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:13.078239 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:13.078450 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:13.078380 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9zrt" podUID="ef5e843c-9ceb-44f3-bfc9-c56be367804e" Oct 30 02:40:13.078563 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:13.078465 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t4cc" podUID="e17aabad-c1ca-490d-904d-ddd40b5fd741" Oct 30 02:40:13.078563 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:13.078258 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:13.078563 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:13.078546 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rx9c" podUID="ee2a4d7e-545a-4946-a44d-cc895441c477" Oct 30 02:40:13.353415 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:13.353339 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc8053d7-4573-4d8a-b8aa-54ad6543d84f" containerID="b3d6f3561058d24be9f22338e9de67e3c78c0cba211f2dd49d126f12db06fcf9" exitCode=0 Oct 30 02:40:13.353415 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:13.353387 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5jm9" event={"ID":"fc8053d7-4573-4d8a-b8aa-54ad6543d84f","Type":"ContainerDied","Data":"b3d6f3561058d24be9f22338e9de67e3c78c0cba211f2dd49d126f12db06fcf9"} Oct 30 02:40:15.080450 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:15.080270 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:15.080831 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:15.080341 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:15.080831 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:15.080524 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9zrt" podUID="ef5e843c-9ceb-44f3-bfc9-c56be367804e" Oct 30 02:40:15.080831 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:15.080385 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:15.080831 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:15.080594 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rx9c" podUID="ee2a4d7e-545a-4946-a44d-cc895441c477" Oct 30 02:40:15.080831 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:15.080688 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t4cc" podUID="e17aabad-c1ca-490d-904d-ddd40b5fd741" Oct 30 02:40:16.362726 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:16.362555 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc8053d7-4573-4d8a-b8aa-54ad6543d84f" containerID="4d26bdf35edf3836ad73e627f090dcf59a717b381266a541a107057ffb13bbf2" exitCode=0 Oct 30 02:40:16.363257 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:16.362648 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5jm9" event={"ID":"fc8053d7-4573-4d8a-b8aa-54ad6543d84f","Type":"ContainerDied","Data":"4d26bdf35edf3836ad73e627f090dcf59a717b381266a541a107057ffb13bbf2"} Oct 30 02:40:16.365654 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:16.365637 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 02:40:16.365972 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:16.365953 2576 generic.go:356] "Generic (PLEG): container finished" podID="1bac52de-7a27-4fce-828b-9eedcfd60de5" containerID="e2905818ee80361b6de23f09a060e7e9824d6cbf42371c3b92475371af8e74ae" exitCode=1 Oct 30 02:40:16.366066 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:16.365981 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" event={"ID":"1bac52de-7a27-4fce-828b-9eedcfd60de5","Type":"ContainerStarted","Data":"cfc1be7280e7a839ea12b3a7d70ea1b7e246db79f512b034dd87d9650de798c8"} Oct 30 02:40:16.366066 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:16.366028 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" event={"ID":"1bac52de-7a27-4fce-828b-9eedcfd60de5","Type":"ContainerStarted","Data":"7ab17c69fdb4192ced201e7f02a3b330fe613cbaba0d81f69eeced2c0edc15c9"} Oct 30 02:40:16.366066 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:16.366057 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" event={"ID":"1bac52de-7a27-4fce-828b-9eedcfd60de5","Type":"ContainerStarted","Data":"7adc532aa2be9f75210ac3d5a35a95f31470bd7e4f981bea0d551fce8410036a"} Oct 30 02:40:16.366220 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:16.366084 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" event={"ID":"1bac52de-7a27-4fce-828b-9eedcfd60de5","Type":"ContainerStarted","Data":"424d92dde806d4a9e7369ec817f3bf2e3f265f72c3eee9b5c9ee9a9ea390aa30"} Oct 30 02:40:16.366220 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:16.366111 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" event={"ID":"1bac52de-7a27-4fce-828b-9eedcfd60de5","Type":"ContainerDied","Data":"e2905818ee80361b6de23f09a060e7e9824d6cbf42371c3b92475371af8e74ae"} Oct 30 02:40:16.366220 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:16.366136 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" event={"ID":"1bac52de-7a27-4fce-828b-9eedcfd60de5","Type":"ContainerStarted","Data":"e0b71dfae15dd7c0322a03974c36559935139ce835fbf1ba63651acdb58d5d9b"} Oct 30 02:40:16.845009 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:16.844978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:16.845162 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:16.845017 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:16.845162 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:16.845115 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:16.845162 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:16.845135 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:16.845275 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:16.845165 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs podName:e17aabad-c1ca-490d-904d-ddd40b5fd741 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:32.84515359 +0000 UTC m=+34.272363533 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs") pod "network-metrics-daemon-9t4cc" (UID: "e17aabad-c1ca-490d-904d-ddd40b5fd741") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:40:16.845275 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:16.845190 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret podName:ef5e843c-9ceb-44f3-bfc9-c56be367804e nodeName:}" failed. No retries permitted until 2026-10-30 02:40:32.845176476 +0000 UTC m=+34.272386418 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret") pod "global-pull-secret-syncer-m9zrt" (UID: "ef5e843c-9ceb-44f3-bfc9-c56be367804e") : object "kube-system"/"original-pull-secret" not registered Oct 30 02:40:16.946259 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:16.946220 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvtmv\" (UniqueName: \"kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv\") pod \"network-check-target-7rx9c\" (UID: \"ee2a4d7e-545a-4946-a44d-cc895441c477\") " pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:16.946426 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:16.946364 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 02:40:16.946426 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:16.946379 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 02:40:16.946426 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:16.946389 2576 projected.go:196] Error preparing data for projected volume kube-api-access-qvtmv for pod openshift-network-diagnostics/network-check-target-7rx9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:16.946531 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:16.946435 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv podName:ee2a4d7e-545a-4946-a44d-cc895441c477 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:32.946421533 +0000 UTC m=+34.373631490 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvtmv" (UniqueName: "kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv") pod "network-check-target-7rx9c" (UID: "ee2a4d7e-545a-4946-a44d-cc895441c477") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:40:17.080022 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:17.079989 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:17.080207 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:17.080067 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:17.080207 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:17.080163 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:17.080207 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:17.080182 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t4cc" podUID="e17aabad-c1ca-490d-904d-ddd40b5fd741" Oct 30 02:40:17.080365 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:17.080234 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rx9c" podUID="ee2a4d7e-545a-4946-a44d-cc895441c477" Oct 30 02:40:17.080365 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:17.080294 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9zrt" podUID="ef5e843c-9ceb-44f3-bfc9-c56be367804e" Oct 30 02:40:17.371307 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:17.371277 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc8053d7-4573-4d8a-b8aa-54ad6543d84f" containerID="80b8a63b4dd2dc1e16e49bcab79fd2cca0ae842575afd2ca71244185d192be6e" exitCode=0 Oct 30 02:40:17.371789 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:17.371317 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5jm9" event={"ID":"fc8053d7-4573-4d8a-b8aa-54ad6543d84f","Type":"ContainerDied","Data":"80b8a63b4dd2dc1e16e49bcab79fd2cca0ae842575afd2ca71244185d192be6e"} Oct 30 02:40:18.376492 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:18.376464 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 02:40:18.376956 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:18.376882 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" event={"ID":"1bac52de-7a27-4fce-828b-9eedcfd60de5","Type":"ContainerStarted","Data":"ba7dca0ad94b9d61aa6d970ad7138b07eb94fca84376ce55ddae4974f9ff56c0"} Oct 30 02:40:18.379778 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:18.379741 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j5jm9" event={"ID":"fc8053d7-4573-4d8a-b8aa-54ad6543d84f","Type":"ContainerStarted","Data":"2b52411baa262ebbbc2baa4378f52c396fa479d6ec48848a4b2818072d757819"} Oct 30 02:40:18.444690 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:18.444635 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j5jm9" podStartSLOduration=9.010248104 podStartE2EDuration="19.444622497s" totalImagesPullingTime="10.434374393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:39:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:18.433712885 +0000 UTC m=+19.860922851" watchObservedRunningTime="2026-10-30 02:40:18.444622497 +0000 UTC m=+19.871832461" Oct 30 02:40:19.078639 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:19.078604 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:19.078861 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:19.078725 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rx9c" podUID="ee2a4d7e-545a-4946-a44d-cc895441c477" Oct 30 02:40:19.078861 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:19.078776 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:19.078974 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:19.078875 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t4cc" podUID="e17aabad-c1ca-490d-904d-ddd40b5fd741" Oct 30 02:40:19.078974 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:19.078898 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:19.078974 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:19.078941 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9zrt" podUID="ef5e843c-9ceb-44f3-bfc9-c56be367804e" Oct 30 02:40:20.386894 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:20.386865 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 02:40:20.387536 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:20.387189 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" event={"ID":"1bac52de-7a27-4fce-828b-9eedcfd60de5","Type":"ContainerStarted","Data":"6d74163da6b021983c3dd4df3a5ff8cb9e990e495348bf02070deb0845e9b421"} Oct 30 02:40:20.387536 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:20.387463 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:20.387612 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:20.387583 2576 scope.go:122] "RemoveContainer" containerID="e2905818ee80361b6de23f09a060e7e9824d6cbf42371c3b92475371af8e74ae" Oct 30 02:40:20.404591 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:20.404574 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:21.078209 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:21.078179 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:21.078209 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:21.078201 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:21.078379 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:21.078242 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:21.078379 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:21.078359 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t4cc" podUID="e17aabad-c1ca-490d-904d-ddd40b5fd741" Oct 30 02:40:21.078491 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:21.078472 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rx9c" podUID="ee2a4d7e-545a-4946-a44d-cc895441c477" Oct 30 02:40:21.078548 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:21.078534 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9zrt" podUID="ef5e843c-9ceb-44f3-bfc9-c56be367804e" Oct 30 02:40:21.393415 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:21.393349 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 02:40:21.393788 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:21.393681 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" event={"ID":"1bac52de-7a27-4fce-828b-9eedcfd60de5","Type":"ContainerStarted","Data":"d50f543e6e4340acce9ba024cb89dc621dcdef3aa40e47d9316f744ba2d8922b"} Oct 30 02:40:21.393837 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:21.393802 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 02:40:21.394083 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:21.394069 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:21.408931 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:21.408910 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:21.476083 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:21.476039 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" podStartSLOduration=8.276398448 podStartE2EDuration="22.47602726s" totalImagesPullingTime="14.199628812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:21.465219551 +0000 UTC m=+22.892429550" watchObservedRunningTime="2026-10-30 02:40:21.47602726 +0000 UTC m=+22.903237224" Oct 30 02:40:22.073486 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:22.073450 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m9zrt"] Oct 30 02:40:22.073677 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:22.073542 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:22.073677 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:22.073630 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9zrt" podUID="ef5e843c-9ceb-44f3-bfc9-c56be367804e" Oct 30 02:40:22.076428 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:22.076402 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9t4cc"] Oct 30 02:40:22.076538 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:22.076503 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:22.076628 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:22.076609 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t4cc" podUID="e17aabad-c1ca-490d-904d-ddd40b5fd741" Oct 30 02:40:22.081245 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:22.081222 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7rx9c"] Oct 30 02:40:22.081369 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:22.081304 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:22.081434 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:22.081389 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rx9c" podUID="ee2a4d7e-545a-4946-a44d-cc895441c477" Oct 30 02:40:22.395947 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:22.395920 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 02:40:23.398202 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:23.397894 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 02:40:24.078329 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:24.078286 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:24.078492 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:24.078335 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:24.078492 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:24.078300 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:24.078492 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:24.078414 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t4cc" podUID="e17aabad-c1ca-490d-904d-ddd40b5fd741" Oct 30 02:40:24.078492 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:24.078488 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rx9c" podUID="ee2a4d7e-545a-4946-a44d-cc895441c477" Oct 30 02:40:24.078707 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:24.078552 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9zrt" podUID="ef5e843c-9ceb-44f3-bfc9-c56be367804e" Oct 30 02:40:26.077763 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:26.077717 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:26.078225 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:26.077770 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:26.078225 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:26.077721 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:26.078225 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:26.077840 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t4cc" podUID="e17aabad-c1ca-490d-904d-ddd40b5fd741" Oct 30 02:40:26.078225 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:26.077920 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9zrt" podUID="ef5e843c-9ceb-44f3-bfc9-c56be367804e" Oct 30 02:40:26.078225 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:26.078010 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rx9c" podUID="ee2a4d7e-545a-4946-a44d-cc895441c477" Oct 30 02:40:28.078271 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.078241 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:28.078271 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.078272 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:28.078664 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:28.078340 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9zrt" podUID="ef5e843c-9ceb-44f3-bfc9-c56be367804e" Oct 30 02:40:28.078664 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.078390 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:28.078664 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:28.078464 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rx9c" podUID="ee2a4d7e-545a-4946-a44d-cc895441c477" Oct 30 02:40:28.078664 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:28.078519 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t4cc" podUID="e17aabad-c1ca-490d-904d-ddd40b5fd741" Oct 30 02:40:28.349623 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.349415 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeReady" Oct 30 02:40:28.349743 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.349708 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 02:40:28.395394 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.395366 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-69v7b"] Oct 30 02:40:28.400287 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.400258 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-69v7b" Oct 30 02:40:28.401118 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.401074 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9s4nm"] Oct 30 02:40:28.403252 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.403232 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6tnpg"] Oct 30 02:40:28.403350 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.403335 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9s4nm" Oct 30 02:40:28.405213 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.405196 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zc4bx"] Oct 30 02:40:28.405368 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.405318 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6tnpg" Oct 30 02:40:28.407300 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.407285 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6"] Oct 30 02:40:28.407414 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.407401 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:28.409532 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.409513 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-lttrq"] Oct 30 02:40:28.409704 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.409687 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" Oct 30 02:40:28.412025 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.412004 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6bffcf78f7-rplsj"] Oct 30 02:40:28.412147 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.412128 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-lttrq" Oct 30 02:40:28.414600 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.414583 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65c57cd659-szcm2"] Oct 30 02:40:28.414740 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.414713 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:28.416636 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.416618 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 02:40:28.417475 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.417458 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv"] Oct 30 02:40:28.417581 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.417567 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.420057 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.420040 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg"] Oct 30 02:40:28.420360 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.420332 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" Oct 30 02:40:28.422331 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.422316 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-vb5mb"] Oct 30 02:40:28.422461 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.422431 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:28.424845 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.424824 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb"] Oct 30 02:40:28.425794 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.425774 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:28.427102 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.427080 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 02:40:28.427816 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.427801 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk"] Oct 30 02:40:28.427908 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.427880 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.430610 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.430582 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-69v7b"] Oct 30 02:40:28.430700 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.430620 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9s4nm"] Oct 30 02:40:28.430700 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.430640 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6tnpg"] Oct 30 02:40:28.430700 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.430661 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jsswm"] Oct 30 02:40:28.430981 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.430737 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:28.433830 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.433803 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sx7m8"] Oct 30 02:40:28.434253 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.433961 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jsswm" Oct 30 02:40:28.436925 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.436906 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r779s"] Oct 30 02:40:28.437080 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.437053 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:28.441204 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.440314 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-lvg49"] Oct 30 02:40:28.441204 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.440408 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r779s" Oct 30 02:40:28.455985 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.455829 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6"] Oct 30 02:40:28.455985 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.455871 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-lttrq"] Oct 30 02:40:28.455985 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.455893 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65c57cd659-szcm2"] Oct 30 02:40:28.455985 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.455942 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6bffcf78f7-rplsj"] Oct 30 02:40:28.456272 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.456031 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lvg49" Oct 30 02:40:28.456568 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.456542 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-vb5mb"] Oct 30 02:40:28.458855 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.458834 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zc4bx"] Oct 30 02:40:28.459814 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.459795 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv"] Oct 30 02:40:28.460821 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.460803 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sx7m8"] Oct 30 02:40:28.461766 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.461737 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-lvg49"] Oct 30 02:40:28.462992 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.462774 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb"] Oct 30 02:40:28.464013 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.463980 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg"] Oct 30 02:40:28.470708 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.470684 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jsswm"] Oct 30 02:40:28.474086 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.474067 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk"] Oct 30 02:40:28.478863 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.478838 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r779s"] Oct 30 02:40:28.517173 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.517146 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 02:40:28.517278 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.517234 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:40:28.517347 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.517280 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 02:40:28.517347 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.517293 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:40:28.517347 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.517286 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 02:40:28.517347 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.517280 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:40:28.517537 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.517364 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 02:40:28.517537 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.517406 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 02:40:28.517537 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.517420 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 02:40:28.517537 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.517443 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 02:40:28.517537 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.517498 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 02:40:28.517537 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.517523 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 02:40:28.517795 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.517548 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 02:40:28.517795 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.517655 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 02:40:28.517795 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.517674 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 02:40:28.517795 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.517735 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 02:40:28.517985 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.517926 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 02:40:28.517985 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.517953 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 02:40:28.518082 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.518001 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 02:40:28.518082 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.518030 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 02:40:28.518082 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.518046 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 02:40:28.527281 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.527248 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfkd4\" (UniqueName: \"kubernetes.io/projected/478f97e0-5e43-4195-9e8c-bd58cc3d6e00-kube-api-access-bfkd4\") pod \"network-check-source-6f78d59f99-lvg49\" (UID: \"478f97e0-5e43-4195-9e8c-bd58cc3d6e00\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lvg49" Oct 30 02:40:28.527281 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.527273 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b0851f53-fa88-40f9-bfdb-beb956a5fb28-snapshots\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:28.527417 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.527319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqr74\" (UniqueName: \"kubernetes.io/projected/5ee44f66-3b88-4914-803d-9dcfbfb0efba-kube-api-access-dqr74\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:28.527417 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.527348 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66622\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-kube-api-access-66622\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.527417 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.527375 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0851f53-fa88-40f9-bfdb-beb956a5fb28-service-ca-bundle\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:28.527417 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.527405 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-bound-sa-token\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.527575 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.527430 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02aed17c-f8e1-4fd0-bb90-4e972bd1a827-cert\") pod \"ingress-canary-r779s\" (UID: \"02aed17c-f8e1-4fd0-bb90-4e972bd1a827\") " pod="openshift-ingress-canary/ingress-canary-r779s" Oct 30 02:40:28.527575 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.527457 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0851f53-fa88-40f9-bfdb-beb956a5fb28-tmp\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:28.527575 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.527492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-image-registry-private-configuration\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.527575 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.527542 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4llrw\" (UniqueName: \"kubernetes.io/projected/bc230021-ead4-497f-9933-45fbdb9efc50-kube-api-access-4llrw\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:28.527834 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.527597 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/547ef373-3835-4ec2-8f70-94954f34f452-image-registry-private-configuration\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.527834 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.527631 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0851f53-fa88-40f9-bfdb-beb956a5fb28-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:28.527834 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.527676 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpw2q\" (UniqueName: \"kubernetes.io/projected/8ab12b0c-6c4b-4915-9920-9eec9c279f3b-kube-api-access-qpw2q\") pod \"kas-connection-checker-685895c7c9-69v7b\" (UID: \"8ab12b0c-6c4b-4915-9920-9eec9c279f3b\") " pod="kube-system/kas-connection-checker-685895c7c9-69v7b" Oct 30 02:40:28.527834 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.527702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:28.527834 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.527793 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnt4b\" (UniqueName: \"kubernetes.io/projected/96981120-5aa0-43e1-9652-8a625d23b6ff-kube-api-access-lnt4b\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6\" (UID: \"96981120-5aa0-43e1-9652-8a625d23b6ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" Oct 30 02:40:28.527834 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.527830 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.528629 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.527875 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-config\") pod \"console-operator-7cd56b8dbc-vb5mb\" (UID: \"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:28.528629 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.527943 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-trusted-ca\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.528629 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528071 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ba886f04-386a-4cb7-aa00-b85f3921e6cc-config-volume\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:28.528629 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528107 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-certificates\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.528629 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m94g4\" (UniqueName: \"kubernetes.io/projected/114fc00f-05b8-4a65-89e1-5997231c05e7-kube-api-access-m94g4\") pod \"kas-connection-checker-685895c7c9-9s4nm\" (UID: \"114fc00f-05b8-4a65-89e1-5997231c05e7\") " pod="kube-system/kas-connection-checker-685895c7c9-9s4nm" Oct 30 02:40:28.528629 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc230021-ead4-497f-9933-45fbdb9efc50-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:28.528629 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:28.528629 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528224 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba886f04-386a-4cb7-aa00-b85f3921e6cc-tmp-dir\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:28.528629 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528239 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blbrw\" (UniqueName: \"kubernetes.io/projected/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-kube-api-access-blbrw\") pod \"console-operator-7cd56b8dbc-vb5mb\" (UID: \"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:28.528629 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528264 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e-serving-cert\") pod \"service-ca-operator-544d749c8c-n22dv\" (UID: \"6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" Oct 30 02:40:28.528629 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528291 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tdht\" (UniqueName: \"kubernetes.io/projected/4295fa46-8c63-4cde-8183-1895149bc2ce-kube-api-access-2tdht\") pod \"volume-data-source-validator-58c9f5f46d-jsswm\" (UID: \"4295fa46-8c63-4cde-8183-1895149bc2ce\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jsswm" Oct 30 02:40:28.528629 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528305 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 02:40:28.528629 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528315 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 02:40:28.528629 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528321 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:40:28.528629 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528317 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3fb3b957-3183-4c9e-a45f-c6930341197c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-lttrq\" (UID: \"3fb3b957-3183-4c9e-a45f-c6930341197c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lttrq" Oct 30 02:40:28.528629 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528343 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 02:40:28.528629 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528371 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 02:40:28.528629 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528411 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-stats-auth\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:28.528629 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528425 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 02:40:28.529912 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528431 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xbl9\" (UniqueName: \"kubernetes.io/projected/da5b2698-93b5-4b4a-8404-f22ec7840967-kube-api-access-9xbl9\") pod \"kas-connection-checker-685895c7c9-6tnpg\" (UID: \"da5b2698-93b5-4b4a-8404-f22ec7840967\") " pod="kube-system/kas-connection-checker-685895c7c9-6tnpg" Oct 30 02:40:28.529912 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528456 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-bound-sa-token\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.529912 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528468 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 02:40:28.529912 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528471 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:40:28.529912 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnh58\" (UniqueName: \"kubernetes.io/projected/02aed17c-f8e1-4fd0-bb90-4e972bd1a827-kube-api-access-nnh58\") pod \"ingress-canary-r779s\" (UID: \"02aed17c-f8e1-4fd0-bb90-4e972bd1a827\") " pod="openshift-ingress-canary/ingress-canary-r779s" Oct 30 02:40:28.529912 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528574 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srs2l\" (UniqueName: \"kubernetes.io/projected/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e-kube-api-access-srs2l\") pod \"service-ca-operator-544d749c8c-n22dv\" (UID: \"6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" Oct 30 02:40:28.529912 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528601 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/547ef373-3835-4ec2-8f70-94954f34f452-ca-trust-extracted\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.529912 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w87m\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-kube-api-access-9w87m\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.529912 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528644 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/547ef373-3835-4ec2-8f70-94954f34f452-installation-pull-secrets\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.529912 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528696 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-metrics-certs\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:28.529912 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96981120-5aa0-43e1-9652-8a625d23b6ff-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6\" (UID: \"96981120-5aa0-43e1-9652-8a625d23b6ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" Oct 30 02:40:28.529912 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528738 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/547ef373-3835-4ec2-8f70-94954f34f452-registry-certificates\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.529912 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-default-certificate\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:28.529912 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-trusted-ca\") pod \"console-operator-7cd56b8dbc-vb5mb\" (UID: \"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:28.529912 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-installation-pull-secrets\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.529912 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0851f53-fa88-40f9-bfdb-beb956a5fb28-serving-cert\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:28.530669 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528952 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5pxf\" (UniqueName: \"kubernetes.io/projected/b0851f53-fa88-40f9-bfdb-beb956a5fb28-kube-api-access-r5pxf\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:28.530669 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.528998 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e-config\") pod \"service-ca-operator-544d749c8c-n22dv\" (UID: \"6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" Oct 30 02:40:28.530669 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.529045 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/547ef373-3835-4ec2-8f70-94954f34f452-trusted-ca\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.530669 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.529084 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-ca-trust-extracted\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.530669 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.529110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96981120-5aa0-43e1-9652-8a625d23b6ff-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6\" (UID: \"96981120-5aa0-43e1-9652-8a625d23b6ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" Oct 30 02:40:28.530669 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.529200 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lttrq\" (UID: \"3fb3b957-3183-4c9e-a45f-c6930341197c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lttrq" Oct 30 02:40:28.530669 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.529256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:28.530669 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.529277 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 02:40:28.530669 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.529284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.530669 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.529306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-serving-cert\") pod \"console-operator-7cd56b8dbc-vb5mb\" (UID: \"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:28.530669 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.529361 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:28.530669 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.529397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:28.530669 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.529432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvvcv\" (UniqueName: \"kubernetes.io/projected/548e9394-45be-4ffa-937a-89fea1da3093-kube-api-access-lvvcv\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:28.530669 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.529482 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 02:40:28.530669 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.529516 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjg4n\" (UniqueName: \"kubernetes.io/projected/ba886f04-386a-4cb7-aa00-b85f3921e6cc-kube-api-access-xjg4n\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:28.530669 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.529531 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 02:40:28.530669 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.529531 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 02:40:28.530669 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.529540 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 02:40:28.531535 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.529525 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 02:40:28.531535 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.529768 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 02:40:28.531535 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.529777 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 02:40:28.538463 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.538442 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 02:40:28.538463 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.538457 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 02:40:28.539502 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.539447 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 02:40:28.539502 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.539477 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 02:40:28.539502 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.539490 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 02:40:28.539781 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.539607 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6t67f\"" Oct 30 02:40:28.539781 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.539618 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8td9f\"" Oct 30 02:40:28.539781 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.539616 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-2549v\"" Oct 30 02:40:28.539781 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.539656 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-npcvz\"" Oct 30 02:40:28.539781 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.539665 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-s6lr5\"" Oct 30 02:40:28.539781 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.539678 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rpstg\"" Oct 30 02:40:28.539781 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.539679 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-g86t2\"" Oct 30 02:40:28.539781 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.539682 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 02:40:28.539781 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.539725 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ksjrs\"" Oct 30 02:40:28.539781 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.539735 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-pr25d\"" Oct 30 02:40:28.540310 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.539837 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-csvhs\"" Oct 30 02:40:28.540310 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.539975 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-5bpds\"" Oct 30 02:40:28.540310 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.540119 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pjcb2\"" Oct 30 02:40:28.541470 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.541370 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-zglns\"" Oct 30 02:40:28.543241 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.543221 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 02:40:28.545345 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.545326 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 02:40:28.545842 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.545826 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 02:40:28.545923 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.545909 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 02:40:28.573551 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.573526 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n96rg" Oct 30 02:40:28.573679 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.573666 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 02:40:28.574128 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.574106 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n96rg" Oct 30 02:40:28.629942 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.629858 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ba886f04-386a-4cb7-aa00-b85f3921e6cc-config-volume\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:28.629942 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.629907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-certificates\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.629942 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.629936 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m94g4\" (UniqueName: \"kubernetes.io/projected/114fc00f-05b8-4a65-89e1-5997231c05e7-kube-api-access-m94g4\") pod \"kas-connection-checker-685895c7c9-9s4nm\" (UID: \"114fc00f-05b8-4a65-89e1-5997231c05e7\") " pod="kube-system/kas-connection-checker-685895c7c9-9s4nm" Oct 30 02:40:28.630198 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.629960 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc230021-ead4-497f-9933-45fbdb9efc50-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:28.630198 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.629984 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:28.630310 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.630276 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba886f04-386a-4cb7-aa00-b85f3921e6cc-tmp-dir\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:28.630382 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.630361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blbrw\" (UniqueName: \"kubernetes.io/projected/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-kube-api-access-blbrw\") pod \"console-operator-7cd56b8dbc-vb5mb\" (UID: \"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:28.630433 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.630398 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e-serving-cert\") pod \"service-ca-operator-544d749c8c-n22dv\" (UID: \"6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" Oct 30 02:40:28.630487 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.630439 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tdht\" (UniqueName: \"kubernetes.io/projected/4295fa46-8c63-4cde-8183-1895149bc2ce-kube-api-access-2tdht\") pod \"volume-data-source-validator-58c9f5f46d-jsswm\" (UID: \"4295fa46-8c63-4cde-8183-1895149bc2ce\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jsswm" Oct 30 02:40:28.630536 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.630485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3fb3b957-3183-4c9e-a45f-c6930341197c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-lttrq\" (UID: \"3fb3b957-3183-4c9e-a45f-c6930341197c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lttrq" Oct 30 02:40:28.630536 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.630520 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-stats-auth\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:28.630635 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.630548 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xbl9\" (UniqueName: \"kubernetes.io/projected/da5b2698-93b5-4b4a-8404-f22ec7840967-kube-api-access-9xbl9\") pod \"kas-connection-checker-685895c7c9-6tnpg\" (UID: \"da5b2698-93b5-4b4a-8404-f22ec7840967\") " pod="kube-system/kas-connection-checker-685895c7c9-6tnpg" Oct 30 02:40:28.630635 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.630579 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-bound-sa-token\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.630635 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.630597 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba886f04-386a-4cb7-aa00-b85f3921e6cc-tmp-dir\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:28.630635 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.630603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnh58\" (UniqueName: \"kubernetes.io/projected/02aed17c-f8e1-4fd0-bb90-4e972bd1a827-kube-api-access-nnh58\") pod \"ingress-canary-r779s\" (UID: \"02aed17c-f8e1-4fd0-bb90-4e972bd1a827\") " pod="openshift-ingress-canary/ingress-canary-r779s" Oct 30 02:40:28.630635 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.630627 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ba886f04-386a-4cb7-aa00-b85f3921e6cc-config-volume\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:28.630911 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.630661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srs2l\" (UniqueName: \"kubernetes.io/projected/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e-kube-api-access-srs2l\") pod \"service-ca-operator-544d749c8c-n22dv\" (UID: \"6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" Oct 30 02:40:28.630911 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.630693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/547ef373-3835-4ec2-8f70-94954f34f452-ca-trust-extracted\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.630911 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.630693 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-certificates\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.630911 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.630863 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:28.631138 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631101 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc230021-ead4-497f-9933-45fbdb9efc50-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:28.631221 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631135 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w87m\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-kube-api-access-9w87m\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.631221 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631173 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/547ef373-3835-4ec2-8f70-94954f34f452-installation-pull-secrets\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.631221 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631196 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-metrics-certs\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:28.631364 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631220 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96981120-5aa0-43e1-9652-8a625d23b6ff-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6\" (UID: \"96981120-5aa0-43e1-9652-8a625d23b6ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" Oct 30 02:40:28.631364 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631258 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/547ef373-3835-4ec2-8f70-94954f34f452-registry-certificates\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.631364 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631281 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-default-certificate\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:28.631364 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631286 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/547ef373-3835-4ec2-8f70-94954f34f452-ca-trust-extracted\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.631364 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-trusted-ca\") pod \"console-operator-7cd56b8dbc-vb5mb\" (UID: \"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:28.631620 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631435 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3fb3b957-3183-4c9e-a45f-c6930341197c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-lttrq\" (UID: \"3fb3b957-3183-4c9e-a45f-c6930341197c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lttrq" Oct 30 02:40:28.631620 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-installation-pull-secrets\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.631620 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0851f53-fa88-40f9-bfdb-beb956a5fb28-serving-cert\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:28.631620 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5pxf\" (UniqueName: \"kubernetes.io/projected/b0851f53-fa88-40f9-bfdb-beb956a5fb28-kube-api-access-r5pxf\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:28.631830 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e-config\") pod \"service-ca-operator-544d749c8c-n22dv\" (UID: \"6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" Oct 30 02:40:28.631830 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631647 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/547ef373-3835-4ec2-8f70-94954f34f452-trusted-ca\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.631830 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631670 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-ca-trust-extracted\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.631830 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:28.631689 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 02:40:28.631830 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96981120-5aa0-43e1-9652-8a625d23b6ff-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6\" (UID: \"96981120-5aa0-43e1-9652-8a625d23b6ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" Oct 30 02:40:28.631830 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631729 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lttrq\" (UID: \"3fb3b957-3183-4c9e-a45f-c6930341197c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lttrq" Oct 30 02:40:28.631830 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:28.631771 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-metrics-certs podName:548e9394-45be-4ffa-937a-89fea1da3093 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:29.131733217 +0000 UTC m=+30.558943178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-metrics-certs") pod "router-default-6bffcf78f7-rplsj" (UID: "548e9394-45be-4ffa-937a-89fea1da3093") : secret "router-metrics-certs-default" not found Oct 30 02:40:28.631830 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631812 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:28.632265 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:28.631831 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 02:40:28.632265 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.632265 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631862 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96981120-5aa0-43e1-9652-8a625d23b6ff-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6\" (UID: \"96981120-5aa0-43e1-9652-8a625d23b6ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" Oct 30 02:40:28.632265 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-serving-cert\") pod \"console-operator-7cd56b8dbc-vb5mb\" (UID: \"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:28.632265 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.631871 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/547ef373-3835-4ec2-8f70-94954f34f452-registry-certificates\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.632265 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:28.631882 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert podName:3fb3b957-3183-4c9e-a45f-c6930341197c nodeName:}" failed. No retries permitted until 2026-10-30 02:40:29.131866773 +0000 UTC m=+30.559076730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lttrq" (UID: "3fb3b957-3183-4c9e-a45f-c6930341197c") : secret "networking-console-plugin-cert" not found Oct 30 02:40:28.632265 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:28.632000 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 02:40:28.632265 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:28.632000 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 02:40:28.632265 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:28.632112 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65c57cd659-szcm2: secret "image-registry-tls" not found Oct 30 02:40:28.632265 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.632202 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-trusted-ca\") pod \"console-operator-7cd56b8dbc-vb5mb\" (UID: \"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:28.632712 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:28.632349 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls podName:ba886f04-386a-4cb7-aa00-b85f3921e6cc nodeName:}" failed. No retries permitted until 2026-10-30 02:40:29.132333271 +0000 UTC m=+30.559543219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls") pod "dns-default-sx7m8" (UID: "ba886f04-386a-4cb7-aa00-b85f3921e6cc") : secret "dns-default-metrics-tls" not found Oct 30 02:40:28.632712 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:28.632382 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls podName:547ef373-3835-4ec2-8f70-94954f34f452 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:29.132373507 +0000 UTC m=+30.559583453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls") pod "image-registry-65c57cd659-szcm2" (UID: "547ef373-3835-4ec2-8f70-94954f34f452") : secret "image-registry-tls" not found Oct 30 02:40:28.632712 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.632423 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:28.632712 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.632448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:28.632712 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.632469 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvvcv\" (UniqueName: \"kubernetes.io/projected/548e9394-45be-4ffa-937a-89fea1da3093-kube-api-access-lvvcv\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:28.632712 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.632498 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e-config\") pod \"service-ca-operator-544d749c8c-n22dv\" (UID: \"6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" Oct 30 02:40:28.632712 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.632517 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjg4n\" (UniqueName: \"kubernetes.io/projected/ba886f04-386a-4cb7-aa00-b85f3921e6cc-kube-api-access-xjg4n\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:28.632712 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.632556 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfkd4\" (UniqueName: \"kubernetes.io/projected/478f97e0-5e43-4195-9e8c-bd58cc3d6e00-kube-api-access-bfkd4\") pod \"network-check-source-6f78d59f99-lvg49\" (UID: \"478f97e0-5e43-4195-9e8c-bd58cc3d6e00\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lvg49" Oct 30 02:40:28.632712 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.632581 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b0851f53-fa88-40f9-bfdb-beb956a5fb28-snapshots\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:28.632712 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.632604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqr74\" (UniqueName: \"kubernetes.io/projected/5ee44f66-3b88-4914-803d-9dcfbfb0efba-kube-api-access-dqr74\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:28.632712 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.632632 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66622\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-kube-api-access-66622\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.632712 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.632663 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0851f53-fa88-40f9-bfdb-beb956a5fb28-service-ca-bundle\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:28.632712 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.632694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-bound-sa-token\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.632712 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.632718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02aed17c-f8e1-4fd0-bb90-4e972bd1a827-cert\") pod \"ingress-canary-r779s\" (UID: \"02aed17c-f8e1-4fd0-bb90-4e972bd1a827\") " pod="openshift-ingress-canary/ingress-canary-r779s" Oct 30 02:40:28.633378 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.632744 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0851f53-fa88-40f9-bfdb-beb956a5fb28-tmp\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:28.633378 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.632797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-image-registry-private-configuration\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.633378 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.633153 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-ca-trust-extracted\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.633378 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.633350 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b0851f53-fa88-40f9-bfdb-beb956a5fb28-snapshots\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:28.633571 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:28.633431 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 02:40:28.633729 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:28.633710 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 02:40:28.634305 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.634275 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0851f53-fa88-40f9-bfdb-beb956a5fb28-tmp\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:28.634398 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.634353 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/547ef373-3835-4ec2-8f70-94954f34f452-trusted-ca\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.634398 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.634356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4llrw\" (UniqueName: \"kubernetes.io/projected/bc230021-ead4-497f-9933-45fbdb9efc50-kube-api-access-4llrw\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:28.634507 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.634413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/547ef373-3835-4ec2-8f70-94954f34f452-image-registry-private-configuration\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.634507 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.634440 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0851f53-fa88-40f9-bfdb-beb956a5fb28-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:28.634507 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.634484 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpw2q\" (UniqueName: \"kubernetes.io/projected/8ab12b0c-6c4b-4915-9920-9eec9c279f3b-kube-api-access-qpw2q\") pod \"kas-connection-checker-685895c7c9-69v7b\" (UID: \"8ab12b0c-6c4b-4915-9920-9eec9c279f3b\") " pod="kube-system/kas-connection-checker-685895c7c9-69v7b" Oct 30 02:40:28.634658 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.634510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:28.634658 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:28.634612 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle podName:548e9394-45be-4ffa-937a-89fea1da3093 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:29.134596546 +0000 UTC m=+30.561806514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle") pod "router-default-6bffcf78f7-rplsj" (UID: "548e9394-45be-4ffa-937a-89fea1da3093") : configmap references non-existent config key: service-ca.crt Oct 30 02:40:28.634836 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.634283 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0851f53-fa88-40f9-bfdb-beb956a5fb28-service-ca-bundle\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:28.634978 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:28.634964 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls podName:5ee44f66-3b88-4914-803d-9dcfbfb0efba nodeName:}" failed. No retries permitted until 2026-10-30 02:40:29.134946248 +0000 UTC m=+30.562156205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n74xk" (UID: "5ee44f66-3b88-4914-803d-9dcfbfb0efba") : secret "samples-operator-tls" not found Oct 30 02:40:28.635079 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:28.635069 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls podName:bc230021-ead4-497f-9933-45fbdb9efc50 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:29.135056464 +0000 UTC m=+30.562266407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bshlg" (UID: "bc230021-ead4-497f-9933-45fbdb9efc50") : secret "cluster-monitoring-operator-tls" not found Oct 30 02:40:28.635347 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.635318 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0851f53-fa88-40f9-bfdb-beb956a5fb28-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:28.635492 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.635471 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnt4b\" (UniqueName: \"kubernetes.io/projected/96981120-5aa0-43e1-9652-8a625d23b6ff-kube-api-access-lnt4b\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6\" (UID: \"96981120-5aa0-43e1-9652-8a625d23b6ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" Oct 30 02:40:28.635559 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.635526 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.635611 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.635564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-config\") pod \"console-operator-7cd56b8dbc-vb5mb\" (UID: \"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:28.635780 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:28.635712 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 02:40:28.635780 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:28.635737 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb: secret "image-registry-tls" not found Oct 30 02:40:28.635900 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:28.635819 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls podName:11560ce6-6d4a-4c30-83eb-b1b7a3e50cae nodeName:}" failed. No retries permitted until 2026-10-30 02:40:29.135803526 +0000 UTC m=+30.563013470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls") pod "image-registry-6bc5fd6d9d-w9mrb" (UID: "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae") : secret "image-registry-tls" not found Oct 30 02:40:28.635900 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.635842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-trusted-ca\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.636078 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.636061 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e-serving-cert\") pod \"service-ca-operator-544d749c8c-n22dv\" (UID: \"6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" Oct 30 02:40:28.636217 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.636192 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-config\") pod \"console-operator-7cd56b8dbc-vb5mb\" (UID: \"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:28.636313 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.636292 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-stats-auth\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:28.636713 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.636673 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-trusted-ca\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.636848 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.636812 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/547ef373-3835-4ec2-8f70-94954f34f452-installation-pull-secrets\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.637786 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.637622 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96981120-5aa0-43e1-9652-8a625d23b6ff-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6\" (UID: \"96981120-5aa0-43e1-9652-8a625d23b6ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" Oct 30 02:40:28.637786 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.637725 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0851f53-fa88-40f9-bfdb-beb956a5fb28-serving-cert\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:28.637945 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.637903 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-default-certificate\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:28.638043 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.637972 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-serving-cert\") pod \"console-operator-7cd56b8dbc-vb5mb\" (UID: \"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:28.638355 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.638316 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/547ef373-3835-4ec2-8f70-94954f34f452-image-registry-private-configuration\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.638453 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.638439 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-installation-pull-secrets\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.638523 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.638506 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02aed17c-f8e1-4fd0-bb90-4e972bd1a827-cert\") pod \"ingress-canary-r779s\" (UID: \"02aed17c-f8e1-4fd0-bb90-4e972bd1a827\") " pod="openshift-ingress-canary/ingress-canary-r779s" Oct 30 02:40:28.638699 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.638683 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-image-registry-private-configuration\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.652863 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.652828 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m94g4\" (UniqueName: \"kubernetes.io/projected/114fc00f-05b8-4a65-89e1-5997231c05e7-kube-api-access-m94g4\") pod \"kas-connection-checker-685895c7c9-9s4nm\" (UID: \"114fc00f-05b8-4a65-89e1-5997231c05e7\") " pod="kube-system/kas-connection-checker-685895c7c9-9s4nm" Oct 30 02:40:28.653913 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.653891 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srs2l\" (UniqueName: \"kubernetes.io/projected/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e-kube-api-access-srs2l\") pod \"service-ca-operator-544d749c8c-n22dv\" (UID: \"6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" Oct 30 02:40:28.654296 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.654246 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnh58\" (UniqueName: \"kubernetes.io/projected/02aed17c-f8e1-4fd0-bb90-4e972bd1a827-kube-api-access-nnh58\") pod \"ingress-canary-r779s\" (UID: \"02aed17c-f8e1-4fd0-bb90-4e972bd1a827\") " pod="openshift-ingress-canary/ingress-canary-r779s" Oct 30 02:40:28.655512 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.655484 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-bound-sa-token\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.656744 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.656719 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blbrw\" (UniqueName: \"kubernetes.io/projected/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-kube-api-access-blbrw\") pod \"console-operator-7cd56b8dbc-vb5mb\" (UID: \"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:28.657270 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.657232 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xbl9\" (UniqueName: \"kubernetes.io/projected/da5b2698-93b5-4b4a-8404-f22ec7840967-kube-api-access-9xbl9\") pod \"kas-connection-checker-685895c7c9-6tnpg\" (UID: \"da5b2698-93b5-4b4a-8404-f22ec7840967\") " pod="kube-system/kas-connection-checker-685895c7c9-6tnpg" Oct 30 02:40:28.657465 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.657442 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w87m\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-kube-api-access-9w87m\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.658269 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.658244 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5pxf\" (UniqueName: \"kubernetes.io/projected/b0851f53-fa88-40f9-bfdb-beb956a5fb28-kube-api-access-r5pxf\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:28.660308 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.660276 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqr74\" (UniqueName: \"kubernetes.io/projected/5ee44f66-3b88-4914-803d-9dcfbfb0efba-kube-api-access-dqr74\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:28.660915 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.660892 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfkd4\" (UniqueName: \"kubernetes.io/projected/478f97e0-5e43-4195-9e8c-bd58cc3d6e00-kube-api-access-bfkd4\") pod \"network-check-source-6f78d59f99-lvg49\" (UID: \"478f97e0-5e43-4195-9e8c-bd58cc3d6e00\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lvg49" Oct 30 02:40:28.662135 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.662057 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4llrw\" (UniqueName: \"kubernetes.io/projected/bc230021-ead4-497f-9933-45fbdb9efc50-kube-api-access-4llrw\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:28.662424 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.662400 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpw2q\" (UniqueName: \"kubernetes.io/projected/8ab12b0c-6c4b-4915-9920-9eec9c279f3b-kube-api-access-qpw2q\") pod \"kas-connection-checker-685895c7c9-69v7b\" (UID: \"8ab12b0c-6c4b-4915-9920-9eec9c279f3b\") " pod="kube-system/kas-connection-checker-685895c7c9-69v7b" Oct 30 02:40:28.662483 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.662408 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-bound-sa-token\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:28.662680 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.662656 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnt4b\" (UniqueName: \"kubernetes.io/projected/96981120-5aa0-43e1-9652-8a625d23b6ff-kube-api-access-lnt4b\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6\" (UID: \"96981120-5aa0-43e1-9652-8a625d23b6ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" Oct 30 02:40:28.663175 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.663156 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjg4n\" (UniqueName: \"kubernetes.io/projected/ba886f04-386a-4cb7-aa00-b85f3921e6cc-kube-api-access-xjg4n\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:28.663311 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.663263 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvvcv\" (UniqueName: \"kubernetes.io/projected/548e9394-45be-4ffa-937a-89fea1da3093-kube-api-access-lvvcv\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:28.663683 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.663661 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tdht\" (UniqueName: \"kubernetes.io/projected/4295fa46-8c63-4cde-8183-1895149bc2ce-kube-api-access-2tdht\") pod \"volume-data-source-validator-58c9f5f46d-jsswm\" (UID: \"4295fa46-8c63-4cde-8183-1895149bc2ce\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jsswm" Oct 30 02:40:28.664211 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.664189 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66622\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-kube-api-access-66622\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:28.688694 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.688670 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:28.688904 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.688885 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 02:40:28.699599 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.699576 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:28.714317 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.714300 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-69v7b" Oct 30 02:40:28.721504 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.721489 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9s4nm" Oct 30 02:40:28.729967 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.729943 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6tnpg" Oct 30 02:40:28.739715 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.739685 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:28.760288 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.760257 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" Oct 30 02:40:28.805259 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.792974 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" Oct 30 02:40:28.811230 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.811005 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:28.821134 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.820944 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jsswm" Oct 30 02:40:28.835104 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.834886 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r779s" Oct 30 02:40:28.840423 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:28.840397 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lvg49" Oct 30 02:40:29.034079 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.034048 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-69v7b"] Oct 30 02:40:29.037904 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:40:29.037879 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab12b0c_6c4b_4915_9920_9eec9c279f3b.slice/crio-c919183d1ac4d100f4eb8d32c483ac44fbd401cc70d61d3fcce1d62908f310c8 WatchSource:0}: Error finding container c919183d1ac4d100f4eb8d32c483ac44fbd401cc70d61d3fcce1d62908f310c8: Status 404 returned error can't find the container with id c919183d1ac4d100f4eb8d32c483ac44fbd401cc70d61d3fcce1d62908f310c8 Oct 30 02:40:29.056829 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.056803 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6tnpg"] Oct 30 02:40:29.059186 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:40:29.059157 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5b2698_93b5_4b4a_8404_f22ec7840967.slice/crio-a8ff87c0e760227f238eae688bf33a51b0ed7cfdf3761efa6c9aa9a1df282334 WatchSource:0}: Error finding container a8ff87c0e760227f238eae688bf33a51b0ed7cfdf3761efa6c9aa9a1df282334: Status 404 returned error can't find the container with id a8ff87c0e760227f238eae688bf33a51b0ed7cfdf3761efa6c9aa9a1df282334 Oct 30 02:40:29.061339 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.061317 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9s4nm"] Oct 30 02:40:29.070560 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.070532 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zc4bx"] Oct 30 02:40:29.077065 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:40:29.077021 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0851f53_fa88_40f9_bfdb_beb956a5fb28.slice/crio-3c15c2ff2643e620839168f7bf2bc9dbb6c2a5be585801771d4401efb12a4f10 WatchSource:0}: Error finding container 3c15c2ff2643e620839168f7bf2bc9dbb6c2a5be585801771d4401efb12a4f10: Status 404 returned error can't find the container with id 3c15c2ff2643e620839168f7bf2bc9dbb6c2a5be585801771d4401efb12a4f10 Oct 30 02:40:29.104544 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.104513 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6"] Oct 30 02:40:29.110663 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:40:29.110632 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96981120_5aa0_43e1_9652_8a625d23b6ff.slice/crio-a3b38091b72a3b2330534bf067a07bbfd2c1de26cf85545d1242461bdf95b9cd WatchSource:0}: Error finding container a3b38091b72a3b2330534bf067a07bbfd2c1de26cf85545d1242461bdf95b9cd: Status 404 returned error can't find the container with id a3b38091b72a3b2330534bf067a07bbfd2c1de26cf85545d1242461bdf95b9cd Oct 30 02:40:29.142448 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.142394 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-metrics-certs\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:29.142554 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.142454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lttrq\" (UID: \"3fb3b957-3183-4c9e-a45f-c6930341197c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lttrq" Oct 30 02:40:29.142554 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.142485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:29.142554 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.142510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:29.142554 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.142548 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:29.142743 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.142575 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:29.142743 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.142644 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:29.142743 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.142680 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:29.144228 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:29.142833 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 02:40:29.144228 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:29.142849 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb: secret "image-registry-tls" not found Oct 30 02:40:29.144228 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:29.142899 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls podName:11560ce6-6d4a-4c30-83eb-b1b7a3e50cae nodeName:}" failed. No retries permitted until 2026-10-30 02:40:30.142882069 +0000 UTC m=+31.570092020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls") pod "image-registry-6bc5fd6d9d-w9mrb" (UID: "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae") : secret "image-registry-tls" not found Oct 30 02:40:29.144228 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:29.143208 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 02:40:29.144228 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:29.143216 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 02:40:29.144228 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:29.143258 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls podName:ba886f04-386a-4cb7-aa00-b85f3921e6cc nodeName:}" failed. No retries permitted until 2026-10-30 02:40:30.143241638 +0000 UTC m=+31.570451585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls") pod "dns-default-sx7m8" (UID: "ba886f04-386a-4cb7-aa00-b85f3921e6cc") : secret "dns-default-metrics-tls" not found Oct 30 02:40:29.144228 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:29.143277 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls podName:5ee44f66-3b88-4914-803d-9dcfbfb0efba nodeName:}" failed. No retries permitted until 2026-10-30 02:40:30.143267373 +0000 UTC m=+31.570477322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n74xk" (UID: "5ee44f66-3b88-4914-803d-9dcfbfb0efba") : secret "samples-operator-tls" not found Oct 30 02:40:29.144228 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:29.143318 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 02:40:29.144228 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:29.143350 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle podName:548e9394-45be-4ffa-937a-89fea1da3093 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:30.143340055 +0000 UTC m=+31.570550004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle") pod "router-default-6bffcf78f7-rplsj" (UID: "548e9394-45be-4ffa-937a-89fea1da3093") : configmap references non-existent config key: service-ca.crt Oct 30 02:40:29.144228 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:29.143367 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls podName:bc230021-ead4-497f-9933-45fbdb9efc50 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:30.143359868 +0000 UTC m=+31.570569817 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bshlg" (UID: "bc230021-ead4-497f-9933-45fbdb9efc50") : secret "cluster-monitoring-operator-tls" not found Oct 30 02:40:29.144228 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:29.143413 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 02:40:29.144228 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:29.143466 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert podName:3fb3b957-3183-4c9e-a45f-c6930341197c nodeName:}" failed. No retries permitted until 2026-10-30 02:40:30.143456786 +0000 UTC m=+31.570666728 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lttrq" (UID: "3fb3b957-3183-4c9e-a45f-c6930341197c") : secret "networking-console-plugin-cert" not found Oct 30 02:40:29.144228 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:29.143517 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 02:40:29.144228 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:29.143563 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-metrics-certs podName:548e9394-45be-4ffa-937a-89fea1da3093 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:30.143551668 +0000 UTC m=+31.570761611 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-metrics-certs") pod "router-default-6bffcf78f7-rplsj" (UID: "548e9394-45be-4ffa-937a-89fea1da3093") : secret "router-metrics-certs-default" not found Oct 30 02:40:29.144228 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:29.143645 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 02:40:29.144228 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:29.143655 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65c57cd659-szcm2: secret "image-registry-tls" not found Oct 30 02:40:29.145140 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:29.143704 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls podName:547ef373-3835-4ec2-8f70-94954f34f452 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:30.143692693 +0000 UTC m=+31.570902636 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls") pod "image-registry-65c57cd659-szcm2" (UID: "547ef373-3835-4ec2-8f70-94954f34f452") : secret "image-registry-tls" not found Oct 30 02:40:29.159095 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.159069 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv"] Oct 30 02:40:29.163408 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:40:29.163378 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aac72f9_3975_48ea_8dc0_f2cceaa0dc1e.slice/crio-007dd77a9212556c96196b0946373ac7249b0d5cb2dbb9235cf38af2a1b82a54 WatchSource:0}: Error finding container 007dd77a9212556c96196b0946373ac7249b0d5cb2dbb9235cf38af2a1b82a54: Status 404 returned error can't find the container with id 007dd77a9212556c96196b0946373ac7249b0d5cb2dbb9235cf38af2a1b82a54 Oct 30 02:40:29.183324 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.182878 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-vb5mb"] Oct 30 02:40:29.187332 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:40:29.187301 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ee3aae_ce3d_4fc0_9419_60fb4c762ad3.slice/crio-f15520740d3037a5315cca4c5e0b1e0b856646888ff9886a230931f8e2d8e4b5 WatchSource:0}: Error finding container f15520740d3037a5315cca4c5e0b1e0b856646888ff9886a230931f8e2d8e4b5: Status 404 returned error can't find the container with id f15520740d3037a5315cca4c5e0b1e0b856646888ff9886a230931f8e2d8e4b5 Oct 30 02:40:29.196920 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.196830 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r779s"] Oct 30 02:40:29.202786 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.199871 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jsswm"] Oct 30 02:40:29.207608 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.207500 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-lvg49"] Oct 30 02:40:29.413270 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.413186 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jsswm" event={"ID":"4295fa46-8c63-4cde-8183-1895149bc2ce","Type":"ContainerStarted","Data":"c21771905628738c026109124a1ea5781f6aaf7a4288b74b1ba5306d9843b98b"} Oct 30 02:40:29.414420 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.414388 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" event={"ID":"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3","Type":"ContainerStarted","Data":"f15520740d3037a5315cca4c5e0b1e0b856646888ff9886a230931f8e2d8e4b5"} Oct 30 02:40:29.415479 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.415447 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" event={"ID":"96981120-5aa0-43e1-9652-8a625d23b6ff","Type":"ContainerStarted","Data":"a3b38091b72a3b2330534bf067a07bbfd2c1de26cf85545d1242461bdf95b9cd"} Oct 30 02:40:29.416950 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.416929 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-69v7b" event={"ID":"8ab12b0c-6c4b-4915-9920-9eec9c279f3b","Type":"ContainerStarted","Data":"d27a564c3b79f54aeb4e49b22c02238164cde82488b2476eb5a09613978fecfa"} Oct 30 02:40:29.417080 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.416969 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-69v7b" event={"ID":"8ab12b0c-6c4b-4915-9920-9eec9c279f3b","Type":"ContainerStarted","Data":"c919183d1ac4d100f4eb8d32c483ac44fbd401cc70d61d3fcce1d62908f310c8"} Oct 30 02:40:29.418175 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.418154 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lvg49" event={"ID":"478f97e0-5e43-4195-9e8c-bd58cc3d6e00","Type":"ContainerStarted","Data":"3feddf09ba135ee458f8e236272dc8edaa6e3e6f20abea4468bb585603ad82cc"} Oct 30 02:40:29.419185 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.419168 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r779s" event={"ID":"02aed17c-f8e1-4fd0-bb90-4e972bd1a827","Type":"ContainerStarted","Data":"57269abdfa4bdc78872a2c1aac3d4ef10f24d18f7440889d7b28d93c0a3a0512"} Oct 30 02:40:29.420133 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.420114 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" event={"ID":"6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e","Type":"ContainerStarted","Data":"007dd77a9212556c96196b0946373ac7249b0d5cb2dbb9235cf38af2a1b82a54"} Oct 30 02:40:29.421175 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.421158 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zc4bx" event={"ID":"b0851f53-fa88-40f9-bfdb-beb956a5fb28","Type":"ContainerStarted","Data":"3c15c2ff2643e620839168f7bf2bc9dbb6c2a5be585801771d4401efb12a4f10"} Oct 30 02:40:29.422526 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.422507 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6tnpg" event={"ID":"da5b2698-93b5-4b4a-8404-f22ec7840967","Type":"ContainerStarted","Data":"40ada932ee501dd195930dd11dfd4d3105bae9745add8e7c6bcb9fce100ad6fc"} Oct 30 02:40:29.422616 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.422545 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6tnpg" event={"ID":"da5b2698-93b5-4b4a-8404-f22ec7840967","Type":"ContainerStarted","Data":"a8ff87c0e760227f238eae688bf33a51b0ed7cfdf3761efa6c9aa9a1df282334"} Oct 30 02:40:29.423916 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.423894 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9s4nm" event={"ID":"114fc00f-05b8-4a65-89e1-5997231c05e7","Type":"ContainerStarted","Data":"bd0f9c402ee3159d617e33449c0c428463aec6191d75503de8e3c1a287e9b445"} Oct 30 02:40:29.423997 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.423934 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9s4nm" event={"ID":"114fc00f-05b8-4a65-89e1-5997231c05e7","Type":"ContainerStarted","Data":"24af01bc9dac9496ab26624f46552fdf41165c8e1d3c804b7188006bec2c6751"} Oct 30 02:40:29.474937 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.474900 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-69v7b" podStartSLOduration=881.474887558 podStartE2EDuration="14m41.474887558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:29.474046337 +0000 UTC m=+30.901256301" watchObservedRunningTime="2026-10-30 02:40:29.474887558 +0000 UTC m=+30.902097522" Oct 30 02:40:29.536254 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.536199 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6tnpg" podStartSLOduration=881.536180123 podStartE2EDuration="14m41.536180123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:29.534819356 +0000 UTC m=+30.962029322" watchObservedRunningTime="2026-10-30 02:40:29.536180123 +0000 UTC m=+30.963390085" Oct 30 02:40:29.578942 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:29.578887 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-9s4nm" podStartSLOduration=881.578868039 podStartE2EDuration="14m41.578868039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:29.577874954 +0000 UTC m=+31.005084921" watchObservedRunningTime="2026-10-30 02:40:29.578868039 +0000 UTC m=+31.006078005" Oct 30 02:40:30.077511 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:30.077475 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:30.078014 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:30.077994 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:30.078440 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:30.078422 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:30.092254 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:30.091916 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 02:40:30.092254 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:30.092014 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2qbtl\"" Oct 30 02:40:30.092254 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:30.092071 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7tzfh\"" Oct 30 02:40:30.092254 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:30.092121 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lz8p4\"" Oct 30 02:40:30.092254 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:30.092176 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 02:40:30.154988 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:30.154283 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:30.154988 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:30.154347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-metrics-certs\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:30.154988 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:30.154440 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 02:40:30.154988 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:30.154455 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb: secret "image-registry-tls" not found Oct 30 02:40:30.154988 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:30.154462 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lttrq\" (UID: \"3fb3b957-3183-4c9e-a45f-c6930341197c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lttrq" Oct 30 02:40:30.154988 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:30.154494 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:30.154988 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:30.154514 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls podName:11560ce6-6d4a-4c30-83eb-b1b7a3e50cae nodeName:}" failed. No retries permitted until 2026-10-30 02:40:32.154496099 +0000 UTC m=+33.581706065 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls") pod "image-registry-6bc5fd6d9d-w9mrb" (UID: "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae") : secret "image-registry-tls" not found Oct 30 02:40:30.154988 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:30.154543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:30.154988 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:30.154558 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 02:40:30.154988 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:30.154593 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls podName:ba886f04-386a-4cb7-aa00-b85f3921e6cc nodeName:}" failed. No retries permitted until 2026-10-30 02:40:32.154581711 +0000 UTC m=+33.581791673 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls") pod "dns-default-sx7m8" (UID: "ba886f04-386a-4cb7-aa00-b85f3921e6cc") : secret "dns-default-metrics-tls" not found Oct 30 02:40:30.154988 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:30.154588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:30.154988 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:30.154623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:30.154988 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:30.154640 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 02:40:30.154988 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:30.154671 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls podName:bc230021-ead4-497f-9933-45fbdb9efc50 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:32.154661712 +0000 UTC m=+33.581871672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bshlg" (UID: "bc230021-ead4-497f-9933-45fbdb9efc50") : secret "cluster-monitoring-operator-tls" not found Oct 30 02:40:30.154988 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:30.154705 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 02:40:30.154988 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:30.154720 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:30.155848 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:30.154738 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls podName:5ee44f66-3b88-4914-803d-9dcfbfb0efba nodeName:}" failed. No retries permitted until 2026-10-30 02:40:32.154728965 +0000 UTC m=+33.581938908 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n74xk" (UID: "5ee44f66-3b88-4914-803d-9dcfbfb0efba") : secret "samples-operator-tls" not found Oct 30 02:40:30.155848 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:30.154806 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle podName:548e9394-45be-4ffa-937a-89fea1da3093 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:32.154795773 +0000 UTC m=+33.582005731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle") pod "router-default-6bffcf78f7-rplsj" (UID: "548e9394-45be-4ffa-937a-89fea1da3093") : configmap references non-existent config key: service-ca.crt Oct 30 02:40:30.155848 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:30.154440 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 02:40:30.155848 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:30.154837 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-metrics-certs podName:548e9394-45be-4ffa-937a-89fea1da3093 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:32.154829099 +0000 UTC m=+33.582039056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-metrics-certs") pod "router-default-6bffcf78f7-rplsj" (UID: "548e9394-45be-4ffa-937a-89fea1da3093") : secret "router-metrics-certs-default" not found Oct 30 02:40:30.155848 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:30.154840 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 02:40:30.155848 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:30.154868 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert podName:3fb3b957-3183-4c9e-a45f-c6930341197c nodeName:}" failed. No retries permitted until 2026-10-30 02:40:32.154860345 +0000 UTC m=+33.582070288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lttrq" (UID: "3fb3b957-3183-4c9e-a45f-c6930341197c") : secret "networking-console-plugin-cert" not found Oct 30 02:40:30.155848 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:30.154919 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 02:40:30.155848 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:30.154929 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65c57cd659-szcm2: secret "image-registry-tls" not found Oct 30 02:40:30.155848 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:30.154958 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls podName:547ef373-3835-4ec2-8f70-94954f34f452 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:32.154949298 +0000 UTC m=+33.582159256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls") pod "image-registry-65c57cd659-szcm2" (UID: "547ef373-3835-4ec2-8f70-94954f34f452") : secret "image-registry-tls" not found Oct 30 02:40:31.437052 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:31.436957 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jsswm" event={"ID":"4295fa46-8c63-4cde-8183-1895149bc2ce","Type":"ContainerStarted","Data":"cb4503136eeedd96ca168ef66acbcd813a84339dc43d86eff51f8bac0553986d"} Oct 30 02:40:31.476554 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:31.475716 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jsswm" podStartSLOduration=841.530363421 podStartE2EDuration="14m3.475702458s" totalImagesPullingTime="1.945339037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:31.474745059 +0000 UTC m=+32.901955026" watchObservedRunningTime="2026-10-30 02:40:31.475702458 +0000 UTC m=+32.902912432" Oct 30 02:40:32.185111 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:32.185071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:32.185267 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:32.185161 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-metrics-certs\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:32.185267 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:32.185213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lttrq\" (UID: \"3fb3b957-3183-4c9e-a45f-c6930341197c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lttrq" Oct 30 02:40:32.185267 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:32.185240 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:32.185267 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:32.185263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:32.185475 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:32.185306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:32.185475 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:32.185331 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:32.185475 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:32.185463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:32.185688 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:32.185671 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle podName:548e9394-45be-4ffa-937a-89fea1da3093 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:36.185651496 +0000 UTC m=+37.612861448 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle") pod "router-default-6bffcf78f7-rplsj" (UID: "548e9394-45be-4ffa-937a-89fea1da3093") : configmap references non-existent config key: service-ca.crt Oct 30 02:40:32.186152 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:32.186132 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 02:40:32.186152 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:32.186154 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb: secret "image-registry-tls" not found Oct 30 02:40:32.186316 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:32.186201 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls podName:11560ce6-6d4a-4c30-83eb-b1b7a3e50cae nodeName:}" failed. No retries permitted until 2026-10-30 02:40:36.186186458 +0000 UTC m=+37.613396400 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls") pod "image-registry-6bc5fd6d9d-w9mrb" (UID: "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae") : secret "image-registry-tls" not found Oct 30 02:40:32.186316 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:32.186263 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 02:40:32.186316 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:32.186298 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-metrics-certs podName:548e9394-45be-4ffa-937a-89fea1da3093 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:36.186287584 +0000 UTC m=+37.613497534 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-metrics-certs") pod "router-default-6bffcf78f7-rplsj" (UID: "548e9394-45be-4ffa-937a-89fea1da3093") : secret "router-metrics-certs-default" not found Oct 30 02:40:32.186472 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:32.186357 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 02:40:32.186472 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:32.186391 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert podName:3fb3b957-3183-4c9e-a45f-c6930341197c nodeName:}" failed. No retries permitted until 2026-10-30 02:40:36.186379212 +0000 UTC m=+37.613589154 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lttrq" (UID: "3fb3b957-3183-4c9e-a45f-c6930341197c") : secret "networking-console-plugin-cert" not found Oct 30 02:40:32.186472 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:32.186448 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 02:40:32.186623 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:32.186477 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls podName:ba886f04-386a-4cb7-aa00-b85f3921e6cc nodeName:}" failed. No retries permitted until 2026-10-30 02:40:36.18646701 +0000 UTC m=+37.613676958 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls") pod "dns-default-sx7m8" (UID: "ba886f04-386a-4cb7-aa00-b85f3921e6cc") : secret "dns-default-metrics-tls" not found Oct 30 02:40:32.186623 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:32.186535 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 02:40:32.186623 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:32.186546 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65c57cd659-szcm2: secret "image-registry-tls" not found Oct 30 02:40:32.186623 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:32.186575 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls podName:547ef373-3835-4ec2-8f70-94954f34f452 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:36.186565417 +0000 UTC m=+37.613775367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls") pod "image-registry-65c57cd659-szcm2" (UID: "547ef373-3835-4ec2-8f70-94954f34f452") : secret "image-registry-tls" not found Oct 30 02:40:32.186849 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:32.186631 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 02:40:32.186849 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:32.186662 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls podName:bc230021-ead4-497f-9933-45fbdb9efc50 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:36.186651976 +0000 UTC m=+37.613861924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bshlg" (UID: "bc230021-ead4-497f-9933-45fbdb9efc50") : secret "cluster-monitoring-operator-tls" not found Oct 30 02:40:32.186849 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:32.186719 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 02:40:32.186849 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:32.186767 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls podName:5ee44f66-3b88-4914-803d-9dcfbfb0efba nodeName:}" failed. No retries permitted until 2026-10-30 02:40:36.186739732 +0000 UTC m=+37.613949681 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n74xk" (UID: "5ee44f66-3b88-4914-803d-9dcfbfb0efba") : secret "samples-operator-tls" not found Oct 30 02:40:32.893727 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:32.893042 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:32.893727 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:32.893120 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:32.893727 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:32.893301 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 02:40:32.893727 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:32.893364 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs podName:e17aabad-c1ca-490d-904d-ddd40b5fd741 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:04.893344832 +0000 UTC m=+66.320554792 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs") pod "network-metrics-daemon-9t4cc" (UID: "e17aabad-c1ca-490d-904d-ddd40b5fd741") : secret "metrics-daemon-secret" not found Oct 30 02:40:32.896476 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:32.896346 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:32.994965 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:32.994247 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvtmv\" (UniqueName: \"kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv\") pod \"network-check-target-7rx9c\" (UID: \"ee2a4d7e-545a-4946-a44d-cc895441c477\") " pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:32.997631 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:32.997530 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvtmv\" (UniqueName: \"kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv\") pod \"network-check-target-7rx9c\" (UID: \"ee2a4d7e-545a-4946-a44d-cc895441c477\") " pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:33.096191 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.096157 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:33.107103 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.107074 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:33.370124 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.368638 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-j2rxp"] Oct 30 02:40:33.375323 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.374965 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:33.391406 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.391309 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 02:40:33.393808 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.393788 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j2rxp"] Oct 30 02:40:33.402939 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.402374 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p7m4t\"" Oct 30 02:40:33.418350 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.418323 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7rx9c"] Oct 30 02:40:33.425562 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:40:33.425523 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee2a4d7e_545a_4946_a44d_cc895441c477.slice/crio-e64e87996858f40a90aca4b0cab20f099f2f6e3001492251b91fe45789518fb6 WatchSource:0}: Error finding container e64e87996858f40a90aca4b0cab20f099f2f6e3001492251b91fe45789518fb6: Status 404 returned error can't find the container with id e64e87996858f40a90aca4b0cab20f099f2f6e3001492251b91fe45789518fb6 Oct 30 02:40:33.458285 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.457691 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" event={"ID":"6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e","Type":"ContainerStarted","Data":"dedf8e40b1fa6b3767e3d6713659d1dfbdeb553ac31a306c0ec795f26b4aa1c5"} Oct 30 02:40:33.458285 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.457806 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m9zrt"] Oct 30 02:40:33.461255 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.461226 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zc4bx" event={"ID":"b0851f53-fa88-40f9-bfdb-beb956a5fb28","Type":"ContainerStarted","Data":"79ededb91e9281203b2625b1f49d55140e69f1a5729184264ae6b96e0e0c209d"} Oct 30 02:40:33.464083 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.464062 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/0.log" Oct 30 02:40:33.464247 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.464227 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" event={"ID":"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3","Type":"ContainerDied","Data":"6180145b1e27ad7242c9ea5d501a88de47aef14f9c56c44ed3a810f692e6a52b"} Oct 30 02:40:33.464305 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.464131 2576 generic.go:356] "Generic (PLEG): container finished" podID="21ee3aae-ce3d-4fc0-9419-60fb4c762ad3" containerID="6180145b1e27ad7242c9ea5d501a88de47aef14f9c56c44ed3a810f692e6a52b" exitCode=255 Oct 30 02:40:33.464446 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.464429 2576 scope.go:122] "RemoveContainer" containerID="6180145b1e27ad7242c9ea5d501a88de47aef14f9c56c44ed3a810f692e6a52b" Oct 30 02:40:33.467301 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.466436 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" event={"ID":"96981120-5aa0-43e1-9652-8a625d23b6ff","Type":"ContainerStarted","Data":"3fd8a99f7e30739cb385ddfe3e6df1cc9e362163131c665d71f0f73d870ce4fd"} Oct 30 02:40:33.469115 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.469034 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7rx9c" event={"ID":"ee2a4d7e-545a-4946-a44d-cc895441c477","Type":"ContainerStarted","Data":"e64e87996858f40a90aca4b0cab20f099f2f6e3001492251b91fe45789518fb6"} Oct 30 02:40:33.490790 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.490746 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 02:40:33.495287 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.495220 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" podStartSLOduration=837.453162 podStartE2EDuration="14m1.495206335s" totalImagesPullingTime="4.042044335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:33.494349041 +0000 UTC m=+34.921558998" watchObservedRunningTime="2026-10-30 02:40:33.495206335 +0000 UTC m=+34.922416300" Oct 30 02:40:33.500908 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.500807 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:33.500908 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.500846 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bl6b\" (UniqueName: \"kubernetes.io/projected/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-api-access-9bl6b\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:33.501059 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.500975 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c24d2a3f-bbb2-491e-9f25-608364102b2a-data-volume\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:33.501111 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.501046 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:33.501171 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.501142 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c24d2a3f-bbb2-491e-9f25-608364102b2a-crio-socket\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:33.535835 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.535582 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-zc4bx" podStartSLOduration=840.899525565 podStartE2EDuration="14m4.535566355s" totalImagesPullingTime="3.63604079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:26:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:33.532941692 +0000 UTC m=+34.960151682" watchObservedRunningTime="2026-10-30 02:40:33.535566355 +0000 UTC m=+34.962776319" Oct 30 02:40:33.571503 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.571431 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" podStartSLOduration=840.897739507 podStartE2EDuration="14m4.57140845s" totalImagesPullingTime="3.673668943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:26:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:33.570241586 +0000 UTC m=+34.997451552" watchObservedRunningTime="2026-10-30 02:40:33.57140845 +0000 UTC m=+34.998618416" Oct 30 02:40:33.602320 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.602212 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c24d2a3f-bbb2-491e-9f25-608364102b2a-data-volume\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:33.602320 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.602272 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:33.602955 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.602936 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c24d2a3f-bbb2-491e-9f25-608364102b2a-data-volume\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:33.603569 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:33.603551 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 02:40:33.603628 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:33.603617 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-rbac-proxy-cm podName:c24d2a3f-bbb2-491e-9f25-608364102b2a nodeName:}" failed. No retries permitted until 2026-10-30 02:40:34.103601624 +0000 UTC m=+35.530811581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j2rxp" (UID: "c24d2a3f-bbb2-491e-9f25-608364102b2a") : configmap "kube-rbac-proxy" not found Oct 30 02:40:33.605318 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.604322 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c24d2a3f-bbb2-491e-9f25-608364102b2a-crio-socket\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:33.605318 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.604434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:33.605318 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.604482 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bl6b\" (UniqueName: \"kubernetes.io/projected/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-api-access-9bl6b\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:33.607046 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.606962 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c24d2a3f-bbb2-491e-9f25-608364102b2a-crio-socket\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:33.608399 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:33.607213 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 02:40:33.608399 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:33.607272 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls podName:c24d2a3f-bbb2-491e-9f25-608364102b2a nodeName:}" failed. No retries permitted until 2026-10-30 02:40:34.107253839 +0000 UTC m=+35.534463805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j2rxp" (UID: "c24d2a3f-bbb2-491e-9f25-608364102b2a") : secret "insights-runtime-extractor-tls" not found Oct 30 02:40:33.636883 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.636853 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bl6b\" (UniqueName: \"kubernetes.io/projected/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-api-access-9bl6b\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:33.699898 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.698585 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nsf8s"] Oct 30 02:40:33.719740 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.719673 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nsf8s" Oct 30 02:40:33.720608 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.720574 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nsf8s"] Oct 30 02:40:33.733865 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.733844 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 02:40:33.736224 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.736202 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fd952\"" Oct 30 02:40:33.834765 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.834707 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 02:40:33.908864 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:33.908831 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64r2m\" (UniqueName: \"kubernetes.io/projected/c9becf8f-e251-42bc-a888-18c72145379d-kube-api-access-64r2m\") pod \"migrator-66df5d496b-nsf8s\" (UID: \"c9becf8f-e251-42bc-a888-18c72145379d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nsf8s" Oct 30 02:40:34.010681 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:34.009882 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64r2m\" (UniqueName: \"kubernetes.io/projected/c9becf8f-e251-42bc-a888-18c72145379d-kube-api-access-64r2m\") pod \"migrator-66df5d496b-nsf8s\" (UID: \"c9becf8f-e251-42bc-a888-18c72145379d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nsf8s" Oct 30 02:40:34.036597 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:34.036532 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64r2m\" (UniqueName: \"kubernetes.io/projected/c9becf8f-e251-42bc-a888-18c72145379d-kube-api-access-64r2m\") pod \"migrator-66df5d496b-nsf8s\" (UID: \"c9becf8f-e251-42bc-a888-18c72145379d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nsf8s" Oct 30 02:40:34.055341 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:34.055306 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nsf8s" Oct 30 02:40:34.111857 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:34.111607 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:34.112741 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:34.112140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:34.112741 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:34.111741 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 02:40:34.112741 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:34.112347 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls podName:c24d2a3f-bbb2-491e-9f25-608364102b2a nodeName:}" failed. No retries permitted until 2026-10-30 02:40:35.112328758 +0000 UTC m=+36.539538708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j2rxp" (UID: "c24d2a3f-bbb2-491e-9f25-608364102b2a") : secret "insights-runtime-extractor-tls" not found Oct 30 02:40:34.112741 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:34.112282 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 02:40:34.112741 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:34.112710 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-rbac-proxy-cm podName:c24d2a3f-bbb2-491e-9f25-608364102b2a nodeName:}" failed. No retries permitted until 2026-10-30 02:40:35.112694866 +0000 UTC m=+36.539904814 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j2rxp" (UID: "c24d2a3f-bbb2-491e-9f25-608364102b2a") : configmap "kube-rbac-proxy" not found Oct 30 02:40:34.411549 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:34.411029 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nsf8s"] Oct 30 02:40:34.416208 ip-10-0-139-19 kubenswrapper[2576]: W1030 02:40:34.416172 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9becf8f_e251_42bc_a888_18c72145379d.slice/crio-5c3ec2955f73a0075e1619a94612ad8bd2987da07de561761850075d581d136b WatchSource:0}: Error finding container 5c3ec2955f73a0075e1619a94612ad8bd2987da07de561761850075d581d136b: Status 404 returned error can't find the container with id 5c3ec2955f73a0075e1619a94612ad8bd2987da07de561761850075d581d136b Oct 30 02:40:34.476254 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:34.476185 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/1.log" Oct 30 02:40:34.476929 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:34.476908 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/0.log" Oct 30 02:40:34.477060 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:34.476948 2576 generic.go:356] "Generic (PLEG): container finished" podID="21ee3aae-ce3d-4fc0-9419-60fb4c762ad3" containerID="86843fc740326707a729126a99a09bc7b7364f00b2a1bdfe4d7804f799d79f71" exitCode=255 Oct 30 02:40:34.477060 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:34.477030 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" event={"ID":"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3","Type":"ContainerDied","Data":"86843fc740326707a729126a99a09bc7b7364f00b2a1bdfe4d7804f799d79f71"} Oct 30 02:40:34.477177 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:34.477092 2576 scope.go:122] "RemoveContainer" containerID="6180145b1e27ad7242c9ea5d501a88de47aef14f9c56c44ed3a810f692e6a52b" Oct 30 02:40:34.477339 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:34.477321 2576 scope.go:122] "RemoveContainer" containerID="86843fc740326707a729126a99a09bc7b7364f00b2a1bdfe4d7804f799d79f71" Oct 30 02:40:34.477558 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:34.477534 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-vb5mb_openshift-console-operator(21ee3aae-ce3d-4fc0-9419-60fb4c762ad3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" podUID="21ee3aae-ce3d-4fc0-9419-60fb4c762ad3" Oct 30 02:40:34.479003 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:34.478980 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nsf8s" event={"ID":"c9becf8f-e251-42bc-a888-18c72145379d","Type":"ContainerStarted","Data":"5c3ec2955f73a0075e1619a94612ad8bd2987da07de561761850075d581d136b"} Oct 30 02:40:34.481811 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:34.481787 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m9zrt" event={"ID":"ef5e843c-9ceb-44f3-bfc9-c56be367804e","Type":"ContainerStarted","Data":"4499f24fbb9f197e95488eecebde2b9f24903864d1607f071fce4b074bd4bf83"} Oct 30 02:40:35.140941 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:35.140902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:35.141345 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:35.141067 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:35.141345 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:35.141237 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 02:40:35.141345 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:35.141305 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-rbac-proxy-cm podName:c24d2a3f-bbb2-491e-9f25-608364102b2a nodeName:}" failed. No retries permitted until 2026-10-30 02:40:37.141284221 +0000 UTC m=+38.568494180 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j2rxp" (UID: "c24d2a3f-bbb2-491e-9f25-608364102b2a") : configmap "kube-rbac-proxy" not found Oct 30 02:40:35.141526 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:35.141410 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 02:40:35.141526 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:35.141465 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls podName:c24d2a3f-bbb2-491e-9f25-608364102b2a nodeName:}" failed. No retries permitted until 2026-10-30 02:40:37.141451577 +0000 UTC m=+38.568661525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j2rxp" (UID: "c24d2a3f-bbb2-491e-9f25-608364102b2a") : secret "insights-runtime-extractor-tls" not found Oct 30 02:40:35.487801 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:35.487744 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lvg49" event={"ID":"478f97e0-5e43-4195-9e8c-bd58cc3d6e00","Type":"ContainerStarted","Data":"0a6757e0e9864f74c5d871e4b060ae06e7c68a05a984da3e150929fdf610a690"} Oct 30 02:40:35.490345 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:35.490316 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r779s" event={"ID":"02aed17c-f8e1-4fd0-bb90-4e972bd1a827","Type":"ContainerStarted","Data":"fac1b9356789e85a9b561db07c457f86819eaf9294d8dd018f5f9facebeec93a"} Oct 30 02:40:35.492231 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:35.492175 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/1.log" Oct 30 02:40:35.492506 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:35.492486 2576 scope.go:122] "RemoveContainer" containerID="86843fc740326707a729126a99a09bc7b7364f00b2a1bdfe4d7804f799d79f71" Oct 30 02:40:35.492676 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:35.492656 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-vb5mb_openshift-console-operator(21ee3aae-ce3d-4fc0-9419-60fb4c762ad3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" podUID="21ee3aae-ce3d-4fc0-9419-60fb4c762ad3" Oct 30 02:40:35.551093 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:35.550926 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lvg49" podStartSLOduration=838.754266777 podStartE2EDuration="14m4.550907992s" totalImagesPullingTime="5.796641215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:35.548562937 +0000 UTC m=+36.975772903" watchObservedRunningTime="2026-10-30 02:40:35.550907992 +0000 UTC m=+36.978117959" Oct 30 02:40:35.587868 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:35.587809 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r779s" podStartSLOduration=2.404706361 podStartE2EDuration="7.587793977s" totalImagesPullingTime="5.183087616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:35.587043038 +0000 UTC m=+37.014253003" watchObservedRunningTime="2026-10-30 02:40:35.587793977 +0000 UTC m=+37.015003942" Oct 30 02:40:36.253329 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:36.253303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-metrics-certs\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:36.253645 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:36.253386 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lttrq\" (UID: \"3fb3b957-3183-4c9e-a45f-c6930341197c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lttrq" Oct 30 02:40:36.253645 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:36.253423 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:36.253645 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:36.253447 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:36.253645 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:36.253477 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:36.253645 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:36.253501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:36.253645 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:36.253572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:36.253645 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:36.253604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:36.253938 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:36.253736 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 02:40:36.253938 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:36.253777 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb: secret "image-registry-tls" not found Oct 30 02:40:36.253938 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:36.253831 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls podName:11560ce6-6d4a-4c30-83eb-b1b7a3e50cae nodeName:}" failed. No retries permitted until 2026-10-30 02:40:44.253813973 +0000 UTC m=+45.681023930 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls") pod "image-registry-6bc5fd6d9d-w9mrb" (UID: "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae") : secret "image-registry-tls" not found Oct 30 02:40:36.254292 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:36.254189 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 02:40:36.254292 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:36.254205 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle podName:548e9394-45be-4ffa-937a-89fea1da3093 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:44.254186981 +0000 UTC m=+45.681396938 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle") pod "router-default-6bffcf78f7-rplsj" (UID: "548e9394-45be-4ffa-937a-89fea1da3093") : configmap references non-existent config key: service-ca.crt Oct 30 02:40:36.254292 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:36.254205 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 02:40:36.254292 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:36.254253 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls podName:bc230021-ead4-497f-9933-45fbdb9efc50 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:44.254236214 +0000 UTC m=+45.681446177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bshlg" (UID: "bc230021-ead4-497f-9933-45fbdb9efc50") : secret "cluster-monitoring-operator-tls" not found Oct 30 02:40:36.254292 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:36.254264 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 02:40:36.254292 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:36.254275 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls podName:5ee44f66-3b88-4914-803d-9dcfbfb0efba nodeName:}" failed. No retries permitted until 2026-10-30 02:40:44.254264431 +0000 UTC m=+45.681474376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n74xk" (UID: "5ee44f66-3b88-4914-803d-9dcfbfb0efba") : secret "samples-operator-tls" not found Oct 30 02:40:36.254590 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:36.254324 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls podName:ba886f04-386a-4cb7-aa00-b85f3921e6cc nodeName:}" failed. No retries permitted until 2026-10-30 02:40:44.254308193 +0000 UTC m=+45.681518145 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls") pod "dns-default-sx7m8" (UID: "ba886f04-386a-4cb7-aa00-b85f3921e6cc") : secret "dns-default-metrics-tls" not found Oct 30 02:40:36.254590 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:36.254371 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 02:40:36.254590 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:36.254425 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-metrics-certs podName:548e9394-45be-4ffa-937a-89fea1da3093 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:44.254407888 +0000 UTC m=+45.681617846 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-metrics-certs") pod "router-default-6bffcf78f7-rplsj" (UID: "548e9394-45be-4ffa-937a-89fea1da3093") : secret "router-metrics-certs-default" not found Oct 30 02:40:36.254590 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:36.254482 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 02:40:36.254590 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:36.254528 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert podName:3fb3b957-3183-4c9e-a45f-c6930341197c nodeName:}" failed. No retries permitted until 2026-10-30 02:40:44.254515317 +0000 UTC m=+45.681725275 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lttrq" (UID: "3fb3b957-3183-4c9e-a45f-c6930341197c") : secret "networking-console-plugin-cert" not found Oct 30 02:40:36.254590 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:36.254566 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 02:40:36.254590 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:36.254580 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65c57cd659-szcm2: secret "image-registry-tls" not found Oct 30 02:40:36.254870 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:36.254626 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls podName:547ef373-3835-4ec2-8f70-94954f34f452 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:44.254614918 +0000 UTC m=+45.681824883 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls") pod "image-registry-65c57cd659-szcm2" (UID: "547ef373-3835-4ec2-8f70-94954f34f452") : secret "image-registry-tls" not found Oct 30 02:40:36.497833 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:36.497780 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7rx9c" event={"ID":"ee2a4d7e-545a-4946-a44d-cc895441c477","Type":"ContainerStarted","Data":"3294b6b4ed07c20d7794411c61faec1c3f90768abcaba31fb41893b5ebbfd021"} Oct 30 02:40:36.498001 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:36.497947 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:36.499590 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:36.499567 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nsf8s" event={"ID":"c9becf8f-e251-42bc-a888-18c72145379d","Type":"ContainerStarted","Data":"3ac1e8f7a44a9be246b55afb2f8c5a563fba22770795ccad8fb3866ac2d101c1"} Oct 30 02:40:36.539525 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:36.539397 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7rx9c" podStartSLOduration=34.95490094 podStartE2EDuration="37.53937814s" totalImagesPullingTime="2.5844772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:36.538816448 +0000 UTC m=+37.966026411" watchObservedRunningTime="2026-10-30 02:40:36.53937814 +0000 UTC m=+37.966588105" Oct 30 02:40:36.580277 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:36.580210 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nsf8s" podStartSLOduration=1.761627716 podStartE2EDuration="3.580192947s" totalImagesPullingTime="1.818565231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:36.578250216 +0000 UTC m=+38.005460192" watchObservedRunningTime="2026-10-30 02:40:36.580192947 +0000 UTC m=+38.007402914" Oct 30 02:40:37.163217 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:37.163171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:37.163440 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:37.163307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:37.163440 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:37.163359 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 02:40:37.163440 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:37.163436 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls podName:c24d2a3f-bbb2-491e-9f25-608364102b2a nodeName:}" failed. No retries permitted until 2026-10-30 02:40:41.163415214 +0000 UTC m=+42.590625174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j2rxp" (UID: "c24d2a3f-bbb2-491e-9f25-608364102b2a") : secret "insights-runtime-extractor-tls" not found Oct 30 02:40:37.163593 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:37.163452 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 02:40:37.163593 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:37.163503 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-rbac-proxy-cm podName:c24d2a3f-bbb2-491e-9f25-608364102b2a nodeName:}" failed. No retries permitted until 2026-10-30 02:40:41.163489053 +0000 UTC m=+42.590699020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j2rxp" (UID: "c24d2a3f-bbb2-491e-9f25-608364102b2a") : configmap "kube-rbac-proxy" not found Oct 30 02:40:38.811563 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:38.811521 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:38.812012 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:38.811607 2576 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:38.812012 ip-10-0-139-19 kubenswrapper[2576]: I1030 02:40:38.811975 2576 scope.go:122] "RemoveContainer" containerID="86843fc740326707a729126a99a09bc7b7364f00b2a1bdfe4d7804f799d79f71" Oct 30 02:40:38.812167 ip-10-0-139-19 kubenswrapper[2576]: E1030 02:40:38.812148 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-vb5mb_openshift-console-operator(21ee3aae-ce3d-4fc0-9419-60fb4c762ad3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" podUID="21ee3aae-ce3d-4fc0-9419-60fb4c762ad3" Oct 30 02:40:38.918706 ip-10-0-139-19 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 02:40:38.934172 ip-10-0-139-19 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 02:40:38.934391 ip-10-0-139-19 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 02:40:38.934450 ip-10-0-139-19 systemd[1]: kubelet.service: Consumed 3.075s CPU time. Oct 30 02:40:38.944437 ip-10-0-139-19 systemd[1]: Starting Kubernetes Kubelet... Oct 30 02:40:39.031866 ip-10-0-139-19 kubenswrapper[4925]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 02:40:39.031866 ip-10-0-139-19 kubenswrapper[4925]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 02:40:39.031866 ip-10-0-139-19 kubenswrapper[4925]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 02:40:39.031866 ip-10-0-139-19 kubenswrapper[4925]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 02:40:39.034112 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034023 4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:40:39.034112 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034112 4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:40:39.034189 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034117 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:40:39.034189 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034120 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:40:39.034189 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034123 4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:40:39.034189 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034126 4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:40:39.034189 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034128 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:40:39.034189 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034131 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:40:39.034189 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034134 4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:40:39.034189 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034137 4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:40:39.034189 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034139 4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:40:39.034189 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034142 4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:40:39.034189 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034145 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:40:39.034189 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034148 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:40:39.034189 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034150 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:40:39.034189 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034153 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:40:39.034189 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034155 4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:40:39.034189 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034158 4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:40:39.034189 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034161 4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:40:39.034189 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034164 4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:40:39.034189 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034166 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:40:39.034189 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034169 4925 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:40:39.034703 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034172 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:40:39.034703 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034175 4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:40:39.034703 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034178 4925 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:40:39.034703 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034181 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:40:39.034703 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034184 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:40:39.034703 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034190 4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:40:39.034703 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034193 4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:40:39.034703 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034196 4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:40:39.034703 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034199 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:40:39.034703 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034202 4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:40:39.034703 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034205 4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:40:39.034703 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034208 4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:40:39.034703 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034210 4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:40:39.034703 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034213 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:40:39.034703 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034216 4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:40:39.034703 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034219 4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:40:39.034703 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034221 4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:40:39.034703 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034224 4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:40:39.034703 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034226 4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:40:39.035194 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034229 4925 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:40:39.035194 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034231 4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:40:39.035194 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034234 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:40:39.035194 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034236 4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:40:39.035194 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034239 4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:40:39.035194 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034242 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:40:39.035194 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034245 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:40:39.035194 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034248 4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:40:39.035194 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034250 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:40:39.035194 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034254 4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:40:39.035194 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034258 4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:40:39.035194 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034260 4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:40:39.035194 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034263 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:40:39.035194 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034267 4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:40:39.035194 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034272 4925 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:40:39.035194 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034276 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:40:39.035194 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034278 4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:40:39.035194 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034281 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:40:39.035194 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034284 4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:40:39.035682 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034286 4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:40:39.035682 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034289 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:40:39.035682 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034291 4925 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:40:39.035682 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034295 4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:40:39.035682 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034298 4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:40:39.035682 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034300 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:40:39.035682 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034303 4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:40:39.035682 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034305 4925 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:40:39.035682 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034308 4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:40:39.035682 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034311 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:40:39.035682 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034313 4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:40:39.035682 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034316 4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:40:39.035682 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034318 4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:40:39.035682 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034321 4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:40:39.035682 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034323 4925 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:40:39.035682 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034326 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:40:39.035682 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034329 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:40:39.035682 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034331 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:40:39.035682 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034334 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:40:39.035682 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034340 4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:40:39.036192 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034342 4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:40:39.036192 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034345 4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:40:39.036192 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034348 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:40:39.036192 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034350 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:40:39.036192 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034353 4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:40:39.036192 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034355 4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:40:39.036192 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034358 4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:40:39.036192 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034362 4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:40:39.036192 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034365 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:40:39.036192 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034367 4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:40:39.036192 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034370 4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:40:39.036192 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034372 4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:40:39.036192 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034375 4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:40:39.036192 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034377 4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:40:39.036192 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034380 4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:40:39.036192 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034383 4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:40:39.036192 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034385 4925 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:40:39.036192 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034388 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:40:39.036192 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034391 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:40:39.036192 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034393 4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:40:39.036742 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034395 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:40:39.036742 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034398 4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:40:39.036742 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034401 4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:40:39.036742 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034403 4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:40:39.036742 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034406 4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:40:39.036742 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.034409 4925 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:40:39.036742 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035169 4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:40:39.036742 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035176 4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:40:39.036742 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035179 4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:40:39.036742 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035182 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:40:39.036742 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035184 4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:40:39.036742 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035187 4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:40:39.036742 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035190 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:40:39.036742 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035193 4925 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:40:39.036742 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035198 4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:40:39.036742 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035202 4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:40:39.036742 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035205 4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:40:39.036742 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035208 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:40:39.036742 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035212 4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:40:39.037339 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035215 4925 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:40:39.037339 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035218 4925 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:40:39.037339 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035221 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:40:39.037339 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035224 4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:40:39.037339 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035226 4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:40:39.037339 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035229 4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:40:39.037339 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035232 4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:40:39.037339 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035234 4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:40:39.037339 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035237 4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:40:39.037339 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035239 4925 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:40:39.037339 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035242 4925 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:40:39.037339 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035245 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:40:39.037339 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035247 4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:40:39.037339 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035250 4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:40:39.037339 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035253 4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:40:39.037339 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035256 4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:40:39.037339 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035259 4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:40:39.037339 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035262 4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:40:39.037339 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035265 4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:40:39.037339 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035267 4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:40:39.037874 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035270 4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:40:39.037874 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035273 4925 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:40:39.037874 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035275 4925 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:40:39.037874 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035278 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:40:39.037874 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035280 4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:40:39.037874 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035283 4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:40:39.037874 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035286 4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:40:39.037874 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035288 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:40:39.037874 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035291 4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:40:39.037874 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035293 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:40:39.037874 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035296 4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:40:39.037874 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035298 4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:40:39.037874 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035300 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:40:39.037874 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035303 4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:40:39.037874 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035305 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:40:39.037874 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035308 4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:40:39.037874 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035310 4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:40:39.037874 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035313 4925 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:40:39.037874 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035315 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:40:39.037874 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035318 4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:40:39.038380 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035320 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:40:39.038380 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035323 4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:40:39.038380 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035326 4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:40:39.038380 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035328 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:40:39.038380 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035331 4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:40:39.038380 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035334 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:40:39.038380 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035336 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:40:39.038380 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035338 4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:40:39.038380 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035341 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:40:39.038380 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035345 4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:40:39.038380 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035349 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:40:39.038380 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035352 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:40:39.038380 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035355 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:40:39.038380 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035357 4925 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:40:39.038380 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035360 4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:40:39.038380 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035363 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:40:39.038380 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035365 4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:40:39.038380 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035368 4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:40:39.038380 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035371 4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:40:39.038877 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035374 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:40:39.038877 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035377 4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:40:39.038877 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035379 4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:40:39.038877 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035382 4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:40:39.038877 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035385 4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:40:39.038877 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035387 4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:40:39.038877 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035389 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:40:39.038877 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035392 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:40:39.038877 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035394 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:40:39.038877 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035397 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:40:39.038877 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035399 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:40:39.038877 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035402 4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:40:39.038877 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035404 4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:40:39.038877 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035407 4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:40:39.038877 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035409 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:40:39.038877 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035412 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:40:39.038877 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035414 4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:40:39.038877 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035417 4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:40:39.038877 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035419 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:40:39.039354 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035422 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:40:39.039354 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035425 4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:40:39.039354 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035428 4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:40:39.039354 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035431 4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:40:39.039354 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035435 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:40:39.039354 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035439 4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:40:39.039354 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035444 4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:40:39.039354 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035448 4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:40:39.039354 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035452 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:40:39.039354 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035456 4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:40:39.039354 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035460 4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:40:39.039354 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035463 4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:40:39.039354 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035465 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:40:39.039354 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035468 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:40:39.039354 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035472 4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:40:39.039354 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035987 4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:40:39.039354 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035993 4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:40:39.039354 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035996 4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:40:39.039354 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.035998 4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:40:39.039354 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036001 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:40:39.039875 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036004 4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:40:39.039875 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036007 4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:40:39.039875 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036010 4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:40:39.039875 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036012 4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:40:39.039875 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036016 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:40:39.039875 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036018 4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:40:39.039875 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036021 4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:40:39.039875 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036023 4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:40:39.039875 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036026 4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:40:39.039875 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036029 4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:40:39.039875 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036032 4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:40:39.039875 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036039 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:40:39.039875 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036041 4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:40:39.039875 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036044 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:40:39.039875 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036047 4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:40:39.039875 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036049 4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:40:39.039875 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036052 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:40:39.039875 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036054 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:40:39.039875 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036057 4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:40:39.039875 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036059 4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:40:39.040653 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036062 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:40:39.040653 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036065 4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:40:39.040653 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036067 4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:40:39.040653 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036070 4925 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:40:39.040653 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036074 4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:40:39.040653 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036078 4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:40:39.040653 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036082 4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:40:39.040653 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036084 4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:40:39.040653 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036088 4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:40:39.040653 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036094 4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:40:39.040653 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036097 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:40:39.040653 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036100 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:40:39.040653 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036102 4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:40:39.040653 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036105 4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:40:39.040653 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036108 4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:40:39.040653 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036110 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:40:39.040653 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036113 4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:40:39.040653 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036116 4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:40:39.040653 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036119 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:40:39.041528 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036121 4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:40:39.041528 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036124 4925 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:40:39.041528 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036127 4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:40:39.041528 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036129 4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:40:39.041528 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036134 4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:40:39.041528 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036137 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:40:39.041528 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036139 4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:40:39.041528 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036142 4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:40:39.041528 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036144 4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:40:39.041528 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036147 4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:40:39.041528 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036149 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:40:39.041528 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036152 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:40:39.041528 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036154 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:40:39.041528 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036157 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:40:39.041528 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036160 4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:40:39.041528 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036163 4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:40:39.041528 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036165 4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:40:39.041528 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036168 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:40:39.041528 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036170 4925 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:40:39.041528 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036173 4925 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:40:39.042268 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036176 4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:40:39.042268 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036179 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:40:39.042268 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036183 4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:40:39.042268 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036187 4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:40:39.042268 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036190 4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:40:39.042268 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036193 4925 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:40:39.042268 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036196 4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:40:39.042268 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036199 4925 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:40:39.042268 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036202 4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:40:39.042268 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036204 4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:40:39.042268 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036207 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:40:39.042268 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036209 4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:40:39.042268 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036212 4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:40:39.042268 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036215 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:40:39.042268 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036217 4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:40:39.042268 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036220 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:40:39.042268 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036224 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:40:39.042268 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036227 4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:40:39.042268 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036230 4925 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:40:39.043120 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036232 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:40:39.043120 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036235 4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:40:39.043120 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036238 4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:40:39.043120 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036240 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:40:39.043120 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036243 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:40:39.043120 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036246 4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:40:39.043120 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036248 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:40:39.043120 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036251 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:40:39.043120 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036253 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:40:39.043120 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036256 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:40:39.043120 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036258 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:40:39.043120 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036261 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:40:39.043120 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036263 4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:40:39.043120 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036265 4925 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:40:39.043120 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036268 4925 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:40:39.043120 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036272 4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:40:39.043120 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036275 4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:40:39.043120 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036278 4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:40:39.043120 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036280 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:40:39.043120 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036283 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036285 4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036288 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.036290 4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036405 4925 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036412 4925 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036419 4925 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036424 4925 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036428 4925 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036431 4925 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036437 4925 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036441 4925 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036445 4925 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036448 4925 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036451 4925 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036455 4925 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036458 4925 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036461 4925 flags.go:64] FLAG: --cgroup-root="" Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036464 4925 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036467 4925 flags.go:64] FLAG: --client-ca-file="" Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036470 4925 flags.go:64] FLAG: --cloud-provider="external" Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036473 4925 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036478 4925 flags.go:64] FLAG: --cluster-domain="" Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036481 4925 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 02:40:39.044061 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036484 4925 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036487 4925 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036490 4925 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036495 4925 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036500 4925 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036503 4925 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036506 4925 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036509 4925 flags.go:64] FLAG: --contention-profiling="false" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036512 4925 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036515 4925 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036518 4925 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036522 4925 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036526 4925 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036529 4925 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036532 4925 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036535 4925 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036538 4925 flags.go:64] FLAG: --enable-server="true" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036541 4925 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036546 4925 flags.go:64] FLAG: --event-burst="100" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036550 4925 flags.go:64] FLAG: --event-qps="50" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036553 4925 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036556 4925 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036559 4925 flags.go:64] FLAG: --eviction-hard="" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036563 4925 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 02:40:39.045185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036566 4925 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036569 4925 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036572 4925 flags.go:64] FLAG: --eviction-soft="" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036575 4925 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036579 4925 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036582 4925 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036585 4925 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036587 4925 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036590 4925 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036593 4925 flags.go:64] FLAG: --feature-gates="" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036597 4925 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036601 4925 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036605 4925 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036609 4925 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036612 4925 flags.go:64] FLAG: --healthz-port="10248" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036615 4925 flags.go:64] FLAG: --help="false" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036618 4925 flags.go:64] FLAG: --hostname-override="ip-10-0-139-19.ec2.internal" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036622 4925 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036625 4925 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036628 4925 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036631 4925 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036635 4925 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036638 4925 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 02:40:39.046315 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036640 4925 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036643 4925 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036646 4925 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036651 4925 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036655 4925 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036658 4925 flags.go:64] FLAG: --kube-reserved="" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036661 4925 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036664 4925 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036667 4925 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036670 4925 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036673 4925 flags.go:64] FLAG: --lock-file="" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036676 4925 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036679 4925 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036682 4925 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036687 4925 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036690 4925 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036693 4925 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036696 4925 flags.go:64] FLAG: --logging-format="text" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036699 4925 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036703 4925 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036706 4925 flags.go:64] FLAG: --manifest-url="" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036710 4925 flags.go:64] FLAG: --manifest-url-header="" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036714 4925 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036723 4925 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036739 4925 flags.go:64] FLAG: --max-pods="110" Oct 30 02:40:39.047325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036743 4925 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036759 4925 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036763 4925 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036766 4925 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036769 4925 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036772 4925 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036775 4925 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036782 4925 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036785 4925 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036789 4925 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036793 4925 flags.go:64] FLAG: --pod-cidr="" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036796 4925 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036799 4925 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036802 4925 flags.go:64] FLAG: --pods-per-core="0" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036805 4925 flags.go:64] FLAG: --port="10250" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036808 4925 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036811 4925 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c3dd801830cd7ae3" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036814 4925 flags.go:64] FLAG: --qos-reserved="" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036817 4925 flags.go:64] FLAG: --read-only-port="10255" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036820 4925 flags.go:64] FLAG: --register-node="true" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036823 4925 flags.go:64] FLAG: --register-with-taints="" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036827 4925 flags.go:64] FLAG: --registry-burst="10" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036830 4925 flags.go:64] FLAG: --registry-qps="5" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036833 4925 flags.go:64] FLAG: --reserved-cpus="" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036836 4925 flags.go:64] FLAG: --reserved-memory="" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036840 4925 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 02:40:39.048402 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036843 4925 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036846 4925 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036850 4925 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036853 4925 flags.go:64] FLAG: --runonce="false" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036856 4925 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036866 4925 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036869 4925 flags.go:64] FLAG: --seccomp-default="false" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036872 4925 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036875 4925 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036878 4925 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036881 4925 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036884 4925 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036887 4925 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036890 4925 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036893 4925 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036896 4925 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036900 4925 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036903 4925 flags.go:64] FLAG: --system-cgroups="" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036906 4925 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036911 4925 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036914 4925 flags.go:64] FLAG: --tls-cert-file="" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036917 4925 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036923 4925 flags.go:64] FLAG: --tls-min-version="" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036925 4925 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036928 4925 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 02:40:39.049532 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036931 4925 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036934 4925 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036937 4925 flags.go:64] FLAG: --v="2" Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036942 4925 flags.go:64] FLAG: --version="false" Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036946 4925 flags.go:64] FLAG: --vmodule="" Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036951 4925 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.036954 4925 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037114 4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037119 4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037123 4925 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037126 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037129 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037132 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037140 4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037143 4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037146 4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037149 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037151 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037154 4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037157 4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037159 4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:40:39.050620 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037162 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:40:39.051477 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037165 4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:40:39.051477 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037168 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:40:39.051477 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037171 4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:40:39.051477 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037173 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:40:39.051477 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037176 4925 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:40:39.051477 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037179 4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:40:39.051477 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037181 4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:40:39.051477 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037184 4925 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:40:39.051477 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037186 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:40:39.051477 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037189 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:40:39.051477 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037192 4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:40:39.051477 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037194 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:40:39.051477 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037197 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:40:39.051477 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037199 4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:40:39.051477 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037202 4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:40:39.051477 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037205 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:40:39.051477 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037207 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:40:39.051477 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037210 4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:40:39.051477 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037212 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:40:39.051477 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037216 4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:40:39.052100 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037219 4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:40:39.052100 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037222 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:40:39.052100 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037224 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:40:39.052100 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037227 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:40:39.052100 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037230 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:40:39.052100 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037233 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:40:39.052100 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037236 4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:40:39.052100 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037238 4925 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:40:39.052100 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037241 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:40:39.052100 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037243 4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:40:39.052100 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037246 4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:40:39.052100 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037249 4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:40:39.052100 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037253 4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:40:39.052100 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037256 4925 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:40:39.052100 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037260 4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:40:39.052100 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037263 4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:40:39.052100 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037266 4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:40:39.052100 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037269 4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:40:39.052100 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037271 4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:40:39.052592 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037274 4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:40:39.052592 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037276 4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:40:39.052592 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037279 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:40:39.052592 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037281 4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:40:39.052592 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037284 4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:40:39.052592 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037286 4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:40:39.052592 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037289 4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:40:39.052592 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037291 4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:40:39.052592 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037294 4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:40:39.052592 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037297 4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:40:39.052592 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037299 4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:40:39.052592 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037302 4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:40:39.052592 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037305 4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:40:39.052592 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037308 4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:40:39.052592 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037312 4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:40:39.052592 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037315 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:40:39.052592 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037317 4925 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:40:39.052592 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037320 4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:40:39.052592 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037323 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:40:39.053099 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037326 4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:40:39.053099 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037329 4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:40:39.053099 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037331 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:40:39.053099 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037334 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:40:39.053099 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037337 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:40:39.053099 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037340 4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:40:39.053099 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037342 4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:40:39.053099 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037345 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:40:39.053099 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037348 4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:40:39.053099 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037351 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:40:39.053099 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037354 4925 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:40:39.053099 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037357 4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:40:39.053099 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037360 4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:40:39.053099 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037364 4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:40:39.053099 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037368 4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:40:39.053099 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037370 4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:40:39.053099 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037373 4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:40:39.053099 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037376 4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:40:39.053099 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037379 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:40:39.053588 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037381 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:40:39.053588 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037384 4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:40:39.053588 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037386 4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:40:39.053588 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037389 4925 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:40:39.053588 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037391 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:40:39.053588 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037394 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:40:39.053588 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037398 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:40:39.053588 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037400 4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:40:39.053588 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037404 4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:40:39.053588 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037407 4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:40:39.053588 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037410 4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:40:39.053588 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037413 4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:40:39.053588 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037416 4925 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:40:39.053588 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037419 4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:40:39.053588 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.037447 4925 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 02:40:39.053588 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037505 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:40:39.054183 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037510 4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:40:39.054183 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037513 4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:40:39.054183 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037516 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:40:39.054183 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037519 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:40:39.054183 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037522 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:40:39.054183 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037525 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:40:39.054183 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037527 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:40:39.054183 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037530 4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:40:39.054183 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037532 4925 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:40:39.054183 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037535 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:40:39.054183 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037538 4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:40:39.054183 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037540 4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:40:39.054183 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037543 4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:40:39.054183 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037547 4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:40:39.054183 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037551 4925 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:40:39.054183 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037555 4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:40:39.054183 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037557 4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:40:39.054183 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037560 4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:40:39.054183 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037564 4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:40:39.054731 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037566 4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:40:39.054731 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037569 4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:40:39.054731 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037572 4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:40:39.054731 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037574 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:40:39.054731 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037578 4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:40:39.054731 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037583 4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:40:39.054731 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037585 4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:40:39.054731 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037588 4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:40:39.054731 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037591 4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:40:39.054731 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037593 4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:40:39.054731 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037596 4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:40:39.054731 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037599 4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:40:39.054731 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037602 4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:40:39.054731 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037605 4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:40:39.054731 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037607 4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:40:39.054731 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037610 4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:40:39.054731 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037612 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:40:39.054731 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037615 4925 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:40:39.054731 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037617 4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:40:39.055278 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037620 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:40:39.055278 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037623 4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:40:39.055278 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037625 4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:40:39.055278 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037628 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:40:39.055278 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037631 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:40:39.055278 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037633 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:40:39.055278 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037636 4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:40:39.055278 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037638 4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:40:39.055278 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037641 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:40:39.055278 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037643 4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:40:39.055278 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037646 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:40:39.055278 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037649 4925 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:40:39.055278 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037651 4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:40:39.055278 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037654 4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:40:39.055278 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037658 4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:40:39.055278 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037662 4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:40:39.055278 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037666 4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:40:39.055278 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037670 4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:40:39.055278 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037674 4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:40:39.055919 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037677 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:40:39.055919 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037680 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:40:39.055919 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037683 4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:40:39.055919 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037685 4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:40:39.055919 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037688 4925 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:40:39.055919 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037690 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:40:39.055919 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037694 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:40:39.055919 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037696 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:40:39.055919 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037699 4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:40:39.055919 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037702 4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:40:39.055919 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037704 4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:40:39.055919 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037707 4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:40:39.055919 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037709 4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:40:39.055919 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037712 4925 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:40:39.055919 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037714 4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:40:39.055919 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037717 4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:40:39.055919 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037719 4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:40:39.055919 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037722 4925 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:40:39.055919 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037725 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:40:39.055919 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037727 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:40:39.056411 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037729 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:40:39.056411 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037732 4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:40:39.056411 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037735 4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:40:39.056411 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037737 4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:40:39.056411 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037740 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:40:39.056411 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037742 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:40:39.056411 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037745 4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:40:39.056411 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037759 4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:40:39.056411 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037762 4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:40:39.056411 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037765 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:40:39.056411 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037769 4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:40:39.056411 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037773 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:40:39.056411 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037776 4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:40:39.056411 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037779 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:40:39.056411 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037781 4925 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:40:39.056411 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037784 4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:40:39.056411 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037787 4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:40:39.056411 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037789 4925 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:40:39.056411 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037792 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:40:39.056411 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037801 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:40:39.056920 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037804 4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:40:39.056920 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037807 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:40:39.056920 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037810 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:40:39.056920 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037812 4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:40:39.056920 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037815 4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:40:39.056920 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037817 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:40:39.056920 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037821 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:40:39.056920 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.037824 4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:40:39.056920 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.043841 4925 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 02:40:39.056920 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.043864 4925 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 02:40:39.056920 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.043896 4925 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 02:40:39.056920 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.043904 4925 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 02:40:39.056920 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044002 4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:40:39.056920 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044010 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:40:39.056920 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044016 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:40:39.056920 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044023 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:40:39.056920 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044028 4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:40:39.056920 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044033 4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:40:39.056920 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044038 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:40:39.056920 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044043 4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:40:39.057449 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044047 4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:40:39.057449 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044052 4925 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:40:39.057449 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044056 4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:40:39.057449 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044060 4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:40:39.057449 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044065 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:40:39.057449 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044069 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:40:39.057449 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044074 4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:40:39.057449 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044078 4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:40:39.057449 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044082 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:40:39.057449 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044086 4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:40:39.057449 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044090 4925 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:40:39.057449 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044095 4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:40:39.057449 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044099 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:40:39.057449 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044103 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:40:39.057449 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044108 4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:40:39.057449 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044116 4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:40:39.057449 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044121 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:40:39.057449 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044127 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:40:39.057449 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044132 4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:40:39.058073 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044137 4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:40:39.058073 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044144 4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:40:39.058073 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044150 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:40:39.058073 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044155 4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:40:39.058073 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044159 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:40:39.058073 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044163 4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:40:39.058073 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044168 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:40:39.058073 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044172 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:40:39.058073 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044176 4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:40:39.058073 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044180 4925 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:40:39.058073 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044185 4925 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:40:39.058073 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044189 4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:40:39.058073 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044193 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:40:39.058073 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044198 4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:40:39.058073 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044202 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:40:39.058073 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044207 4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:40:39.058073 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044211 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:40:39.058073 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044215 4925 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:40:39.058073 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044220 4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:40:39.058073 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044224 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:40:39.058733 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044228 4925 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:40:39.058733 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044232 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:40:39.058733 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044237 4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:40:39.058733 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044241 4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:40:39.058733 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044246 4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:40:39.058733 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044251 4925 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:40:39.058733 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044255 4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:40:39.058733 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044259 4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:40:39.058733 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044263 4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:40:39.058733 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044268 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:40:39.058733 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044272 4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:40:39.058733 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044276 4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:40:39.058733 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044283 4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:40:39.058733 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044290 4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:40:39.058733 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044295 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:40:39.058733 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044300 4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:40:39.058733 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044305 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:40:39.058733 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044309 4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:40:39.058733 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044315 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:40:39.059491 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044319 4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:40:39.059491 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044324 4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:40:39.059491 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044328 4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:40:39.059491 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044332 4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:40:39.059491 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044337 4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:40:39.059491 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044342 4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:40:39.059491 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044346 4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:40:39.059491 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044350 4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:40:39.059491 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044355 4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:40:39.059491 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044359 4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:40:39.059491 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044363 4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:40:39.059491 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044368 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:40:39.059491 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044372 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:40:39.059491 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044378 4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:40:39.059491 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044382 4925 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:40:39.059491 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044387 4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:40:39.059491 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044391 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:40:39.059491 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044396 4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:40:39.059491 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044400 4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:40:39.060227 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044405 4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:40:39.060227 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044409 4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:40:39.060227 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044413 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:40:39.060227 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044417 4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:40:39.060227 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044422 4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:40:39.060227 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044427 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:40:39.060227 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044431 4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:40:39.060227 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044435 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:40:39.060227 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044440 4925 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:40:39.060227 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044444 4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:40:39.060227 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044449 4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:40:39.060227 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044453 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:40:39.060227 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044457 4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:40:39.060227 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044461 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:40:39.060227 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044466 4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:40:39.060227 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044470 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:40:39.060227 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044475 4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:40:39.060227 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044479 4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:40:39.060227 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044483 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:40:39.060227 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044488 4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:40:39.060854 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044492 4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:40:39.060854 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.044535 4925 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 02:40:39.060854 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044782 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:40:39.060854 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044791 4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:40:39.060854 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044796 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:40:39.060854 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044800 4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:40:39.060854 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044805 4925 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:40:39.060854 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044812 4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:40:39.060854 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044817 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:40:39.060854 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044821 4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:40:39.060854 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044826 4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:40:39.060854 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044830 4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:40:39.060854 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044835 4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:40:39.060854 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044840 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:40:39.060854 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044844 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:40:39.060854 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044848 4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:40:39.061272 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044853 4925 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:40:39.061272 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044857 4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:40:39.061272 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044861 4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:40:39.061272 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044866 4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:40:39.061272 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044870 4925 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:40:39.061272 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044874 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:40:39.061272 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044879 4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:40:39.061272 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044883 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:40:39.061272 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044887 4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:40:39.061272 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044891 4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:40:39.061272 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044896 4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:40:39.061272 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044901 4925 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:40:39.061272 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044905 4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:40:39.061272 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044909 4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:40:39.061272 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044913 4925 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:40:39.061272 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044918 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:40:39.061272 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044922 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:40:39.061272 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044927 4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:40:39.061272 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044931 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:40:39.061272 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044935 4925 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:40:39.061792 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044939 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:40:39.061792 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044944 4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:40:39.061792 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044948 4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:40:39.061792 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044953 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:40:39.061792 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044959 4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:40:39.061792 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044963 4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:40:39.061792 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044967 4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:40:39.061792 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044973 4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:40:39.061792 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044980 4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:40:39.061792 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044985 4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:40:39.061792 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044991 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:40:39.061792 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044995 4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:40:39.061792 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.044999 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:40:39.061792 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045004 4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:40:39.061792 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045008 4925 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:40:39.061792 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045012 4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:40:39.061792 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045017 4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:40:39.061792 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045021 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:40:39.061792 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045025 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:40:39.062266 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045029 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:40:39.062266 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045034 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:40:39.062266 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045038 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:40:39.062266 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045043 4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:40:39.062266 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045047 4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:40:39.062266 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045051 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:40:39.062266 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045087 4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:40:39.062266 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045091 4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:40:39.062266 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045096 4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:40:39.062266 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045101 4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:40:39.062266 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045105 4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:40:39.062266 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045110 4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:40:39.062266 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045114 4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:40:39.062266 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045119 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:40:39.062266 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045123 4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:40:39.062266 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045128 4925 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:40:39.062266 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045132 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:40:39.062266 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045137 4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:40:39.062266 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045142 4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:40:39.062825 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045146 4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:40:39.062825 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045151 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:40:39.062825 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045155 4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:40:39.062825 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045160 4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:40:39.062825 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045167 4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:40:39.062825 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045173 4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:40:39.062825 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045178 4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:40:39.062825 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045183 4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:40:39.062825 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045188 4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:40:39.062825 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045193 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:40:39.062825 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045197 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:40:39.062825 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045201 4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:40:39.062825 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045205 4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:40:39.062825 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045209 4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:40:39.062825 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045214 4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:40:39.062825 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045218 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:40:39.062825 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045222 4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:40:39.062825 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045227 4925 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:40:39.062825 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045231 4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:40:39.063287 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045235 4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:40:39.063287 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045240 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:40:39.063287 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045245 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:40:39.063287 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045250 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:40:39.063287 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045254 4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:40:39.063287 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045258 4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:40:39.063287 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045262 4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:40:39.063287 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045267 4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:40:39.063287 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045271 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:40:39.063287 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045275 4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:40:39.063287 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045280 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:40:39.063287 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045284 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:40:39.063287 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045289 4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:40:39.063287 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045294 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:40:39.063287 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045299 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:40:39.063287 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.045344 4925 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 02:40:39.063727 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045401 4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:40:39.063727 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045409 4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:40:39.063727 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045413 4925 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:40:39.063727 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045418 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:40:39.063727 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045422 4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:40:39.063727 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045426 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:40:39.063727 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045431 4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:40:39.063727 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045435 4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:40:39.063727 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045439 4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:40:39.063727 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045444 4925 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:40:39.063727 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045449 4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:40:39.063727 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045453 4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:40:39.063727 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045458 4925 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:40:39.063727 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045462 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:40:39.063727 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045466 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:40:39.063727 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045471 4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:40:39.063727 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045475 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:40:39.063727 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045479 4925 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:40:39.063727 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045484 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:40:39.063727 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045488 4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:40:39.064226 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045492 4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:40:39.064226 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045497 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:40:39.064226 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045501 4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:40:39.064226 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045505 4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:40:39.064226 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045510 4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:40:39.064226 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045516 4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:40:39.064226 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045522 4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:40:39.064226 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045527 4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:40:39.064226 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045532 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:40:39.064226 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045537 4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:40:39.064226 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045542 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:40:39.064226 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045546 4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:40:39.064226 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045550 4925 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:40:39.064226 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045555 4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:40:39.064226 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045559 4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:40:39.064226 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045563 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:40:39.064226 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045567 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:40:39.064226 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045572 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:40:39.064226 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045576 4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:40:39.064723 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045581 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:40:39.064723 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045585 4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:40:39.064723 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045589 4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:40:39.064723 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045593 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:40:39.064723 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045598 4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:40:39.064723 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045602 4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:40:39.064723 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045607 4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:40:39.064723 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045611 4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:40:39.064723 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045615 4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:40:39.064723 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045620 4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:40:39.064723 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045624 4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:40:39.064723 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045629 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:40:39.064723 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045633 4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:40:39.064723 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045637 4925 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:40:39.064723 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045641 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:40:39.064723 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045645 4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:40:39.064723 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045649 4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:40:39.064723 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045654 4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:40:39.064723 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045658 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:40:39.065218 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045663 4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:40:39.065218 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045669 4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:40:39.065218 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045675 4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:40:39.065218 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045682 4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:40:39.065218 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045689 4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:40:39.065218 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045698 4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:40:39.065218 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045703 4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:40:39.065218 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045707 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:40:39.065218 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045712 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:40:39.065218 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045717 4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:40:39.065218 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045721 4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:40:39.065218 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045726 4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:40:39.065218 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045730 4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:40:39.065218 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045734 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:40:39.065218 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045739 4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:40:39.065218 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045743 4925 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:40:39.065218 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045765 4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:40:39.065218 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045771 4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:40:39.065218 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045781 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:40:39.065696 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045786 4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:40:39.065696 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045791 4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:40:39.065696 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045795 4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:40:39.065696 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045799 4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:40:39.065696 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045804 4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:40:39.065696 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045808 4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:40:39.065696 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045812 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:40:39.065696 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045817 4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:40:39.065696 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045821 4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:40:39.065696 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045825 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:40:39.065696 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045829 4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:40:39.065696 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045833 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:40:39.065696 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045838 4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:40:39.065696 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045842 4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:40:39.065696 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045847 4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:40:39.065696 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045851 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:40:39.065696 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045855 4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:40:39.065696 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045859 4925 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:40:39.065696 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045867 4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:40:39.066252 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045872 4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:40:39.066252 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045876 4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:40:39.066252 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045882 4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:40:39.066252 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045887 4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:40:39.066252 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045891 4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:40:39.066252 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045895 4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:40:39.066252 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045900 4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:40:39.066252 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045904 4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:40:39.066252 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045908 4925 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:40:39.066252 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:39.045912 4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:40:39.066252 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.046006 4925 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 02:40:39.066252 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.047628 4925 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 02:40:39.066252 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.047704 4925 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 02:40:39.066252 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.047995 4925 server.go:1014] "Starting client certificate rotation" Oct 30 02:40:39.066252 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.048168 4925 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 02:40:39.066252 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.048220 4925 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:34:59 +0000 UTC" deadline="2028-05-25 11:07:53.19999127 +0000 UTC" Oct 30 02:40:39.066252 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.048258 4925 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13760h27m14.151735177s" Oct 30 02:40:39.066718 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.048765 4925 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 02:40:39.066718 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.050502 4925 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 02:40:39.066718 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.051991 4925 log.go:25] "Validated CRI v1 runtime API" Oct 30 02:40:39.066718 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.053721 4925 log.go:25] "Validated CRI v1 image API" Oct 30 02:40:39.066718 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.054483 4925 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 02:40:39.066718 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.056385 4925 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9210d514-1422-44a9-b70e-0566f3bc391b:/dev/nvme0n1p4 b5ffe6e2-0bff-420a-85e3-881607245aee:/dev/nvme0n1p3] Oct 30 02:40:39.067001 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.056404 4925 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/007dd77a9212556c96196b0946373ac7249b0d5cb2dbb9235cf38af2a1b82a54/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/007dd77a9212556c96196b0946373ac7249b0d5cb2dbb9235cf38af2a1b82a54/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17a3b3a87deed02ee0d9fac868e411b2563d287e9f7c6b0c2015afc561f02154/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17a3b3a87deed02ee0d9fac868e411b2563d287e9f7c6b0c2015afc561f02154/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/205ec985195e5559c3c07665ac1ecf35cf3dcfcc72314eeb7961592015d0b1bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/205ec985195e5559c3c07665ac1ecf35cf3dcfcc72314eeb7961592015d0b1bc/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2252f4bec55cb55bf6e70ece220dec0aef9bc971cffb1f483b6a7d363f960c62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2252f4bec55cb55bf6e70ece220dec0aef9bc971cffb1f483b6a7d363f960c62/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24af01bc9dac9496ab26624f46552fdf41165c8e1d3c804b7188006bec2c6751/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24af01bc9dac9496ab26624f46552fdf41165c8e1d3c804b7188006bec2c6751/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b75e3b02471030d2f34e5c1c6e7fedcc8bf89131b0d82e8a9a5d6798fca3293/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b75e3b02471030d2f34e5c1c6e7fedcc8bf89131b0d82e8a9a5d6798fca3293/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c15c2ff2643e620839168f7bf2bc9dbb6c2a5be585801771d4401efb12a4f10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c15c2ff2643e620839168f7bf2bc9dbb6c2a5be585801771d4401efb12a4f10/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3feddf09ba135ee458f8e236272dc8edaa6e3e6f20abea4468bb585603ad82cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3feddf09ba135ee458f8e236272dc8edaa6e3e6f20abea4468bb585603ad82cc/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4499f24fbb9f197e95488eecebde2b9f24903864d1607f071fce4b074bd4bf83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4499f24fbb9f197e95488eecebde2b9f24903864d1607f071fce4b074bd4bf83/userdata/shm major:0 minor:300 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57269abdfa4bdc78872a2c1aac3d4ef10f24d18f7440889d7b28d93c0a3a0512/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57269abdfa4bdc78872a2c1aac3d4ef10f24d18f7440889d7b28d93c0a3a0512/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c3ec2955f73a0075e1619a94612ad8bd2987da07de561761850075d581d136b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c3ec2955f73a0075e1619a94612ad8bd2987da07de561761850075d581d136b/userdata/shm major:0 minor:321 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/631d8a03a67067aed1a5182295c2e02528b887eeeef12b507519128a5056b9e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/631d8a03a67067aed1a5182295c2e02528b887eeeef12b507519128a5056b9e6/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/821a01fbf56f6767e3d2f96c33227981bf771479c90ab3360e5e0d5908d9a59a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/821a01fbf56f6767e3d2f96c33227981bf771479c90ab3360e5e0d5908d9a59a/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3b38091b72a3b2330534bf067a07bbfd2c1de26cf85545d1242461bdf95b9cd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3b38091b72a3b2330534bf067a07bbfd2c1de26cf85545d1242461bdf95b9cd/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8ff87c0e760227f238eae688bf33a51b0ed7cfdf3761efa6c9aa9a1df282334/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8ff87c0e760227f238eae688bf33a51b0ed7cfdf3761efa6c9aa9a1df282334/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c21771905628738c026109124a1ea5781f6aaf7a4288b74b1ba5306d9843b98b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c21771905628738c026109124a1ea5781f6aaf7a4288b74b1ba5306d9843b98b/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c919183d1ac4d100f4eb8d32c483ac44fbd401cc70d61d3fcce1d62908f310c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c919183d1ac4d100f4eb8d32c483ac44fbd401cc70d61d3fcce1d62908f310c8/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d381d99baff9993609cbfad7dadb508331dcdab3c39ecb12958af1b9b47231c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d381d99baff9993609cbfad7dadb508331dcdab3c39ecb12958af1b9b47231c1/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e2fbcb710ca85ed9015e60ab5d3b24f22a3f019a47fa9b88ad70d270ec92383a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e2fbcb710ca85ed9015e60ab5d3b24f22a3f019a47fa9b88ad70d270ec92383a/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5c5c48eb62d87c86e2f5f5edfc784e6d16c943c7ce87bbaa00e34c55763d4f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5c5c48eb62d87c86e2f5f5edfc784e6d16c943c7ce87bbaa00e34c55763d4f2/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e64e87996858f40a90aca4b0cab20f099f2f6e3001492251b91fe45789518fb6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e64e87996858f40a90aca4b0cab20f099f2f6e3001492251b91fe45789518fb6/userdata/shm major:0 minor:295 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed70e3f25a89f353dd0328f78fa5b6fe4bb43ad9a65ec276e26ab3f525cf3bb5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed70e3f25a89f353dd0328f78fa5b6fe4bb43ad9a65ec276e26ab3f525cf3bb5/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f15520740d3037a5315cca4c5e0b1e0b856646888ff9886a230931f8e2d8e4b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f15520740d3037a5315cca4c5e0b1e0b856646888ff9886a230931f8e2d8e4b5/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02aed17c-f8e1-4fd0-bb90-4e972bd1a827/volumes/kubernetes.io~projected/kube-api-access-nnh58:{mountpoint:/var/lib/kubelet/pods/02aed17c-f8e1-4fd0-bb90-4e972bd1a827/volumes/kubernetes.io~projected/kube-api-access-nnh58 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02aed17c-f8e1-4fd0-bb90-4e972bd1a827/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/02aed17c-f8e1-4fd0-bb90-4e972bd1a827/volumes/kubernetes.io~secret/cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06222d65-ed37-48ff-9b12-19271e1f9662/volumes/kubernetes.io~projected/kube-api-access-djdvk:{mountpoint:/var/lib/kubelet/pods/06222d65-ed37-48ff-9b12-19271e1f9662/volumes/kubernetes.io~projected/kube-api-access-djdvk major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/114fc00f-05b8-4a65-89e1-5997231c05e7/volumes/kubernetes.io~projected/kube-api-access-m94g4:{mountpoint:/var/lib/kubelet/pods/114fc00f-05b8-4a65-89e1-5997231c05e7/volumes/kubernetes.io~projected/kube-api-access-m94g4 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae/volumes/kubernetes.io~projected/kube-api-access-66622:{mountpoint:/var/lib/kubelet/pods/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae/volumes/kubernetes.io~projected/kube-api-access-66622 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/164920df-32a0-48ab-830a-171a05caab08/volumes/kubernetes.io~projected/kube-api-access-gh7dx:{mountpoint:/var/lib/kubelet/pods/164920df-32a0-48ab-830a-171a05caab08/volumes/kubernetes.io~projected/kube-api-access-gh7dx major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/183230bf-332c-465f-8525-2094db22e38a/volumes/kubernetes.io~projected/kube-api-access-tjbgs:{mountpoint:/var/lib/kubelet/pods/183230bf-332c-465f-8525-2094db22e38a/volumes/kubernetes.io~projected/kube-api-access-tjbgs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bac52de-7a27-4fce-828b-9eedcfd60de5/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1bac52de-7a27-4fce-828b-9eedcfd60de5/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bac52de-7a27-4fce-828b-9eedcfd60de5/volumes/kubernetes.io~projected/kube-api-access-rckxh:{mountpoint:/var/lib/kubelet/pods/1bac52de-7a27-4fce-828b-9eedcfd60de5/volumes/kubernetes.io~projected/kube-api-access-rckxh major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bac52de-7a27-4fce-828b-9eedcfd60de5/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1bac52de-7a27-4fce-828b-9eedcfd60de5/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/volumes/kubernetes.io~projected/kube-api-access-blbrw:{mountpoint:/var/lib/kubelet/pods/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/volumes/kubernetes.io~projected/kube-api-access-blbrw major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4295fa46-8c63-4cde-8183-1895149bc2ce/volumes/kubernetes.io~projected/kube-api-access-2tdht:{mountpoint:/var/lib/kubelet/pods/4295fa46-8c63-4cde-8183-1895149bc2ce/volumes/kubernetes.io~projected/kube-api-access-2tdht major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/478f97e0-5e43-4195-9e8c-bd58cc3d6e00/volumes/kubernetes.io~projected/kube-api-access-bfkd4:{mountpoint:/var/lib/kubelet/pods/478f97e0-5e43-4195-9e8c-bd58cc3d6e00/volumes/kubernetes.io~projected/kube-api-access-bfkd4 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/547ef373-3835-4ec2-8f70-94954f34f452/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/547ef373-3835-4ec2-8f70-94954f34f452/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/547ef373-3835-4ec2-8f70-94954f34f452/volumes/kubernetes.io~projected/kube-api-access-9w87m:{mountpoint:/var/lib/kubelet/pods/547ef373-3835-4ec2-8f70-94954f34f452/volumes/kubernetes.io~projected/kube-api-access-9w87m major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/547ef373-3835-4ec2-8f70-94954f34f452/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/547ef373-3835-4ec2-8f70-94954f34f452/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/547ef373-3835-4ec2-8f70-94954f34f452/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/547ef373-3835-4ec2-8f70-94954f34f452/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/548e9394-45be-4ffa-937a-89fea1da3093/volumes/kubernetes.io~projected/kube-api-access-lvvcv:{mountpoint:/var/lib/kubelet/pods/548e9394-45be-4ffa-937a-89fea1da3093/volumes/kubernetes.io~projected/kube-api-access-lvvcv major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/548e9394-45be-4ffa-937a-89fea1da3093/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/548e9394-45be-4ffa-937a-89fea1da3093/volumes/kubernetes.io~secret/default-certificate major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/548e9394-45be-4ffa-937a-89fea1da3093/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/548e9394-45be-4ffa-937a-89fea1da3093/volumes/kubernetes.io~secret/stats-auth major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e9def4d-df87-404d-90eb-1b21b023fbd8/volumes/kubernetes.io~projected/kube-api-access-m6llj:{mountpoint:/var/lib/kubelet/pods/5e9def4d-df87-404d-90eb-1b21b023fbd8/volumes/kubernetes.io~projected/kube-api-access-m6llj major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ee44f66-3b88-4914-803d-9dcfbfb0efba/volumes/kubernetes.io~projected/kube-api-access-dqr74:{mountpoint:/var/lib/kubelet/pods/5ee44f66-3b88-4914-803d-9dcfbfb0efba/volumes/kubernetes.io~projected/kube-api-access-dqr74 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e/volumes/kubernetes.io~projected/kube-api-access-srs2l:{mountpoint:/var/lib/kubelet/pods/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e/volumes/kubernetes.io~projected/kube-api-access-srs2l major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ab12b0c-6c4b-4915-9920-9eec9c279f3b/volumes/kubernetes.io~projected/kube-api-access-qpw2q:{mountpoint:/var/lib/kubelet/pods/8ab12b0c-6c4b-4915-9920-9eec9c279f3b/volumes/kubernetes.io~projected/kube-api-access-qpw2q major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f17a0e8-ce89-4a88-98f2-c7782260534d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/8f17a0e8-ce89-4a88-98f2-c7782260534d/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96981120-5aa0-43e1-9652-8a625d23b6ff/volumes/kubernetes.io~projected/kube-api-access-lnt4b:{mountpoint:/var/lib/kubelet/pods/96981120-5aa0-43e1-9652-8a625d23b6ff/volumes/kubernetes.io~projected/kube-api-access-lnt4b major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96981120-5aa0-43e1-9652-8a625d23b6ff/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/96981120-5aa0-43e1-9652-8a625d23b6ff/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3c34ce6-91f3-42e4-ba49-a905f477571e/volumes/kubernetes.io~projected/kube-api-access-4wkjm:{mountpoint:/var/lib/kubelet/pods/a3c34ce6-91f3-42e4-ba49-a905f477571e/volumes/kubernetes.io~projected/kube-api-access-4wkjm major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0851f53-fa88-40f9-bfdb-beb956a5fb28/volumes/kubernetes.io~projected/kube-api-access-r5pxf:{mountpoint:/var/lib/kubelet/pods/b0851f53-fa88-40f9-bfdb-beb956a5fb28/volumes/kubernetes.io~projected/kube-api-access-r5pxf major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0851f53-fa88-40f9-bfdb-beb956a5fb28/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b0851f53-fa88-40f9-bfdb-beb956a5fb28/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba886f04-386a-4cb7-aa00-b85f3921e6cc/volumes/kubernetes.io~projected/kube-api-access-xjg4n:{mountpoint:/var/lib/kubelet/pods/ba886f04-386a-4cb7-aa00-b85f3921e6cc/volumes/kubernetes.io~projected/kube-api-access-xjg4n major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc230021-ead4-497f-9933-45fbdb9efc50/volumes/kubernetes.io~projected/kube-api-access-4llrw:{mountpoint:/var/lib/kubelet/pods/bc230021-ead4-497f-9933-45fbdb9efc50/volumes/kubernetes.io~projected/kube-api-access-4llrw major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c24d2a3f-bbb2-491e-9f25-608364102b2a/volumes/kubernetes.io~projected/kube-api-access-9bl6b:{mountpoint:/var/lib/kubelet/pods/c24d2a3f-bbb2-491e-9f25-608364102b2a/volumes/kubernetes.io~projected/kube-api-access-9bl6b major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2b91cd0-34ba-453f-8088-13e814a3a367/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c2b91cd0-34ba-453f-8088-13e814a3a367/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2b91cd0-34ba-453f-8088-13e814a3a367/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c2b91cd0-34ba-453f-8088-13e814a3a367/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2b91cd0-34ba-453f-8088-13e814a3a367/volumes/kubernetes.io~projected/kube-api-access-4bjwl:{mountpoint:/var/lib/kubelet/pods/c2b91cd0-34ba-453f-8088-13e814a3a367/volumes/kubernetes.io~projected/kube-api-access-4bjwl major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9becf8f-e251-42bc-a888-18c72145379d/volumes/kubernetes.io~projected/kube-api-access-64r2m:{mountpoint:/var/lib/kubelet/pods/c9becf8f-e251-42bc-a888-18c72145379d/volumes/kubernetes.io~projected/kube-api-access-64r2m major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5b2698-93b5-4b4a-8404-f22ec7840967/volumes/kubernetes.io~projected/kube-api-access-9xbl9:{mountpoint:/var/lib/kubelet/pods/da5b2698-93b5-4b4a-8404-f22ec7840967/volumes/kubernetes.io~projected/kube-api-access-9xbl9 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e17aabad-c1ca-490d-904d-ddd40b5fd741/volumes/kubernetes.io~projected/kube-api-access-2kzk2:{mountpoint:/var/lib/kubelet/pods/e17aabad-c1ca-490d-904d-ddd40b5fd741/volumes/kubernetes.io~projected/kube-api-access-2kzk2 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee2a4d7e-545a-4946-a44d-cc895441c477/volumes/kubernetes.io~projected/kube-api-access-qvtmv:{mountpoint:/var/lib/kubelet/pods/ee2a4d7e-545a-4946-a44d-cc895441c477/volumes/kubernetes.io~projected/kube-api-access-qvtmv major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef5e843c-9ceb-44f3-bfc9-c56be367804e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ef5e843c-9ceb-44f3-bfc9-c56be367804e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc8053d7-4573-4d8a-b8aa-54ad6543d84f/volumes/kubernetes.io~projected/kube-api-access-vw82d:{mountpoint:/var/lib/kubelet/pods/fc8053d7-4573-4d8a-b8aa-54ad6543d84f/volumes/kubernetes.io~projected/kube-api-access-vw82d major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ce1d4d66f75665fe6736200cdee0a7717ec44cbf86d439bcf95b2786a66414c2/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/2c8280081f49b9dcacdbc3f7354a39bcd35b4353a5c05b69e91dadb1126525ba/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5ad93089633d112058375b7e05e476a8dd6f29af0fe0713abcd9cb084978978d/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/1db0468227c42789a688ceb60e30746da8b5b02302dcd9b0a623c0b414388089/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/a8aa83402708e46a560357cee8fdb36ea057371a7340adf82032c245906246de/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b52f871bd163e29463d403dca8fb3094f291193fa1fe53c30df7d361e662448f/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5e44ab14f70efa01db23007297d4731b097ff4c276a920a4a70bb276d0225887/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/4859ed3c8cd0edd01e6981d34ecbbc4961d25d6773433ed607e611f9eb612496/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/6ef17e3b88755059347ca429b6ea1d242c6462683925ceaf2486cbd655fa06f9/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/0713999801bbf47bbcdcdfc43a3a79c968a559658ecba4345ea0bb7e30996fcd/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/997f4bfbbe5ebfd37258803a17a2329a2d8e3415280b1b0f5fac4a573dd3696d/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/dfc4f304654cb0c0084e0b85ea8f7d63a16e645bed869b96e98882eb7494f972/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/cb978ff3512326d3eed996526ce8dbe74101ea71a6c34202d2a58b61db76fde2/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/13f68c9de6b864d6c67c840f9dfad4cf11ac585c6babd42c03cb93efc4a171d1/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/560cf90cf8bf89ef3c5682bffade163ce42b1f8413fcebbd6cb5087fbc0f496e/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d8a2d8f891e1aa7ea48af9827a7e11f824513cd28ff0b30c39fb662ac9cdc10d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/1341f74f5a24a285eba8b0d08d70697ba24dc8b26ada5a103d509396bb53e796/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/5b4672694ee3dc992c3b70dcae32129151cea3c9476d7c36c256c572e01c5bc7/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/bafe2ec20f390fc37ac3e24191faf5f6160dbdb712f43184245a900148d2ac55/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/cce64e5e39a9443928cd9e595010bcb70c9b7511b0424fe3dcf85a6a4fe466d2/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d57dd2a9f45783b7959c39c4e812187704a64529e15647124c431d166bf4936c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/71cc6954207d2b649f48939aa4eab6997d8a2e60860e58968d1660be9cbf377e/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/64606523eabbff040d83ff7f6f7d26821e5ed1ab22d696f213d639df2efeabf6/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/9c0c3852a11ad696fbd7ec673705164fefa8e2d9bf1a8fab48157ed29b23e9e5/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/9635fb445ed7bfcb8a8912a633947513e26c8eebf549564ad38cb2099591a0e4/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/fa85a663b42fd0033f1b00c2816fe778d594e038669763cb7950f682e7dbf645/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/09d8d62b82ad9742d4cc8f03560377fa272ab5cd0254dfca92467859c8b80951/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/e2959f334f54015a90c4d46a8831e9221f594dadb0e633e9003f04004afb1351/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/4cfe71c3f7df032ce4adf232386c30411aaccbc8c9b4d4f0a8b2ef6fb3404850/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/c9dd9159c91d7ed0f04b09f5fe1588589584ea768d3ddcbe6dd5d871fdf36fcd/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/5f6a3c35b47957c280bafddbd6be3aadc156fe0707ae1d78ce98c04a98ca3d0e/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/baa326469fbaa1c4dd66398784cc85684b4f41f60abe2826c142c1eb5304c45d/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/454c193549307934bfa2531651762c48e2261625ca193dbde7cc9bc404a5610f/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/73acfce7a7143b2c3cbab57546f5b68c0e3f6b273a53b929430aafe43225c66a/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/8742541a0e0d06328215ed5abd88bcf8a3760f2824c938a1974927b476a9d248/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/d4cb3d2f372fbbd5d99828dec7a93d7f91edb7e8176c70bd638ffb754a369163/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/b3307c48c26081161eedbb83653227695d6255ec1b923e04dddbb707c28d7bf9/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/994f5bfe12eada9001e32ee22e552e913bc1257a046584f2deb17a389c15a1e4/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/e3a21afb69f8bcbcc65f6c25bdbfce29fc7a3081cb054011fe9ea5de196ddfa4/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/5f0b621814906dcb0693248cc85f724b5ebee44fc7a02e93f9d6d7a3e9c1fc74/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/f34912476022c58bb4e48c049db1f61b87657073cbfe3309f3cb89e84b63bbe4/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/1ffcca2d0c36f0a18f1ee0b38fb199ebd87a1965e3a0ef2b59954bfe70add299/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/2c3b3c708a2b0eb19b7b3e7ab9433bf0c40e74cb0fd58fddfb862c0621abfa4d/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/9f56abb83f8ccf6e0d06490522682734d174b7b8ac75d190d6d57b7b087d619d/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/bf0920e3da657f2af401f5093ba69f8bf749eb82486188f6f732b104d80c7d30/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/cc2e752cc3a1e29efa6e74c05733ce5bf9ea5b73c114c36ab3c11853413ab5a1/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/e52e95f0eb2e4e94aedc6373349732faf64ff3c1002dc9ef2a43fb6bfc91896f/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/50b35a162831c9eaaead7739743a288eb722e6dd16830c9679091ae5cff1bfd5/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/fe113adbfb3beb6cb800d277a4e80bdd4bd8e33447685400ed76bbde2d09a0fe/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/0d6957083a1fc071fdf5cf3644b0907e2bac0f7e0acdfa723cba1df1b720b6e1/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/aaecb4e86eed716b0de09f4976893ab4ea990077fd028a8cda19e8295a640b9a/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/ad48f6caac4c1b8653b972ca606647ff6d8819d852be39daf0989c0e623244a3/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/e73ed23ff929015178dd3086b4b30ebc06e1dd2cb8caa8782c976119032dcf79/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/dfe50577af5af6238cd1156d8241a5fade7568689c0716c6e3bd180576553556/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-95:{mountpoint:/var/lib/containers/storage/overlay/175bc3f4bfb18be673ab253f50ddb78478c80a579d870d2c09f2ced4488d8d55/merged major:0 minor:95 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/8e426d7a3b803dfb1c10003ccc543aaf132bf28ceb1f45be36e2605dc9964c8c/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 02:40:39.067249 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.065212 4925 manager.go:217] Machine: {Timestamp:2026-10-30 02:40:39.064727364 +0000 UTC m=+0.078005458 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100072 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23303e67edfa373af0977749bcfa97 SystemUUID:ec23303e-67ed-fa37-3af0-977749bcfa97 BootID:97a968b1-eefe-4f25-8bb8-8b2265ab78bb Filesystems:[{Device:/run/containers/storage/overlay-containers/821a01fbf56f6767e3d2f96c33227981bf771479c90ab3360e5e0d5908d9a59a/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c2b91cd0-34ba-453f-8088-13e814a3a367/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0851f53-fa88-40f9-bfdb-beb956a5fb28/volumes/kubernetes.io~projected/kube-api-access-r5pxf DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c21771905628738c026109124a1ea5781f6aaf7a4288b74b1ba5306d9843b98b/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ab12b0c-6c4b-4915-9920-9eec9c279f3b/volumes/kubernetes.io~projected/kube-api-access-qpw2q DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/547ef373-3835-4ec2-8f70-94954f34f452/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c24d2a3f-bbb2-491e-9f25-608364102b2a/volumes/kubernetes.io~projected/kube-api-access-9bl6b DeviceMajor:0 DeviceMinor:322 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b75e3b02471030d2f34e5c1c6e7fedcc8bf89131b0d82e8a9a5d6798fca3293/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17a3b3a87deed02ee0d9fac868e411b2563d287e9f7c6b0c2015afc561f02154/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/548e9394-45be-4ffa-937a-89fea1da3093/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/631d8a03a67067aed1a5182295c2e02528b887eeeef12b507519128a5056b9e6/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1bac52de-7a27-4fce-828b-9eedcfd60de5/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/96981120-5aa0-43e1-9652-8a625d23b6ff/volumes/kubernetes.io~projected/kube-api-access-lnt4b DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/164920df-32a0-48ab-830a-171a05caab08/volumes/kubernetes.io~projected/kube-api-access-gh7dx DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bc230021-ead4-497f-9933-45fbdb9efc50/volumes/kubernetes.io~projected/kube-api-access-4llrw DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3feddf09ba135ee458f8e236272dc8edaa6e3e6f20abea4468bb585603ad82cc/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/478f97e0-5e43-4195-9e8c-bd58cc3d6e00/volumes/kubernetes.io~projected/kube-api-access-bfkd4 DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae/volumes/kubernetes.io~projected/kube-api-access-66622 DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57269abdfa4bdc78872a2c1aac3d4ef10f24d18f7440889d7b28d93c0a3a0512/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ee2a4d7e-545a-4946-a44d-cc895441c477/volumes/kubernetes.io~projected/kube-api-access-qvtmv DeviceMajor:0 DeviceMinor:294 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1bac52de-7a27-4fce-828b-9eedcfd60de5/volumes/kubernetes.io~projected/kube-api-access-rckxh DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2252f4bec55cb55bf6e70ece220dec0aef9bc971cffb1f483b6a7d363f960c62/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/548e9394-45be-4ffa-937a-89fea1da3093/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8ff87c0e760227f238eae688bf33a51b0ed7cfdf3761efa6c9aa9a1df282334/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/205ec985195e5559c3c07665ac1ecf35cf3dcfcc72314eeb7961592015d0b1bc/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8f17a0e8-ce89-4a88-98f2-c7782260534d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b0851f53-fa88-40f9-bfdb-beb956a5fb28/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba886f04-386a-4cb7-aa00-b85f3921e6cc/volumes/kubernetes.io~projected/kube-api-access-xjg4n DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c919183d1ac4d100f4eb8d32c483ac44fbd401cc70d61d3fcce1d62908f310c8/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24af01bc9dac9496ab26624f46552fdf41165c8e1d3c804b7188006bec2c6751/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/06222d65-ed37-48ff-9b12-19271e1f9662/volumes/kubernetes.io~projected/kube-api-access-djdvk DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f15520740d3037a5315cca4c5e0b1e0b856646888ff9886a230931f8e2d8e4b5/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c3ec2955f73a0075e1619a94612ad8bd2987da07de561761850075d581d136b/userdata/shm DeviceMajor:0 DeviceMinor:321 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/1bac52de-7a27-4fce-828b-9eedcfd60de5/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fc8053d7-4573-4d8a-b8aa-54ad6543d84f/volumes/kubernetes.io~projected/kube-api-access-vw82d DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed70e3f25a89f353dd0328f78fa5b6fe4bb43ad9a65ec276e26ab3f525cf3bb5/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/96981120-5aa0-43e1-9652-8a625d23b6ff/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e2fbcb710ca85ed9015e60ab5d3b24f22a3f019a47fa9b88ad70d270ec92383a/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/548e9394-45be-4ffa-937a-89fea1da3093/volumes/kubernetes.io~projected/kube-api-access-lvvcv DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e9def4d-df87-404d-90eb-1b21b023fbd8/volumes/kubernetes.io~projected/kube-api-access-m6llj DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/547ef373-3835-4ec2-8f70-94954f34f452/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e/volumes/kubernetes.io~projected/kube-api-access-srs2l DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/da5b2698-93b5-4b4a-8404-f22ec7840967/volumes/kubernetes.io~projected/kube-api-access-9xbl9 DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3b38091b72a3b2330534bf067a07bbfd2c1de26cf85545d1242461bdf95b9cd/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/547ef373-3835-4ec2-8f70-94954f34f452/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/02aed17c-f8e1-4fd0-bb90-4e972bd1a827/volumes/kubernetes.io~projected/kube-api-access-nnh58 DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/183230bf-332c-465f-8525-2094db22e38a/volumes/kubernetes.io~projected/kube-api-access-tjbgs DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/547ef373-3835-4ec2-8f70-94954f34f452/volumes/kubernetes.io~projected/kube-api-access-9w87m DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c15c2ff2643e620839168f7bf2bc9dbb6c2a5be585801771d4401efb12a4f10/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/volumes/kubernetes.io~projected/kube-api-access-blbrw DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/007dd77a9212556c96196b0946373ac7249b0d5cb2dbb9235cf38af2a1b82a54/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5c5c48eb62d87c86e2f5f5edfc784e6d16c943c7ce87bbaa00e34c55763d4f2/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-95 DeviceMajor:0 DeviceMinor:95 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4499f24fbb9f197e95488eecebde2b9f24903864d1607f071fce4b074bd4bf83/userdata/shm DeviceMajor:0 DeviceMinor:300 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c9becf8f-e251-42bc-a888-18c72145379d/volumes/kubernetes.io~projected/kube-api-access-64r2m DeviceMajor:0 DeviceMinor:320 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/02aed17c-f8e1-4fd0-bb90-4e972bd1a827/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e64e87996858f40a90aca4b0cab20f099f2f6e3001492251b91fe45789518fb6/userdata/shm DeviceMajor:0 DeviceMinor:295 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4295fa46-8c63-4cde-8183-1895149bc2ce/volumes/kubernetes.io~projected/kube-api-access-2tdht DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/114fc00f-05b8-4a65-89e1-5997231c05e7/volumes/kubernetes.io~projected/kube-api-access-m94g4 DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5ee44f66-3b88-4914-803d-9dcfbfb0efba/volumes/kubernetes.io~projected/kube-api-access-dqr74 DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e17aabad-c1ca-490d-904d-ddd40b5fd741/volumes/kubernetes.io~projected/kube-api-access-2kzk2 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d381d99baff9993609cbfad7dadb508331dcdab3c39ecb12958af1b9b47231c1/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2b91cd0-34ba-453f-8088-13e814a3a367/volumes/kubernetes.io~projected/kube-api-access-4bjwl DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a3c34ce6-91f3-42e4-ba49-a905f477571e/volumes/kubernetes.io~projected/kube-api-access-4wkjm DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/c2b91cd0-34ba-453f-8088-13e814a3a367/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef5e843c-9ceb-44f3-bfc9-c56be367804e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:298 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:007dd77a9212556 MacAddress:0e:fc:4e:47:00:b7 Speed:10000 Mtu:8901} {Name:24af01bc9dac949 MacAddress:4e:02:83:a3:7e:99 Speed:10000 Mtu:8901} {Name:3c15c2ff2643e62 MacAddress:1e:14:48:44:ff:cb Speed:10000 Mtu:8901} {Name:3feddf09ba135ee MacAddress:9a:c3:42:d9:93:97 Speed:10000 Mtu:8901} {Name:4499f24fbb9f197 MacAddress:5a:c4:3f:72:d1:5f Speed:10000 Mtu:8901} {Name:57269abdfa4bdc7 MacAddress:8a:af:f0:8c:b7:36 Speed:10000 Mtu:8901} {Name:5c3ec2955f73a00 MacAddress:2a:d1:a7:60:1e:1c Speed:10000 Mtu:8901} {Name:a3b38091b72a3b2 MacAddress:ce:cb:e6:3d:b5:9f Speed:10000 Mtu:8901} {Name:a8ff87c0e760227 MacAddress:7a:09:39:12:b6:06 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:fc:b8:b8:5d Speed:0 Mtu:9001} {Name:br-int MacAddress:72:5b:8b:d3:42:e7 Speed:0 Mtu:8901} {Name:c21771905628738 MacAddress:f6:22:41:6d:b5:b7 Speed:10000 Mtu:8901} {Name:c919183d1ac4d10 MacAddress:92:9b:44:85:8a:77 Speed:10000 Mtu:8901} {Name:e64e87996858f40 MacAddress:0e:88:46:c1:70:ed Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:fc:b8:b8:5d Speed:0 Mtu:9001} {Name:f15520740d3037a MacAddress:3e:de:76:72:2d:5e Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ba:c1:df:dd:3d:16 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b2:ca:96:02:af:33 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 02:40:39.067249 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.065561 4925 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 02:40:39.067249 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.065606 4925 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 02:40:39.067249 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.065988 4925 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 02:40:39.067249 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.066013 4925 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-19.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 02:40:39.067249 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.066141 4925 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 02:40:39.067249 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.066148 4925 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 02:40:39.067249 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.066158 4925 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 02:40:39.067249 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.066177 4925 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 02:40:39.067249 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.066195 4925 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 02:40:39.067249 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.066370 4925 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 02:40:39.067249 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.066478 4925 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 02:40:39.067249 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.066511 4925 kubelet.go:493] "Attempting to sync node with API server" Oct 30 02:40:39.067249 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.066520 4925 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 02:40:39.067249 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.066537 4925 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 02:40:39.067249 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.066546 4925 kubelet.go:405] "Adding apiserver pod source" Oct 30 02:40:39.067249 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.066554 4925 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 02:40:39.068052 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.067678 4925 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 02:40:39.068052 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.067689 4925 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 02:40:39.068248 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.068232 4925 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 02:40:39.071077 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.071061 4925 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 02:40:39.071077 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.071074 4925 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 02:40:39.071194 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.071184 4925 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 02:40:39.071392 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.071380 4925 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 02:40:39.071424 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.071411 4925 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 02:40:39.071507 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.071498 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 02:40:39.071545 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.071513 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 02:40:39.071545 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.071519 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 02:40:39.071545 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.071525 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 02:40:39.071545 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.071531 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 02:40:39.071545 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.071537 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 02:40:39.071545 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.071542 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 02:40:39.071770 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.071548 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 02:40:39.071770 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.071555 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 02:40:39.071770 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.071561 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 02:40:39.071770 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.071575 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 02:40:39.071770 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.071584 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 02:40:39.071770 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.071628 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 02:40:39.071770 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.071634 4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 02:40:39.075016 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.075003 4925 server.go:1263] "Started kubelet" Oct 30 02:40:39.075160 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.075126 4925 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 02:40:39.075519 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.075090 4925 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 02:40:39.075519 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.075307 4925 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 02:40:39.075650 ip-10-0-139-19 systemd[1]: Started Kubernetes Kubelet. Oct 30 02:40:39.079476 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.075996 4925 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 02:40:39.079578 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.079552 4925 server.go:317] "Adding debug handlers to kubelet server" Oct 30 02:40:39.084269 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.084250 4925 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 02:40:39.084366 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.084312 4925 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 02:40:39.085286 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.084600 4925 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:35:01 +0000 UTC" deadline="2028-07-26 08:22:59.666272358 +0000 UTC" Oct 30 02:40:39.085286 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.084638 4925 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15245h42m20.581637481s" Oct 30 02:40:39.085286 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.084691 4925 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 02:40:39.085286 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.084704 4925 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 02:40:39.085286 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.084808 4925 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 02:40:39.086024 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:39.085847 4925 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-19.ec2.internal\" not found" Oct 30 02:40:39.087038 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.087017 4925 factory.go:55] Registering systemd factory Oct 30 02:40:39.087038 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.087039 4925 factory.go:223] Registration of the systemd container factory successfully Oct 30 02:40:39.087465 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.087316 4925 factory.go:153] Registering CRI-O factory Oct 30 02:40:39.087465 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.087334 4925 factory.go:223] Registration of the crio container factory successfully Oct 30 02:40:39.087465 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.087410 4925 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 02:40:39.087465 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.087439 4925 factory.go:103] Registering Raw factory Oct 30 02:40:39.087465 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.087458 4925 manager.go:1199] Started watching for new ooms in manager Oct 30 02:40:39.087901 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.087886 4925 manager.go:319] Starting recovery of all containers Oct 30 02:40:39.088460 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088410 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0851f53-fa88-40f9-bfdb-beb956a5fb28" volumeName="kubernetes.io/empty-dir/b0851f53-fa88-40f9-bfdb-beb956a5fb28-tmp" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088463 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba886f04-386a-4cb7-aa00-b85f3921e6cc" volumeName="kubernetes.io/empty-dir/ba886f04-386a-4cb7-aa00-b85f3921e6cc-tmp-dir" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088474 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02aed17c-f8e1-4fd0-bb90-4e972bd1a827" volumeName="kubernetes.io/projected/02aed17c-f8e1-4fd0-bb90-4e972bd1a827-kube-api-access-nnh58" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088482 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9becf8f-e251-42bc-a888-18c72145379d" volumeName="kubernetes.io/projected/c9becf8f-e251-42bc-a888-18c72145379d-kube-api-access-64r2m" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088491 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" volumeName="kubernetes.io/secret/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-image-registry-private-configuration" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088500 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="478f97e0-5e43-4195-9e8c-bd58cc3d6e00" volumeName="kubernetes.io/projected/478f97e0-5e43-4195-9e8c-bd58cc3d6e00-kube-api-access-bfkd4" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088525 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3c34ce6-91f3-42e4-ba49-a905f477571e" volumeName="kubernetes.io/configmap/a3c34ce6-91f3-42e4-ba49-a905f477571e-serviceca" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088534 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2b91cd0-34ba-453f-8088-13e814a3a367" volumeName="kubernetes.io/empty-dir/c2b91cd0-34ba-453f-8088-13e814a3a367-tmp" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088543 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2b91cd0-34ba-453f-8088-13e814a3a367" volumeName="kubernetes.io/projected/c2b91cd0-34ba-453f-8088-13e814a3a367-kube-api-access-4bjwl" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088552 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="164920df-32a0-48ab-830a-171a05caab08" volumeName="kubernetes.io/projected/164920df-32a0-48ab-830a-171a05caab08-kube-api-access-gh7dx" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088562 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bac52de-7a27-4fce-828b-9eedcfd60de5" volumeName="kubernetes.io/configmap/1bac52de-7a27-4fce-828b-9eedcfd60de5-env-overrides" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088571 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21ee3aae-ce3d-4fc0-9419-60fb4c762ad3" volumeName="kubernetes.io/projected/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-kube-api-access-blbrw" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088579 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4295fa46-8c63-4cde-8183-1895149bc2ce" volumeName="kubernetes.io/projected/4295fa46-8c63-4cde-8183-1895149bc2ce-kube-api-access-2tdht" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088592 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5b2698-93b5-4b4a-8404-f22ec7840967" volumeName="kubernetes.io/projected/da5b2698-93b5-4b4a-8404-f22ec7840967-kube-api-access-9xbl9" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088605 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f17a0e8-ce89-4a88-98f2-c7782260534d" volumeName="kubernetes.io/secret/8f17a0e8-ce89-4a88-98f2-c7782260534d-agent-certs" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088613 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96981120-5aa0-43e1-9652-8a625d23b6ff" volumeName="kubernetes.io/configmap/96981120-5aa0-43e1-9652-8a625d23b6ff-config" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088621 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0851f53-fa88-40f9-bfdb-beb956a5fb28" volumeName="kubernetes.io/configmap/b0851f53-fa88-40f9-bfdb-beb956a5fb28-service-ca-bundle" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088638 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c24d2a3f-bbb2-491e-9f25-608364102b2a" volumeName="kubernetes.io/empty-dir/c24d2a3f-bbb2-491e-9f25-608364102b2a-data-volume" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088646 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef5e843c-9ceb-44f3-bfc9-c56be367804e" volumeName="kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088653 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc8053d7-4573-4d8a-b8aa-54ad6543d84f" volumeName="kubernetes.io/configmap/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-cni-binary-copy" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088660 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="183230bf-332c-465f-8525-2094db22e38a" volumeName="kubernetes.io/configmap/183230bf-332c-465f-8525-2094db22e38a-cni-binary-copy" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088667 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba886f04-386a-4cb7-aa00-b85f3921e6cc" volumeName="kubernetes.io/projected/ba886f04-386a-4cb7-aa00-b85f3921e6cc-kube-api-access-xjg4n" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088674 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2b91cd0-34ba-453f-8088-13e814a3a367" volumeName="kubernetes.io/empty-dir/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-tuned" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088682 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="183230bf-332c-465f-8525-2094db22e38a" volumeName="kubernetes.io/configmap/183230bf-332c-465f-8525-2094db22e38a-multus-daemon-config" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088689 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21ee3aae-ce3d-4fc0-9419-60fb4c762ad3" volumeName="kubernetes.io/secret/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-serving-cert" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088697 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="547ef373-3835-4ec2-8f70-94954f34f452" volumeName="kubernetes.io/empty-dir/547ef373-3835-4ec2-8f70-94954f34f452-ca-trust-extracted" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088704 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e9def4d-df87-404d-90eb-1b21b023fbd8" volumeName="kubernetes.io/projected/5e9def4d-df87-404d-90eb-1b21b023fbd8-kube-api-access-m6llj" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088713 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0851f53-fa88-40f9-bfdb-beb956a5fb28" volumeName="kubernetes.io/secret/b0851f53-fa88-40f9-bfdb-beb956a5fb28-serving-cert" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088724 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba886f04-386a-4cb7-aa00-b85f3921e6cc" volumeName="kubernetes.io/configmap/ba886f04-386a-4cb7-aa00-b85f3921e6cc-config-volume" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088733 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="183230bf-332c-465f-8525-2094db22e38a" volumeName="kubernetes.io/projected/183230bf-332c-465f-8525-2094db22e38a-kube-api-access-tjbgs" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088744 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bac52de-7a27-4fce-828b-9eedcfd60de5" volumeName="kubernetes.io/projected/1bac52de-7a27-4fce-828b-9eedcfd60de5-kube-api-access-rckxh" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088768 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="547ef373-3835-4ec2-8f70-94954f34f452" volumeName="kubernetes.io/configmap/547ef373-3835-4ec2-8f70-94954f34f452-registry-certificates" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088779 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ab12b0c-6c4b-4915-9920-9eec9c279f3b" volumeName="kubernetes.io/projected/8ab12b0c-6c4b-4915-9920-9eec9c279f3b-kube-api-access-qpw2q" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088801 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c24d2a3f-bbb2-491e-9f25-608364102b2a" volumeName="kubernetes.io/projected/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-api-access-9bl6b" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088811 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21ee3aae-ce3d-4fc0-9419-60fb4c762ad3" volumeName="kubernetes.io/configmap/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-config" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088818 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fb3b957-3183-4c9e-a45f-c6930341197c" volumeName="kubernetes.io/configmap/3fb3b957-3183-4c9e-a45f-c6930341197c-nginx-conf" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088826 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="547ef373-3835-4ec2-8f70-94954f34f452" volumeName="kubernetes.io/configmap/547ef373-3835-4ec2-8f70-94954f34f452-trusted-ca" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088834 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="547ef373-3835-4ec2-8f70-94954f34f452" volumeName="kubernetes.io/secret/547ef373-3835-4ec2-8f70-94954f34f452-installation-pull-secrets" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088840 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc8053d7-4573-4d8a-b8aa-54ad6543d84f" volumeName="kubernetes.io/configmap/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088849 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" volumeName="kubernetes.io/configmap/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-certificates" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088856 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="548e9394-45be-4ffa-937a-89fea1da3093" volumeName="kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-default-certificate" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088864 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e9def4d-df87-404d-90eb-1b21b023fbd8" volumeName="kubernetes.io/empty-dir/5e9def4d-df87-404d-90eb-1b21b023fbd8-tmp-dir" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088872 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0851f53-fa88-40f9-bfdb-beb956a5fb28" volumeName="kubernetes.io/configmap/b0851f53-fa88-40f9-bfdb-beb956a5fb28-trusted-ca-bundle" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088880 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" volumeName="kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-bound-sa-token" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088887 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bac52de-7a27-4fce-828b-9eedcfd60de5" volumeName="kubernetes.io/secret/1bac52de-7a27-4fce-828b-9eedcfd60de5-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088894 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="547ef373-3835-4ec2-8f70-94954f34f452" volumeName="kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-bound-sa-token" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088902 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="547ef373-3835-4ec2-8f70-94954f34f452" volumeName="kubernetes.io/secret/547ef373-3835-4ec2-8f70-94954f34f452-image-registry-private-configuration" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088910 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ee44f66-3b88-4914-803d-9dcfbfb0efba" volumeName="kubernetes.io/configmap/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-config" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088919 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96981120-5aa0-43e1-9652-8a625d23b6ff" volumeName="kubernetes.io/projected/96981120-5aa0-43e1-9652-8a625d23b6ff-kube-api-access-lnt4b" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088927 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0851f53-fa88-40f9-bfdb-beb956a5fb28" volumeName="kubernetes.io/projected/b0851f53-fa88-40f9-bfdb-beb956a5fb28-kube-api-access-r5pxf" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088934 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc230021-ead4-497f-9933-45fbdb9efc50" volumeName="kubernetes.io/configmap/bc230021-ead4-497f-9933-45fbdb9efc50-telemetry-config" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088942 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="114fc00f-05b8-4a65-89e1-5997231c05e7" volumeName="kubernetes.io/projected/114fc00f-05b8-4a65-89e1-5997231c05e7-kube-api-access-m94g4" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088950 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" volumeName="kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-kube-api-access-66622" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088958 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21ee3aae-ce3d-4fc0-9419-60fb4c762ad3" volumeName="kubernetes.io/configmap/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-trusted-ca" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088966 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="547ef373-3835-4ec2-8f70-94954f34f452" volumeName="kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-kube-api-access-9w87m" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088974 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f17a0e8-ce89-4a88-98f2-c7782260534d" volumeName="kubernetes.io/configmap/8f17a0e8-ce89-4a88-98f2-c7782260534d-konnectivity-ca" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088986 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e17aabad-c1ca-490d-904d-ddd40b5fd741" volumeName="kubernetes.io/projected/e17aabad-c1ca-490d-904d-ddd40b5fd741-kube-api-access-2kzk2" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.088994 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ee44f66-3b88-4914-803d-9dcfbfb0efba" volumeName="kubernetes.io/projected/5ee44f66-3b88-4914-803d-9dcfbfb0efba-kube-api-access-dqr74" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089003 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96981120-5aa0-43e1-9652-8a625d23b6ff" volumeName="kubernetes.io/secret/96981120-5aa0-43e1-9652-8a625d23b6ff-serving-cert" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089011 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0851f53-fa88-40f9-bfdb-beb956a5fb28" volumeName="kubernetes.io/empty-dir/b0851f53-fa88-40f9-bfdb-beb956a5fb28-snapshots" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089018 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" volumeName="kubernetes.io/secret/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-installation-pull-secrets" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089026 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="548e9394-45be-4ffa-937a-89fea1da3093" volumeName="kubernetes.io/projected/548e9394-45be-4ffa-937a-89fea1da3093-kube-api-access-lvvcv" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089034 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e" volumeName="kubernetes.io/configmap/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e-config" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089049 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e" volumeName="kubernetes.io/secret/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e-serving-cert" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089069 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3c34ce6-91f3-42e4-ba49-a905f477571e" volumeName="kubernetes.io/projected/a3c34ce6-91f3-42e4-ba49-a905f477571e-kube-api-access-4wkjm" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089077 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc230021-ead4-497f-9933-45fbdb9efc50" volumeName="kubernetes.io/projected/bc230021-ead4-497f-9933-45fbdb9efc50-kube-api-access-4llrw" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089086 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc8053d7-4573-4d8a-b8aa-54ad6543d84f" volumeName="kubernetes.io/configmap/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089093 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc8053d7-4573-4d8a-b8aa-54ad6543d84f" volumeName="kubernetes.io/projected/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-kube-api-access-vw82d" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089100 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06222d65-ed37-48ff-9b12-19271e1f9662" volumeName="kubernetes.io/projected/06222d65-ed37-48ff-9b12-19271e1f9662-kube-api-access-djdvk" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089107 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="164920df-32a0-48ab-830a-171a05caab08" volumeName="kubernetes.io/configmap/164920df-32a0-48ab-830a-171a05caab08-iptables-alerter-script" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089114 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee2a4d7e-545a-4946-a44d-cc895441c477" volumeName="kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089122 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" volumeName="kubernetes.io/empty-dir/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-ca-trust-extracted" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089129 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bac52de-7a27-4fce-828b-9eedcfd60de5" volumeName="kubernetes.io/configmap/1bac52de-7a27-4fce-828b-9eedcfd60de5-ovnkube-script-lib" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089136 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e" volumeName="kubernetes.io/projected/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e-kube-api-access-srs2l" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089144 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02aed17c-f8e1-4fd0-bb90-4e972bd1a827" volumeName="kubernetes.io/secret/02aed17c-f8e1-4fd0-bb90-4e972bd1a827-cert" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089151 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" volumeName="kubernetes.io/configmap/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-trusted-ca" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089158 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bac52de-7a27-4fce-828b-9eedcfd60de5" volumeName="kubernetes.io/configmap/1bac52de-7a27-4fce-828b-9eedcfd60de5-ovnkube-config" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089165 4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="548e9394-45be-4ffa-937a-89fea1da3093" volumeName="kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-stats-auth" seLinuxMountContext="" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089173 4925 reconstruct.go:97] "Volume reconstruction finished" Oct 30 02:40:39.093416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.089179 4925 reconciler.go:29] "Reconciler: start to sync state" Oct 30 02:40:39.103376 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.103349 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/1.log" Oct 30 02:40:39.103962 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.103943 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/1.log" Oct 30 02:40:39.111735 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.111700 4925 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 02:40:39.112995 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.112974 4925 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 02:40:39.112995 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.112998 4925 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 02:40:39.113120 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.113015 4925 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 02:40:39.113120 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:39.113047 4925 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 02:40:39.114525 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.114499 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 02:40:39.114717 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.114686 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 02:40:39.138342 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.138301 4925 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:40:39.138342 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.138327 4925 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 02:40:39.145123 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.145096 4925 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:40:39.151066 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.151042 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 02:40:39.151455 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.151420 4925 generic.go:356] "Generic (PLEG): container finished" podID="1bac52de-7a27-4fce-828b-9eedcfd60de5" containerID="e2905818ee80361b6de23f09a060e7e9824d6cbf42371c3b92475371af8e74ae" exitCode=1 Oct 30 02:40:39.158990 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.158968 4925 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:40:39.159123 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.159104 4925 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:40:39.159194 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.159162 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/1.log" Oct 30 02:40:39.159194 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.159182 4925 generic.go:356] "Generic (PLEG): container finished" podID="21ee3aae-ce3d-4fc0-9419-60fb4c762ad3" containerID="86843fc740326707a729126a99a09bc7b7364f00b2a1bdfe4d7804f799d79f71" exitCode=255 Oct 30 02:40:39.162521 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.162497 4925 generic.go:356] "Generic (PLEG): container finished" podID="fc8053d7-4573-4d8a-b8aa-54ad6543d84f" containerID="80b8a63b4dd2dc1e16e49bcab79fd2cca0ae842575afd2ca71244185d192be6e" exitCode=0 Oct 30 02:40:39.162608 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.162520 4925 generic.go:356] "Generic (PLEG): container finished" podID="fc8053d7-4573-4d8a-b8aa-54ad6543d84f" containerID="4d26bdf35edf3836ad73e627f090dcf59a717b381266a541a107057ffb13bbf2" exitCode=0 Oct 30 02:40:39.162608 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.162534 4925 generic.go:356] "Generic (PLEG): container finished" podID="fc8053d7-4573-4d8a-b8aa-54ad6543d84f" containerID="b3d6f3561058d24be9f22338e9de67e3c78c0cba211f2dd49d126f12db06fcf9" exitCode=0 Oct 30 02:40:39.162608 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.162542 4925 generic.go:356] "Generic (PLEG): container finished" podID="fc8053d7-4573-4d8a-b8aa-54ad6543d84f" containerID="1f951b9f87d1877a340950e65b596d305adeb1f27630386162f42a194584d01b" exitCode=0 Oct 30 02:40:39.162608 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.162552 4925 generic.go:356] "Generic (PLEG): container finished" podID="fc8053d7-4573-4d8a-b8aa-54ad6543d84f" containerID="6ca2d65db303a6e699f88cede89813705750b12d2d600a70e451e4222ad66b8b" exitCode=0 Oct 30 02:40:39.162608 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.162560 4925 generic.go:356] "Generic (PLEG): container finished" podID="fc8053d7-4573-4d8a-b8aa-54ad6543d84f" containerID="e522105e3ca4076b6eb499877c9a61ac3a62a598a74ba60155be9a092f48eaa9" exitCode=0 Oct 30 02:40:39.166401 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.166370 4925 generic.go:356] "Generic (PLEG): container finished" podID="cf4d60387e18d9cce88d9d9a653a225b" containerID="5dbd685629ba6c174535543bd936fc9ebb44c53654687950c5ebf1a3d2e1312b" exitCode=0 Oct 30 02:40:39.178547 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.178499 4925 manager.go:324] Recovery completed Oct 30 02:40:39.199330 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.199302 4925 cpu_manager.go:226] "Starting" policy="none" Oct 30 02:40:39.199330 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.199321 4925 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 02:40:39.199512 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.199343 4925 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 02:40:39.199512 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.199494 4925 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 02:40:39.199611 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.199510 4925 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 02:40:39.199611 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.199535 4925 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 02:40:39.199611 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.199544 4925 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 02:40:39.199611 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.199551 4925 policy_none.go:50] "Start" Oct 30 02:40:39.199611 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.199563 4925 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 02:40:39.199611 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.199577 4925 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 02:40:39.199910 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.199779 4925 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 02:40:39.199910 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.199792 4925 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 02:40:39.199910 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.199801 4925 policy_none.go:44] "Start" Oct 30 02:40:39.204073 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.204046 4925 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 02:40:39.206593 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.206578 4925 manager.go:341] "Starting Device Plugin manager" Oct 30 02:40:39.207187 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:39.206605 4925 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 02:40:39.207248 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.207193 4925 server.go:87] "Starting device plugin registration server" Oct 30 02:40:39.207442 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.207431 4925 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 02:40:39.207487 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.207443 4925 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 02:40:39.207544 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.207525 4925 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 02:40:39.207678 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.207642 4925 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 02:40:39.207742 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.207689 4925 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 02:40:39.207742 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.207696 4925 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 02:40:39.207866 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.207736 4925 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:40:39.207677259Z","UUID":"7b8bc8f2-7794-4485-94b9-b8317023cdf8","Handler":null,"Name":"","Endpoint":""} Oct 30 02:40:39.209160 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.209142 4925 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 02:40:39.209257 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.209171 4925 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 02:40:39.209319 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:39.209273 4925 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 02:40:39.213117 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.213087 4925 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal"] Oct 30 02:40:39.213794 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.213777 4925 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" Oct 30 02:40:39.213794 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.213772 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal" event={"ID":"36dcf6a72ae2b4d471dead03fce936be","Type":"ContainerStarted","Data":"681e4122a6d10a058855b024c38aa4bdf2d89e24d7b44a1b5eadc684c4bf0737"} Oct 30 02:40:39.213946 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.213813 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal" event={"ID":"36dcf6a72ae2b4d471dead03fce936be","Type":"ContainerStarted","Data":"821a01fbf56f6767e3d2f96c33227981bf771479c90ab3360e5e0d5908d9a59a"} Oct 30 02:40:39.213946 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.213919 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" event={"ID":"cf4d60387e18d9cce88d9d9a653a225b","Type":"ContainerStarted","Data":"15757182b73257031d3a7bd3bc767128143e22bcb6022d796a8779f8e99b61f6"} Oct 30 02:40:39.213946 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.213942 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" event={"ID":"cf4d60387e18d9cce88d9d9a653a225b","Type":"ContainerDied","Data":"5dbd685629ba6c174535543bd936fc9ebb44c53654687950c5ebf1a3d2e1312b"} Oct 30 02:40:39.214085 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.213961 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" event={"ID":"cf4d60387e18d9cce88d9d9a653a225b","Type":"ContainerStarted","Data":"205ec985195e5559c3c07665ac1ecf35cf3dcfcc72314eeb7961592015d0b1bc"} Oct 30 02:40:39.214085 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.213996 4925 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal" Oct 30 02:40:39.252809 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.252785 4925 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 02:40:39.252926 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:39.252843 4925 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" Oct 30 02:40:39.252926 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.252873 4925 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 02:40:39.252926 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:39.252910 4925 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-19.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal" Oct 30 02:40:39.307899 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.307864 4925 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:40:39.309141 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.309120 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:40:39.309252 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.309155 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:40:39.309252 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.309170 4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-19.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:40:39.309252 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.309226 4925 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-19.ec2.internal" Oct 30 02:40:39.339504 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.339476 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf4d60387e18d9cce88d9d9a653a225b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal\" (UID: \"cf4d60387e18d9cce88d9d9a653a225b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" Oct 30 02:40:39.339607 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.339507 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf4d60387e18d9cce88d9d9a653a225b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal\" (UID: \"cf4d60387e18d9cce88d9d9a653a225b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" Oct 30 02:40:39.339607 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.339524 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36dcf6a72ae2b4d471dead03fce936be-config\") pod \"kube-apiserver-proxy-ip-10-0-139-19.ec2.internal\" (UID: \"36dcf6a72ae2b4d471dead03fce936be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal" Oct 30 02:40:39.348619 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.348599 4925 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-19.ec2.internal" Oct 30 02:40:39.348672 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.348666 4925 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-19.ec2.internal" Oct 30 02:40:39.440418 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.440345 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf4d60387e18d9cce88d9d9a653a225b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal\" (UID: \"cf4d60387e18d9cce88d9d9a653a225b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" Oct 30 02:40:39.440418 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.440378 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf4d60387e18d9cce88d9d9a653a225b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal\" (UID: \"cf4d60387e18d9cce88d9d9a653a225b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" Oct 30 02:40:39.440418 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.440408 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36dcf6a72ae2b4d471dead03fce936be-config\") pod \"kube-apiserver-proxy-ip-10-0-139-19.ec2.internal\" (UID: \"36dcf6a72ae2b4d471dead03fce936be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal" Oct 30 02:40:39.440624 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.440451 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36dcf6a72ae2b4d471dead03fce936be-config\") pod \"kube-apiserver-proxy-ip-10-0-139-19.ec2.internal\" (UID: \"36dcf6a72ae2b4d471dead03fce936be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal" Oct 30 02:40:39.440624 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.440463 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf4d60387e18d9cce88d9d9a653a225b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal\" (UID: \"cf4d60387e18d9cce88d9d9a653a225b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" Oct 30 02:40:39.440624 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:39.440451 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf4d60387e18d9cce88d9d9a653a225b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal\" (UID: \"cf4d60387e18d9cce88d9d9a653a225b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" Oct 30 02:40:40.067595 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.067559 4925 apiserver.go:52] "Watching apiserver" Oct 30 02:40:40.132849 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.132815 4925 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 02:40:40.133626 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.133596 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sx7m8","openshift-insights/insights-runtime-extractor-j2rxp","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6","kube-system/kas-connection-checker-685895c7c9-6tnpg","openshift-dns/node-resolver-hbr7c","kube-system/kas-connection-checker-685895c7c9-9s4nm","openshift-multus/multus-additional-cni-plugins-j5jm9","openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb","openshift-ingress-canary/ingress-canary-r779s","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jsswm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf","openshift-ingress/router-default-6bffcf78f7-rplsj","openshift-multus/network-metrics-daemon-9t4cc","openshift-ovn-kubernetes/ovnkube-node-b995w","openshift-network-operator/iptables-alerter-8gjpf","openshift-network-console/networking-console-plugin-86c478bb88-lttrq","kube-system/kas-connection-checker-685895c7c9-69v7b","openshift-cluster-node-tuning-operator/tuned-76vs8","openshift-console-operator/console-operator-7cd56b8dbc-vb5mb","openshift-multus/multus-gk587","openshift-insights/insights-operator-7c4986d449-zc4bx","openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv","openshift-network-diagnostics/network-check-source-6f78d59f99-lvg49","openshift-image-registry/node-ca-zg4qh","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal","openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg","openshift-kube-storage-version-migrator/migrator-66df5d496b-nsf8s","kube-system/konnectivity-agent-n96rg","kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal","openshift-network-diagnostics/network-check-target-7rx9c","openshift-image-registry/image-registry-65c57cd659-szcm2","kube-system/global-pull-secret-syncer-m9zrt"] Oct 30 02:40:40.134796 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.134776 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:40.135393 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.135373 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:40.137332 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.137312 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:40.137873 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.137857 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:40.138644 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.138629 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-lttrq" Oct 30 02:40:40.139312 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.139295 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:40.148423 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.148400 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:40.149616 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.149600 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:40.150590 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.150574 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:40.169232 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.169213 4925 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" Oct 30 02:40:40.169347 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.169333 4925 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal" Oct 30 02:40:40.187122 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.187093 4925 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 02:40:40.199462 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.199440 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 02:40:40.199772 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.199727 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 02:40:40.199865 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.199815 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:40:40.199865 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.199822 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 02:40:40.199865 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.199849 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 02:40:40.200009 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.199862 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 02:40:40.200009 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.199873 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:40:40.200009 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.199852 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 02:40:40.200009 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.199917 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 02:40:40.200009 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.199927 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 02:40:40.200009 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.199941 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 02:40:40.200009 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.199954 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 02:40:40.200009 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.199970 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 02:40:40.200009 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.199980 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:40:40.200009 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.199986 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:40:40.200009 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.199981 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 02:40:40.200009 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200005 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 02:40:40.200009 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200015 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 02:40:40.200516 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200085 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 02:40:40.200516 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200108 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 02:40:40.200516 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200133 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 02:40:40.200516 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200150 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 02:40:40.200516 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200118 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 02:40:40.200516 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200136 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 02:40:40.200516 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200188 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 02:40:40.200516 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200202 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 02:40:40.200516 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200216 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 02:40:40.200516 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200236 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 02:40:40.200516 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200264 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 02:40:40.200516 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200277 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 02:40:40.200516 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200270 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 02:40:40.200516 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200268 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:40:40.200516 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200348 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 02:40:40.200516 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200397 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 02:40:40.200516 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200427 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 02:40:40.201072 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200552 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6t67f\"" Oct 30 02:40:40.201072 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200573 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 02:40:40.201072 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200605 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 02:40:40.201072 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200610 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-g86t2\"" Oct 30 02:40:40.201072 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200614 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 02:40:40.201072 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200626 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 02:40:40.201072 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200627 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 02:40:40.201072 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200654 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 02:40:40.201072 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200697 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 02:40:40.201072 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.200766 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 02:40:40.213827 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.213800 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-npcvz\"" Oct 30 02:40:40.213901 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.213858 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 02:40:40.213901 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.213873 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-pr25d\"" Oct 30 02:40:40.213901 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.213862 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-s6lr5\"" Oct 30 02:40:40.214119 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.214099 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 02:40:40.214119 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.214110 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-5bpds\"" Oct 30 02:40:40.214300 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.214126 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rpstg\"" Oct 30 02:40:40.214300 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.214131 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 02:40:40.214300 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.214130 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pjcb2\"" Oct 30 02:40:40.214300 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.214161 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-2549v\"" Oct 30 02:40:40.214456 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.214386 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-zglns\"" Oct 30 02:40:40.214456 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.214450 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dfk55\"" Oct 30 02:40:40.214663 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.214650 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-74jzq\"" Oct 30 02:40:40.214711 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.214693 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q9zqs\"" Oct 30 02:40:40.215117 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.215103 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lz8p4\"" Oct 30 02:40:40.227940 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.227922 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 02:40:40.228217 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.228205 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7tzfh\"" Oct 30 02:40:40.228428 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.228413 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 02:40:40.228712 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.228695 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 02:40:40.229169 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.229151 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-csvhs\"" Oct 30 02:40:40.229328 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.229305 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 02:40:40.229388 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.229335 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 02:40:40.229436 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.229388 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 02:40:40.229436 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.229409 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 02:40:40.229526 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.229516 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:40:40.229584 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.229567 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 02:40:40.229584 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.229572 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 02:40:40.229674 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.229627 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 02:40:40.229674 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.229630 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 02:40:40.229674 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.229637 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 02:40:40.229813 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.229679 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 02:40:40.229813 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.229706 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 02:40:40.229922 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.229819 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 02:40:40.229922 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.229890 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:40:40.229922 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.229892 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 02:40:40.230034 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.229923 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 02:40:40.230034 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.229954 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 02:40:40.230034 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.229982 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 02:40:40.230034 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.230010 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 02:40:40.230034 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.229999 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 02:40:40.230247 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.230045 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 02:40:40.230247 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.230049 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 02:40:40.230247 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.230064 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 02:40:40.230247 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.230089 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 02:40:40.230247 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.230099 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 02:40:40.230247 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.230188 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xkr2x\"" Oct 30 02:40:40.231041 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.231019 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 02:40:40.231108 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.231082 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2trkp\"" Oct 30 02:40:40.231253 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.231233 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fd952\"" Oct 30 02:40:40.231483 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.231466 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ksjrs\"" Oct 30 02:40:40.231575 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.231497 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 02:40:40.231659 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.231549 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nd26x\"" Oct 30 02:40:40.231810 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.231787 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nmhn4\"" Oct 30 02:40:40.232133 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.232116 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 02:40:40.232306 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.232286 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 02:40:40.232366 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.232345 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 02:40:40.232496 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.232347 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8td9f\"" Oct 30 02:40:40.232679 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.232661 4925 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 02:40:40.232814 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.232729 4925 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-19.ec2.internal" Oct 30 02:40:40.232814 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.232736 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p7m4t\"" Oct 30 02:40:40.233060 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.233030 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cb5f2\"" Oct 30 02:40:40.233329 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.233304 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 02:40:40.233412 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.233357 4925 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 02:40:40.233412 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.233382 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 02:40:40.233505 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.233410 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jvpkf\"" Oct 30 02:40:40.233505 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.233410 4925 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-19.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-19.ec2.internal" Oct 30 02:40:40.233619 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.233602 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2qbtl\"" Oct 30 02:40:40.234567 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.234549 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 02:40:40.247934 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.247913 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0851f53-fa88-40f9-bfdb-beb956a5fb28-serving-cert\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:40.248051 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.247945 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1bac52de-7a27-4fce-828b-9eedcfd60de5-ovnkube-config\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.248051 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.247975 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-cnibin\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.248051 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.247999 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6llj\" (UniqueName: \"kubernetes.io/projected/5e9def4d-df87-404d-90eb-1b21b023fbd8-kube-api-access-m6llj\") pod \"node-resolver-hbr7c\" (UID: \"5e9def4d-df87-404d-90eb-1b21b023fbd8\") " pod="openshift-dns/node-resolver-hbr7c" Oct 30 02:40:40.248051 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248024 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfkd4\" (UniqueName: \"kubernetes.io/projected/478f97e0-5e43-4195-9e8c-bd58cc3d6e00-kube-api-access-bfkd4\") pod \"network-check-source-6f78d59f99-lvg49\" (UID: \"478f97e0-5e43-4195-9e8c-bd58cc3d6e00\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lvg49" Oct 30 02:40:40.248242 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248073 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tdht\" (UniqueName: \"kubernetes.io/projected/4295fa46-8c63-4cde-8183-1895149bc2ce-kube-api-access-2tdht\") pod \"volume-data-source-validator-58c9f5f46d-jsswm\" (UID: \"4295fa46-8c63-4cde-8183-1895149bc2ce\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jsswm" Oct 30 02:40:40.248242 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248106 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-run-ovn-kubernetes\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.248242 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248142 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-var-lib-cni-multus\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.248242 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248170 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-image-registry-private-configuration\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:40.248242 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248201 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0851f53-fa88-40f9-bfdb-beb956a5fb28-serving-cert\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:40.248242 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248219 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e9def4d-df87-404d-90eb-1b21b023fbd8-tmp-dir\") pod \"node-resolver-hbr7c\" (UID: \"5e9def4d-df87-404d-90eb-1b21b023fbd8\") " pod="openshift-dns/node-resolver-hbr7c" Oct 30 02:40:40.248242 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248228 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1bac52de-7a27-4fce-828b-9eedcfd60de5-ovnkube-config\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.248543 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248308 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2b91cd0-34ba-453f-8088-13e814a3a367-tmp\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.248543 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248332 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-systemd-units\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.248543 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248395 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-image-registry-private-configuration\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:40.248543 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248406 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-run-netns\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.248543 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248417 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e9def4d-df87-404d-90eb-1b21b023fbd8-tmp-dir\") pod \"node-resolver-hbr7c\" (UID: \"5e9def4d-df87-404d-90eb-1b21b023fbd8\") " pod="openshift-dns/node-resolver-hbr7c" Oct 30 02:40:40.248543 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248436 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blbrw\" (UniqueName: \"kubernetes.io/projected/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-kube-api-access-blbrw\") pod \"console-operator-7cd56b8dbc-vb5mb\" (UID: \"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:40.248543 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248438 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2b91cd0-34ba-453f-8088-13e814a3a367-tmp\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.248543 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248475 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvtmv\" (UniqueName: \"kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv\") pod \"network-check-target-7rx9c\" (UID: \"ee2a4d7e-545a-4946-a44d-cc895441c477\") " pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:40.248870 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248584 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-sys\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.248870 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248690 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-var-lib-kubelet\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.248870 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248734 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-host\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.248870 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248797 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-metrics-certs\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:40.248870 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248835 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rckxh\" (UniqueName: \"kubernetes.io/projected/1bac52de-7a27-4fce-828b-9eedcfd60de5-kube-api-access-rckxh\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.248870 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248852 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-config\") pod \"console-operator-7cd56b8dbc-vb5mb\" (UID: \"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:40.248870 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248868 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/183230bf-332c-465f-8525-2094db22e38a-multus-daemon-config\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.249138 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.248935 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-run-multus-certs\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.249138 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249032 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m94g4\" (UniqueName: \"kubernetes.io/projected/114fc00f-05b8-4a65-89e1-5997231c05e7-kube-api-access-m94g4\") pod \"kas-connection-checker-685895c7c9-9s4nm\" (UID: \"114fc00f-05b8-4a65-89e1-5997231c05e7\") " pod="kube-system/kas-connection-checker-685895c7c9-9s4nm" Oct 30 02:40:40.249138 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249072 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-config\") pod \"console-operator-7cd56b8dbc-vb5mb\" (UID: \"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:40.249138 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249075 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-modprobe-d\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.249138 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249118 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-sysctl-d\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.249375 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249157 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-socket-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:40.249375 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249174 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-slash\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.249375 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249190 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-cni-binary-copy\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:40.249375 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249202 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/183230bf-332c-465f-8525-2094db22e38a-multus-daemon-config\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.249375 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249205 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:40.249375 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249237 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-kubernetes\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.249375 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249272 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-etc-selinux\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:40.249375 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249313 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdvk\" (UniqueName: \"kubernetes.io/projected/06222d65-ed37-48ff-9b12-19271e1f9662-kube-api-access-djdvk\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:40.249375 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249345 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3fb3b957-3183-4c9e-a45f-c6930341197c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-lttrq\" (UID: \"3fb3b957-3183-4c9e-a45f-c6930341197c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lttrq" Oct 30 02:40:40.249375 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249374 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:40.249924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249398 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-multus-cni-dir\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.249924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249424 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-ca-trust-extracted\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:40.249924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249447 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-trusted-ca\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:40.249924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249479 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0851f53-fa88-40f9-bfdb-beb956a5fb28-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:40.249924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249487 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-cni-binary-copy\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:40.249924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249527 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-ca-trust-extracted\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:40.249924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249540 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lttrq\" (UID: \"3fb3b957-3183-4c9e-a45f-c6930341197c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lttrq" Oct 30 02:40:40.249924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249583 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3fb3b957-3183-4c9e-a45f-c6930341197c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-lttrq\" (UID: \"3fb3b957-3183-4c9e-a45f-c6930341197c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lttrq" Oct 30 02:40:40.249924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249624 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-etc-openvswitch\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.249924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249663 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-bound-sa-token\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:40.249924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249679 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-system-cni-dir\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.249924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249706 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-run-netns\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.249924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249814 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjg4n\" (UniqueName: \"kubernetes.io/projected/ba886f04-386a-4cb7-aa00-b85f3921e6cc-kube-api-access-xjg4n\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:40.249924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249840 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-run\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.249924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249858 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:40.249924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249884 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c24d2a3f-bbb2-491e-9f25-608364102b2a-crio-socket\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:40.249924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249817 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0851f53-fa88-40f9-bfdb-beb956a5fb28-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:40.249924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249817 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-trusted-ca\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:40.249924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249930 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249954 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64r2m\" (UniqueName: \"kubernetes.io/projected/c9becf8f-e251-42bc-a888-18c72145379d-kube-api-access-64r2m\") pod \"migrator-66df5d496b-nsf8s\" (UID: \"c9becf8f-e251-42bc-a888-18c72145379d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nsf8s" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.249973 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66622\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-kube-api-access-66622\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250000 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f17a0e8-ce89-4a88-98f2-c7782260534d-konnectivity-ca\") pod \"konnectivity-agent-n96rg\" (UID: \"8f17a0e8-ce89-4a88-98f2-c7782260534d\") " pod="kube-system/konnectivity-agent-n96rg" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250026 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnh58\" (UniqueName: \"kubernetes.io/projected/02aed17c-f8e1-4fd0-bb90-4e972bd1a827-kube-api-access-nnh58\") pod \"ingress-canary-r779s\" (UID: \"02aed17c-f8e1-4fd0-bb90-4e972bd1a827\") " pod="openshift-ingress-canary/ingress-canary-r779s" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250051 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh7dx\" (UniqueName: \"kubernetes.io/projected/164920df-32a0-48ab-830a-171a05caab08-kube-api-access-gh7dx\") pod \"iptables-alerter-8gjpf\" (UID: \"164920df-32a0-48ab-830a-171a05caab08\") " pod="openshift-network-operator/iptables-alerter-8gjpf" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250076 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250089 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250104 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5e9def4d-df87-404d-90eb-1b21b023fbd8-hosts-file\") pod \"node-resolver-hbr7c\" (UID: \"5e9def4d-df87-404d-90eb-1b21b023fbd8\") " pod="openshift-dns/node-resolver-hbr7c" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250131 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0851f53-fa88-40f9-bfdb-beb956a5fb28-tmp\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250173 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kzk2\" (UniqueName: \"kubernetes.io/projected/e17aabad-c1ca-490d-904d-ddd40b5fd741-kube-api-access-2kzk2\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250206 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0851f53-fa88-40f9-bfdb-beb956a5fb28-tmp\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250213 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250223 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f17a0e8-ce89-4a88-98f2-c7782260534d-konnectivity-ca\") pod \"konnectivity-agent-n96rg\" (UID: \"8f17a0e8-ce89-4a88-98f2-c7782260534d\") " pod="kube-system/konnectivity-agent-n96rg" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250262 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bl6b\" (UniqueName: \"kubernetes.io/projected/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-api-access-9bl6b\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250291 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-sysctl-conf\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250316 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-device-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250360 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-cnibin\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250392 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1bac52de-7a27-4fce-828b-9eedcfd60de5-ovnkube-script-lib\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250439 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wkjm\" (UniqueName: \"kubernetes.io/projected/a3c34ce6-91f3-42e4-ba49-a905f477571e-kube-api-access-4wkjm\") pod \"node-ca-zg4qh\" (UID: \"a3c34ce6-91f3-42e4-ba49-a905f477571e\") " pod="openshift-image-registry/node-ca-zg4qh" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250478 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/547ef373-3835-4ec2-8f70-94954f34f452-trusted-ca\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250496 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/183230bf-332c-465f-8525-2094db22e38a-cni-binary-copy\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250514 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-system-cni-dir\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250532 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-multus-socket-dir-parent\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250548 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-installation-pull-secrets\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250571 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:40.250612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250577 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1bac52de-7a27-4fce-828b-9eedcfd60de5-ovnkube-script-lib\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250677 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-registration-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250730 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-installation-pull-secrets\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250738 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/183230bf-332c-465f-8525-2094db22e38a-cni-binary-copy\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250731 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvvcv\" (UniqueName: \"kubernetes.io/projected/548e9394-45be-4ffa-937a-89fea1da3093-kube-api-access-lvvcv\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250808 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/547ef373-3835-4ec2-8f70-94954f34f452-ca-trust-extracted\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250822 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/547ef373-3835-4ec2-8f70-94954f34f452-trusted-ca\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250838 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250840 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef5e843c-9ceb-44f3-bfc9-c56be367804e-original-pull-secret\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250869 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4llrw\" (UniqueName: \"kubernetes.io/projected/bc230021-ead4-497f-9933-45fbdb9efc50-kube-api-access-4llrw\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250885 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/547ef373-3835-4ec2-8f70-94954f34f452-ca-trust-extracted\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250895 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-var-lib-cni-bin\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250921 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bjwl\" (UniqueName: \"kubernetes.io/projected/c2b91cd0-34ba-453f-8088-13e814a3a367-kube-api-access-4bjwl\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250946 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw82d\" (UniqueName: \"kubernetes.io/projected/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-kube-api-access-vw82d\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.250981 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/547ef373-3835-4ec2-8f70-94954f34f452-registry-certificates\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251048 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5pxf\" (UniqueName: \"kubernetes.io/projected/b0851f53-fa88-40f9-bfdb-beb956a5fb28-kube-api-access-r5pxf\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251081 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-cni-bin\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251107 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-cni-netd\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251126 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/547ef373-3835-4ec2-8f70-94954f34f452-registry-certificates\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251133 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-run-k8s-cni-cncf-io\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251173 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-trusted-ca\") pod \"console-operator-7cd56b8dbc-vb5mb\" (UID: \"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251202 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnt4b\" (UniqueName: \"kubernetes.io/projected/96981120-5aa0-43e1-9652-8a625d23b6ff-kube-api-access-lnt4b\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6\" (UID: \"96981120-5aa0-43e1-9652-8a625d23b6ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251239 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ba886f04-386a-4cb7-aa00-b85f3921e6cc-config-volume\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251264 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/164920df-32a0-48ab-830a-171a05caab08-host-slash\") pod \"iptables-alerter-8gjpf\" (UID: \"164920df-32a0-48ab-830a-171a05caab08\") " pod="openshift-network-operator/iptables-alerter-8gjpf" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251289 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/547ef373-3835-4ec2-8f70-94954f34f452-installation-pull-secrets\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251331 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjbgs\" (UniqueName: \"kubernetes.io/projected/183230bf-332c-465f-8525-2094db22e38a-kube-api-access-tjbgs\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251408 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-trusted-ca\") pod \"console-operator-7cd56b8dbc-vb5mb\" (UID: \"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251434 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-certificates\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251487 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ba886f04-386a-4cb7-aa00-b85f3921e6cc-config-volume\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251530 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-log-socket\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251560 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251561 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-certificates\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251603 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251642 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-sysconfig\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251655 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/547ef373-3835-4ec2-8f70-94954f34f452-installation-pull-secrets\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251664 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251701 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-stats-auth\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251732 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251788 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96981120-5aa0-43e1-9652-8a625d23b6ff-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6\" (UID: \"96981120-5aa0-43e1-9652-8a625d23b6ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251826 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-run-openvswitch\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.251841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251843 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w87m\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-kube-api-access-9w87m\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251883 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpw2q\" (UniqueName: \"kubernetes.io/projected/8ab12b0c-6c4b-4915-9920-9eec9c279f3b-kube-api-access-qpw2q\") pod \"kas-connection-checker-685895c7c9-69v7b\" (UID: \"8ab12b0c-6c4b-4915-9920-9eec9c279f3b\") " pod="kube-system/kas-connection-checker-685895c7c9-69v7b" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251905 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-stats-auth\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.251912 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0851f53-fa88-40f9-bfdb-beb956a5fb28-service-ca-bundle\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252002 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-node-log\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252056 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96981120-5aa0-43e1-9652-8a625d23b6ff-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6\" (UID: \"96981120-5aa0-43e1-9652-8a625d23b6ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252096 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96981120-5aa0-43e1-9652-8a625d23b6ff-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6\" (UID: \"96981120-5aa0-43e1-9652-8a625d23b6ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252124 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-var-lib-kubelet\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252128 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0851f53-fa88-40f9-bfdb-beb956a5fb28-service-ca-bundle\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252143 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef5e843c-9ceb-44f3-bfc9-c56be367804e-dbus\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252169 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b0851f53-fa88-40f9-bfdb-beb956a5fb28-snapshots\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252265 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c24d2a3f-bbb2-491e-9f25-608364102b2a-data-volume\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252287 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqr74\" (UniqueName: \"kubernetes.io/projected/5ee44f66-3b88-4914-803d-9dcfbfb0efba-kube-api-access-dqr74\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252304 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252371 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b0851f53-fa88-40f9-bfdb-beb956a5fb28-snapshots\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252374 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c24d2a3f-bbb2-491e-9f25-608364102b2a-data-volume\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252392 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-systemd\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252409 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e-serving-cert\") pod \"service-ca-operator-544d749c8c-n22dv\" (UID: \"6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252424 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-run-systemd\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252426 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96981120-5aa0-43e1-9652-8a625d23b6ff-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6\" (UID: \"96981120-5aa0-43e1-9652-8a625d23b6ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252441 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1bac52de-7a27-4fce-828b-9eedcfd60de5-env-overrides\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252468 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3c34ce6-91f3-42e4-ba49-a905f477571e-host\") pod \"node-ca-zg4qh\" (UID: \"a3c34ce6-91f3-42e4-ba49-a905f477571e\") " pod="openshift-image-registry/node-ca-zg4qh" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252525 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02aed17c-f8e1-4fd0-bb90-4e972bd1a827-cert\") pod \"ingress-canary-r779s\" (UID: \"02aed17c-f8e1-4fd0-bb90-4e972bd1a827\") " pod="openshift-ingress-canary/ingress-canary-r779s" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252542 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1bac52de-7a27-4fce-828b-9eedcfd60de5-env-overrides\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252555 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-var-lib-openvswitch\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252580 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-etc-kubernetes\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252600 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e-serving-cert\") pod \"service-ca-operator-544d749c8c-n22dv\" (UID: \"6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252610 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f17a0e8-ce89-4a88-98f2-c7782260534d-agent-certs\") pod \"konnectivity-agent-n96rg\" (UID: \"8f17a0e8-ce89-4a88-98f2-c7782260534d\") " pod="kube-system/konnectivity-agent-n96rg" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252642 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e-config\") pod \"service-ca-operator-544d749c8c-n22dv\" (UID: \"6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252676 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02aed17c-f8e1-4fd0-bb90-4e972bd1a827-cert\") pod \"ingress-canary-r779s\" (UID: \"02aed17c-f8e1-4fd0-bb90-4e972bd1a827\") " pod="openshift-ingress-canary/ingress-canary-r779s" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252676 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-kubelet\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252801 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-serving-cert\") pod \"console-operator-7cd56b8dbc-vb5mb\" (UID: \"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252808 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e-config\") pod \"service-ca-operator-544d749c8c-n22dv\" (UID: \"6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252832 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xbl9\" (UniqueName: \"kubernetes.io/projected/da5b2698-93b5-4b4a-8404-f22ec7840967-kube-api-access-9xbl9\") pod \"kas-connection-checker-685895c7c9-6tnpg\" (UID: \"da5b2698-93b5-4b4a-8404-f22ec7840967\") " pod="kube-system/kas-connection-checker-685895c7c9-6tnpg" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252831 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f17a0e8-ce89-4a88-98f2-c7782260534d-agent-certs\") pod \"konnectivity-agent-n96rg\" (UID: \"8f17a0e8-ce89-4a88-98f2-c7782260534d\") " pod="kube-system/konnectivity-agent-n96rg" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252917 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-serving-cert\") pod \"console-operator-7cd56b8dbc-vb5mb\" (UID: \"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252942 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-run-ovn\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.252972 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/547ef373-3835-4ec2-8f70-94954f34f452-image-registry-private-configuration\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253003 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-hostroot\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.253042 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253040 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba886f04-386a-4cb7-aa00-b85f3921e6cc-tmp-dir\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253080 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-tuned\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253127 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba886f04-386a-4cb7-aa00-b85f3921e6cc-tmp-dir\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253160 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-sys-fs\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253166 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-tuned\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253178 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/547ef373-3835-4ec2-8f70-94954f34f452-image-registry-private-configuration\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253208 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-default-certificate\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253240 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srs2l\" (UniqueName: \"kubernetes.io/projected/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e-kube-api-access-srs2l\") pod \"service-ca-operator-544d749c8c-n22dv\" (UID: \"6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253341 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3c34ce6-91f3-42e4-ba49-a905f477571e-serviceca\") pod \"node-ca-zg4qh\" (UID: \"a3c34ce6-91f3-42e4-ba49-a905f477571e\") " pod="openshift-image-registry/node-ca-zg4qh" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253360 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-default-certificate\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253411 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-multus-conf-dir\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253443 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-lib-modules\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253460 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253491 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bac52de-7a27-4fce-828b-9eedcfd60de5-ovn-node-metrics-cert\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253555 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3c34ce6-91f3-42e4-ba49-a905f477571e-serviceca\") pod \"node-ca-zg4qh\" (UID: \"a3c34ce6-91f3-42e4-ba49-a905f477571e\") " pod="openshift-image-registry/node-ca-zg4qh" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253587 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef5e843c-9ceb-44f3-bfc9-c56be367804e-kubelet-config\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253628 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/164920df-32a0-48ab-830a-171a05caab08-iptables-alerter-script\") pod \"iptables-alerter-8gjpf\" (UID: \"164920df-32a0-48ab-830a-171a05caab08\") " pod="openshift-network-operator/iptables-alerter-8gjpf" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253638 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253671 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253705 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bac52de-7a27-4fce-828b-9eedcfd60de5-ovn-node-metrics-cert\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253763 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc230021-ead4-497f-9933-45fbdb9efc50-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253800 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-bound-sa-token\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253831 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253849 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/164920df-32a0-48ab-830a-171a05caab08-iptables-alerter-script\") pod \"iptables-alerter-8gjpf\" (UID: \"164920df-32a0-48ab-830a-171a05caab08\") " pod="openshift-network-operator/iptables-alerter-8gjpf" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253861 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:40.254395 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.253959 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc230021-ead4-497f-9933-45fbdb9efc50-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:40.285035 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.285004 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfkd4\" (UniqueName: \"kubernetes.io/projected/478f97e0-5e43-4195-9e8c-bd58cc3d6e00-kube-api-access-bfkd4\") pod \"network-check-source-6f78d59f99-lvg49\" (UID: \"478f97e0-5e43-4195-9e8c-bd58cc3d6e00\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lvg49" Oct 30 02:40:40.292336 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.292310 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh7dx\" (UniqueName: \"kubernetes.io/projected/164920df-32a0-48ab-830a-171a05caab08-kube-api-access-gh7dx\") pod \"iptables-alerter-8gjpf\" (UID: \"164920df-32a0-48ab-830a-171a05caab08\") " pod="openshift-network-operator/iptables-alerter-8gjpf" Oct 30 02:40:40.292451 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.292437 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blbrw\" (UniqueName: \"kubernetes.io/projected/21ee3aae-ce3d-4fc0-9419-60fb4c762ad3-kube-api-access-blbrw\") pod \"console-operator-7cd56b8dbc-vb5mb\" (UID: \"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:40.292676 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.292654 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4llrw\" (UniqueName: \"kubernetes.io/projected/bc230021-ead4-497f-9933-45fbdb9efc50-kube-api-access-4llrw\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:40.292775 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.292743 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnh58\" (UniqueName: \"kubernetes.io/projected/02aed17c-f8e1-4fd0-bb90-4e972bd1a827-kube-api-access-nnh58\") pod \"ingress-canary-r779s\" (UID: \"02aed17c-f8e1-4fd0-bb90-4e972bd1a827\") " pod="openshift-ingress-canary/ingress-canary-r779s" Oct 30 02:40:40.295461 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.295443 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw82d\" (UniqueName: \"kubernetes.io/projected/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-kube-api-access-vw82d\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:40.297284 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.297268 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66622\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-kube-api-access-66622\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:40.298138 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.298119 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tdht\" (UniqueName: \"kubernetes.io/projected/4295fa46-8c63-4cde-8183-1895149bc2ce-kube-api-access-2tdht\") pod \"volume-data-source-validator-58c9f5f46d-jsswm\" (UID: \"4295fa46-8c63-4cde-8183-1895149bc2ce\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jsswm" Oct 30 02:40:40.298873 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.298854 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnt4b\" (UniqueName: \"kubernetes.io/projected/96981120-5aa0-43e1-9652-8a625d23b6ff-kube-api-access-lnt4b\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6\" (UID: \"96981120-5aa0-43e1-9652-8a625d23b6ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" Oct 30 02:40:40.303511 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.303493 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w87m\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-kube-api-access-9w87m\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:40.305369 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.305353 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpw2q\" (UniqueName: \"kubernetes.io/projected/8ab12b0c-6c4b-4915-9920-9eec9c279f3b-kube-api-access-qpw2q\") pod \"kas-connection-checker-685895c7c9-69v7b\" (UID: \"8ab12b0c-6c4b-4915-9920-9eec9c279f3b\") " pod="kube-system/kas-connection-checker-685895c7c9-69v7b" Oct 30 02:40:40.305418 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.305382 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjg4n\" (UniqueName: \"kubernetes.io/projected/ba886f04-386a-4cb7-aa00-b85f3921e6cc-kube-api-access-xjg4n\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:40.305590 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.305574 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wkjm\" (UniqueName: \"kubernetes.io/projected/a3c34ce6-91f3-42e4-ba49-a905f477571e-kube-api-access-4wkjm\") pod \"node-ca-zg4qh\" (UID: \"a3c34ce6-91f3-42e4-ba49-a905f477571e\") " pod="openshift-image-registry/node-ca-zg4qh" Oct 30 02:40:40.306308 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.306293 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m94g4\" (UniqueName: \"kubernetes.io/projected/114fc00f-05b8-4a65-89e1-5997231c05e7-kube-api-access-m94g4\") pod \"kas-connection-checker-685895c7c9-9s4nm\" (UID: \"114fc00f-05b8-4a65-89e1-5997231c05e7\") " pod="kube-system/kas-connection-checker-685895c7c9-9s4nm" Oct 30 02:40:40.306725 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.306708 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6llj\" (UniqueName: \"kubernetes.io/projected/5e9def4d-df87-404d-90eb-1b21b023fbd8-kube-api-access-m6llj\") pod \"node-resolver-hbr7c\" (UID: \"5e9def4d-df87-404d-90eb-1b21b023fbd8\") " pod="openshift-dns/node-resolver-hbr7c" Oct 30 02:40:40.307100 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.307084 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqr74\" (UniqueName: \"kubernetes.io/projected/5ee44f66-3b88-4914-803d-9dcfbfb0efba-kube-api-access-dqr74\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:40.307177 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.307162 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-bound-sa-token\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:40.307317 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.307301 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvvcv\" (UniqueName: \"kubernetes.io/projected/548e9394-45be-4ffa-937a-89fea1da3093-kube-api-access-lvvcv\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:40.308650 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.308630 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kzk2\" (UniqueName: \"kubernetes.io/projected/e17aabad-c1ca-490d-904d-ddd40b5fd741-kube-api-access-2kzk2\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:40.308888 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.308874 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srs2l\" (UniqueName: \"kubernetes.io/projected/6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e-kube-api-access-srs2l\") pod \"service-ca-operator-544d749c8c-n22dv\" (UID: \"6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" Oct 30 02:40:40.310031 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.310016 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjbgs\" (UniqueName: \"kubernetes.io/projected/183230bf-332c-465f-8525-2094db22e38a-kube-api-access-tjbgs\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.310347 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.310332 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xbl9\" (UniqueName: \"kubernetes.io/projected/da5b2698-93b5-4b4a-8404-f22ec7840967-kube-api-access-9xbl9\") pod \"kas-connection-checker-685895c7c9-6tnpg\" (UID: \"da5b2698-93b5-4b4a-8404-f22ec7840967\") " pod="kube-system/kas-connection-checker-685895c7c9-6tnpg" Oct 30 02:40:40.311740 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.311722 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rckxh\" (UniqueName: \"kubernetes.io/projected/1bac52de-7a27-4fce-828b-9eedcfd60de5-kube-api-access-rckxh\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.312761 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.312737 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djdvk\" (UniqueName: \"kubernetes.io/projected/06222d65-ed37-48ff-9b12-19271e1f9662-kube-api-access-djdvk\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:40.313032 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.313017 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-bound-sa-token\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:40.313479 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.313465 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64r2m\" (UniqueName: \"kubernetes.io/projected/c9becf8f-e251-42bc-a888-18c72145379d-kube-api-access-64r2m\") pod \"migrator-66df5d496b-nsf8s\" (UID: \"c9becf8f-e251-42bc-a888-18c72145379d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nsf8s" Oct 30 02:40:40.315387 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.315372 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bjwl\" (UniqueName: \"kubernetes.io/projected/c2b91cd0-34ba-453f-8088-13e814a3a367-kube-api-access-4bjwl\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.316087 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.316071 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvtmv\" (UniqueName: \"kubernetes.io/projected/ee2a4d7e-545a-4946-a44d-cc895441c477-kube-api-access-qvtmv\") pod \"network-check-target-7rx9c\" (UID: \"ee2a4d7e-545a-4946-a44d-cc895441c477\") " pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:40.318035 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.317995 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5pxf\" (UniqueName: \"kubernetes.io/projected/b0851f53-fa88-40f9-bfdb-beb956a5fb28-kube-api-access-r5pxf\") pod \"insights-operator-7c4986d449-zc4bx\" (UID: \"b0851f53-fa88-40f9-bfdb-beb956a5fb28\") " pod="openshift-insights/insights-operator-7c4986d449-zc4bx" Oct 30 02:40:40.319390 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.319373 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bl6b\" (UniqueName: \"kubernetes.io/projected/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-api-access-9bl6b\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:40.354555 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.354517 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:40.354555 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.354551 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:40.354729 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.354568 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-sysconfig\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.354729 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.354583 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:40.354729 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.354600 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:40.354729 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.354620 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-run-openvswitch\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.354729 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.354644 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-node-log\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.354729 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.354673 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-sysconfig\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.354729 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.354678 4925 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 02:40:40.354729 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.354711 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-run-openvswitch\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.354781 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-var-lib-kubelet\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.354816 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef5e843c-9ceb-44f3-bfc9-c56be367804e-dbus\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.354852 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls podName:bc230021-ead4-497f-9933-45fbdb9efc50 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:40.854830211 +0000 UTC m=+1.868108318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bshlg" (UID: "bc230021-ead4-497f-9933-45fbdb9efc50") : secret "cluster-monitoring-operator-tls" not found Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.354859 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-node-log\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.354871 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-var-lib-kubelet\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.354880 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef5e843c-9ceb-44f3-bfc9-c56be367804e-dbus\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.354892 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.354920 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.354936 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle podName:548e9394-45be-4ffa-937a-89fea1da3093 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:40.85491603 +0000 UTC m=+1.868194125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle") pod "router-default-6bffcf78f7-rplsj" (UID: "548e9394-45be-4ffa-937a-89fea1da3093") : configmap references non-existent config key: service-ca.crt Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.354965 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-systemd\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.354969 4925 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.354985 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-run-systemd\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355004 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-systemd\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.354966 4925 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355008 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3c34ce6-91f3-42e4-ba49-a905f477571e-host\") pod \"node-ca-zg4qh\" (UID: \"a3c34ce6-91f3-42e4-ba49-a905f477571e\") " pod="openshift-image-registry/node-ca-zg4qh" Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355042 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3c34ce6-91f3-42e4-ba49-a905f477571e-host\") pod \"node-ca-zg4qh\" (UID: \"a3c34ce6-91f3-42e4-ba49-a905f477571e\") " pod="openshift-image-registry/node-ca-zg4qh" Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355048 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-var-lib-openvswitch\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.355063 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs podName:e17aabad-c1ca-490d-904d-ddd40b5fd741 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:40.855049772 +0000 UTC m=+1.868327879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs") pod "network-metrics-daemon-9t4cc" (UID: "e17aabad-c1ca-490d-904d-ddd40b5fd741") : secret "metrics-daemon-secret" not found Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355059 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-run-systemd\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355081 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-var-lib-openvswitch\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.355094 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355094 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-etc-kubernetes\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355112 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-etc-kubernetes\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355124 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-kubelet\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355145 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-run-ovn\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355162 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-hostroot\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355178 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-sys-fs\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355203 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-multus-conf-dir\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355209 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-hostroot\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355218 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-lib-modules\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355235 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef5e843c-9ceb-44f3-bfc9-c56be367804e-kubelet-config\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355244 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-run-ovn\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355212 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-kubelet\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355264 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-multus-conf-dir\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355259 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355305 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-cnibin\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355314 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-sys-fs\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355316 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef5e843c-9ceb-44f3-bfc9-c56be367804e-kubelet-config\") pod \"global-pull-secret-syncer-m9zrt\" (UID: \"ef5e843c-9ceb-44f3-bfc9-c56be367804e\") " pod="kube-system/global-pull-secret-syncer-m9zrt" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355333 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-run-ovn-kubernetes\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355352 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355366 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-var-lib-cni-multus\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355371 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-lib-modules\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355373 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-run-ovn-kubernetes\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355385 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-cnibin\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355392 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-systemd-units\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355413 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-systemd-units\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355417 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-run-netns\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355436 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-run-netns\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355444 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-sys\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355478 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-var-lib-kubelet\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355478 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-var-lib-cni-multus\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355507 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-host\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355482 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-sys\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355520 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-var-lib-kubelet\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355536 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-metrics-certs\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355589 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-host\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355608 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-run-multus-certs\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355631 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-modprobe-d\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355653 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-sysctl-d\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355678 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-socket-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355680 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-run-multus-certs\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355714 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-slash\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355735 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-socket-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355742 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355783 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-kubernetes\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355742 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-modprobe-d\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355807 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-etc-selinux\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.355825 4925 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355786 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-slash\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355836 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355842 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-kubernetes\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355830 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-sysctl-d\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355868 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-multus-cni-dir\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.355874 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls podName:ba886f04-386a-4cb7-aa00-b85f3921e6cc nodeName:}" failed. No retries permitted until 2026-10-30 02:40:40.855860685 +0000 UTC m=+1.869138788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls") pod "dns-default-sx7m8" (UID: "ba886f04-386a-4cb7-aa00-b85f3921e6cc") : secret "dns-default-metrics-tls" not found Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355876 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-etc-selinux\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355913 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lttrq\" (UID: \"3fb3b957-3183-4c9e-a45f-c6930341197c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lttrq" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355928 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-multus-cni-dir\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355938 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-etc-openvswitch\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355956 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-system-cni-dir\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355971 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-run-netns\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.355958 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.355993 4925 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356023 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-run-netns\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356028 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-run\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.355992 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-run\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.356031 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert podName:3fb3b957-3183-4c9e-a45f-c6930341197c nodeName:}" failed. No retries permitted until 2026-10-30 02:40:40.856018616 +0000 UTC m=+1.869296722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lttrq" (UID: "3fb3b957-3183-4c9e-a45f-c6930341197c") : secret "networking-console-plugin-cert" not found Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356037 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-system-cni-dir\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356078 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c24d2a3f-bbb2-491e-9f25-608364102b2a-crio-socket\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356107 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356080 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-etc-openvswitch\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356139 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356147 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c24d2a3f-bbb2-491e-9f25-608364102b2a-crio-socket\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356164 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5e9def4d-df87-404d-90eb-1b21b023fbd8-hosts-file\") pod \"node-resolver-hbr7c\" (UID: \"5e9def4d-df87-404d-90eb-1b21b023fbd8\") " pod="openshift-dns/node-resolver-hbr7c" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356194 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356215 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-sysctl-conf\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356216 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5e9def4d-df87-404d-90eb-1b21b023fbd8-hosts-file\") pod \"node-resolver-hbr7c\" (UID: \"5e9def4d-df87-404d-90eb-1b21b023fbd8\") " pod="openshift-dns/node-resolver-hbr7c" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356262 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356289 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-device-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.356303 4925 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356317 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-cnibin\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356340 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2b91cd0-34ba-453f-8088-13e814a3a367-etc-sysctl-conf\") pod \"tuned-76vs8\" (UID: \"c2b91cd0-34ba-453f-8088-13e814a3a367\") " pod="openshift-cluster-node-tuning-operator/tuned-76vs8" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356347 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-system-cni-dir\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.356357 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls podName:5ee44f66-3b88-4914-803d-9dcfbfb0efba nodeName:}" failed. No retries permitted until 2026-10-30 02:40:40.856343455 +0000 UTC m=+1.869621550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n74xk" (UID: "5ee44f66-3b88-4914-803d-9dcfbfb0efba") : secret "samples-operator-tls" not found Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356350 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c24d2a3f-bbb2-491e-9f25-608364102b2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356395 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-system-cni-dir\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356397 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-device-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356423 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-multus-socket-dir-parent\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356392 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc8053d7-4573-4d8a-b8aa-54ad6543d84f-cnibin\") pod \"multus-additional-cni-plugins-j5jm9\" (UID: \"fc8053d7-4573-4d8a-b8aa-54ad6543d84f\") " pod="openshift-multus/multus-additional-cni-plugins-j5jm9" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356462 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-registration-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356489 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356516 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-var-lib-cni-bin\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356527 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06222d65-ed37-48ff-9b12-19271e1f9662-registration-dir\") pod \"aws-ebs-csi-driver-node-s9ldf\" (UID: \"06222d65-ed37-48ff-9b12-19271e1f9662\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9ldf" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356541 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-multus-socket-dir-parent\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356546 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-cni-bin\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356582 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-cni-bin\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356586 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-var-lib-cni-bin\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356588 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-cni-netd\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.356612 4925 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356618 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-run-k8s-cni-cncf-io\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356649 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/183230bf-332c-465f-8525-2094db22e38a-host-run-k8s-cni-cncf-io\") pod \"multus-gk587\" (UID: \"183230bf-332c-465f-8525-2094db22e38a\") " pod="openshift-multus/multus-gk587" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356620 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-host-cni-netd\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.356672 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls podName:c24d2a3f-bbb2-491e-9f25-608364102b2a nodeName:}" failed. No retries permitted until 2026-10-30 02:40:40.856657912 +0000 UTC m=+1.869936011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j2rxp" (UID: "c24d2a3f-bbb2-491e-9f25-608364102b2a") : secret "insights-runtime-extractor-tls" not found Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356649 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/164920df-32a0-48ab-830a-171a05caab08-host-slash\") pod \"iptables-alerter-8gjpf\" (UID: \"164920df-32a0-48ab-830a-171a05caab08\") " pod="openshift-network-operator/iptables-alerter-8gjpf" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356689 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/164920df-32a0-48ab-830a-171a05caab08-host-slash\") pod \"iptables-alerter-8gjpf\" (UID: \"164920df-32a0-48ab-830a-171a05caab08\") " pod="openshift-network-operator/iptables-alerter-8gjpf" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356708 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-log-socket\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.356790 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1bac52de-7a27-4fce-828b-9eedcfd60de5-log-socket\") pod \"ovnkube-node-b995w\" (UID: \"1bac52de-7a27-4fce-828b-9eedcfd60de5\") " pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.357708 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls\") pod \"image-registry-6bc5fd6d9d-w9mrb\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:40.358466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.358076 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548e9394-45be-4ffa-937a-89fea1da3093-metrics-certs\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:40.360715 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.358568 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls\") pod \"image-registry-65c57cd659-szcm2\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:40.435879 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.435845 4925 scope.go:122] "RemoveContainer" containerID="86843fc740326707a729126a99a09bc7b7364f00b2a1bdfe4d7804f799d79f71" Oct 30 02:40:40.438766 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.438734 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:40.448838 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.448822 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:40.482029 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.481843 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hbr7c_5e9def4d-df87-404d-90eb-1b21b023fbd8/dns-node-resolver/0.log" Oct 30 02:40:40.798859 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.798809 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb"] Oct 30 02:40:40.802320 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:40.802291 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11560ce6_6d4a_4c30_83eb_b1b7a3e50cae.slice/crio-8353eae6e45f0eb7533a8e09e5baa4280812964092d91db6d64f61335d7e7115 WatchSource:0}: Error finding container 8353eae6e45f0eb7533a8e09e5baa4280812964092d91db6d64f61335d7e7115: Status 404 returned error can't find the container with id 8353eae6e45f0eb7533a8e09e5baa4280812964092d91db6d64f61335d7e7115 Oct 30 02:40:40.837346 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.837301 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65c57cd659-szcm2"] Oct 30 02:40:40.845837 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:40.845814 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547ef373_3835_4ec2_8f70_94954f34f452.slice/crio-a2d79f50696f747be59fe2864bcb00b5501135a5daf7d631ccdfbc53c806337a WatchSource:0}: Error finding container a2d79f50696f747be59fe2864bcb00b5501135a5daf7d631ccdfbc53c806337a: Status 404 returned error can't find the container with id a2d79f50696f747be59fe2864bcb00b5501135a5daf7d631ccdfbc53c806337a Oct 30 02:40:40.861318 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.861294 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:40.861421 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.861334 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lttrq\" (UID: \"3fb3b957-3183-4c9e-a45f-c6930341197c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lttrq" Oct 30 02:40:40.861421 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.861368 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:40.861421 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.861398 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:40.861580 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.861426 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:40.861580 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.861450 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:40.861580 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.861484 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:40.861580 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.861483 4925 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 02:40:40.861580 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.861557 4925 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 02:40:40.861774 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.861580 4925 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 02:40:40.861774 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.861602 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls podName:ba886f04-386a-4cb7-aa00-b85f3921e6cc nodeName:}" failed. No retries permitted until 2026-10-30 02:40:41.861585074 +0000 UTC m=+2.874863170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls") pod "dns-default-sx7m8" (UID: "ba886f04-386a-4cb7-aa00-b85f3921e6cc") : secret "dns-default-metrics-tls" not found Oct 30 02:40:40.861774 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.861519 4925 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 02:40:40.861774 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.861620 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls podName:bc230021-ead4-497f-9933-45fbdb9efc50 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:41.861611182 +0000 UTC m=+2.874889267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bshlg" (UID: "bc230021-ead4-497f-9933-45fbdb9efc50") : secret "cluster-monitoring-operator-tls" not found Oct 30 02:40:40.861774 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.861565 4925 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 02:40:40.861774 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.861635 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs podName:e17aabad-c1ca-490d-904d-ddd40b5fd741 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:41.861626792 +0000 UTC m=+2.874904876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs") pod "network-metrics-daemon-9t4cc" (UID: "e17aabad-c1ca-490d-904d-ddd40b5fd741") : secret "metrics-daemon-secret" not found Oct 30 02:40:40.861774 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.861649 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle podName:548e9394-45be-4ffa-937a-89fea1da3093 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:41.861641864 +0000 UTC m=+2.874919945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle") pod "router-default-6bffcf78f7-rplsj" (UID: "548e9394-45be-4ffa-937a-89fea1da3093") : configmap references non-existent config key: service-ca.crt Oct 30 02:40:40.861774 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.861662 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls podName:5ee44f66-3b88-4914-803d-9dcfbfb0efba nodeName:}" failed. No retries permitted until 2026-10-30 02:40:41.861655218 +0000 UTC m=+2.874933300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n74xk" (UID: "5ee44f66-3b88-4914-803d-9dcfbfb0efba") : secret "samples-operator-tls" not found Oct 30 02:40:40.861774 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:40.861672 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls podName:c24d2a3f-bbb2-491e-9f25-608364102b2a nodeName:}" failed. No retries permitted until 2026-10-30 02:40:41.861668067 +0000 UTC m=+2.874946147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j2rxp" (UID: "c24d2a3f-bbb2-491e-9f25-608364102b2a") : secret "insights-runtime-extractor-tls" not found Oct 30 02:40:40.863488 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:40.863471 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3fb3b957-3183-4c9e-a45f-c6930341197c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lttrq\" (UID: \"3fb3b957-3183-4c9e-a45f-c6930341197c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lttrq" Oct 30 02:40:41.039948 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.039904 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-lttrq" Oct 30 02:40:41.175247 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.175207 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" event={"ID":"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae","Type":"ContainerStarted","Data":"9a782f2eee34b69429576128a8d9a987254570e7ef4a13e89e4000874f3f1fae"} Oct 30 02:40:41.175679 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.175276 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" event={"ID":"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae","Type":"ContainerStarted","Data":"8353eae6e45f0eb7533a8e09e5baa4280812964092d91db6d64f61335d7e7115"} Oct 30 02:40:41.176966 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.176939 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 02:40:41.177386 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.177363 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/1.log" Oct 30 02:40:41.177497 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.177396 4925 generic.go:356] "Generic (PLEG): container finished" podID="21ee3aae-ce3d-4fc0-9419-60fb4c762ad3" containerID="5626e8723a67325e5de7b43358a3a1364529b3225c245de561a382755c0dc0ce" exitCode=255 Oct 30 02:40:41.177497 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.177474 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" event={"ID":"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3","Type":"ContainerDied","Data":"5626e8723a67325e5de7b43358a3a1364529b3225c245de561a382755c0dc0ce"} Oct 30 02:40:41.177593 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.177535 4925 scope.go:122] "RemoveContainer" containerID="86843fc740326707a729126a99a09bc7b7364f00b2a1bdfe4d7804f799d79f71" Oct 30 02:40:41.178046 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.177852 4925 scope.go:122] "RemoveContainer" containerID="5626e8723a67325e5de7b43358a3a1364529b3225c245de561a382755c0dc0ce" Oct 30 02:40:41.178139 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:41.178055 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-vb5mb_openshift-console-operator(21ee3aae-ce3d-4fc0-9419-60fb4c762ad3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" podUID="21ee3aae-ce3d-4fc0-9419-60fb4c762ad3" Oct 30 02:40:41.179490 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.179459 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65c57cd659-szcm2" event={"ID":"547ef373-3835-4ec2-8f70-94954f34f452","Type":"ContainerStarted","Data":"f47d89273550df0a368ecb58f11bd9c421f36f43ab43a5ced156ab203f1f44f5"} Oct 30 02:40:41.179573 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.179502 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65c57cd659-szcm2" event={"ID":"547ef373-3835-4ec2-8f70-94954f34f452","Type":"ContainerStarted","Data":"a2d79f50696f747be59fe2864bcb00b5501135a5daf7d631ccdfbc53c806337a"} Oct 30 02:40:41.353501 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.353413 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-lttrq"] Oct 30 02:40:41.356178 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:41.356145 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb3b957_3183_4c9e_a45f_c6930341197c.slice/crio-0a080a4ea53520dcdcfbab8d279019bbfd4dd94c984c986bfd8cbadd44d147d4 WatchSource:0}: Error finding container 0a080a4ea53520dcdcfbab8d279019bbfd4dd94c984c986bfd8cbadd44d147d4: Status 404 returned error can't find the container with id 0a080a4ea53520dcdcfbab8d279019bbfd4dd94c984c986bfd8cbadd44d147d4 Oct 30 02:40:41.358312 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.358294 4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:40:41.474902 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.474875 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zg4qh_a3c34ce6-91f3-42e4-ba49-a905f477571e/node-ca/0.log" Oct 30 02:40:41.789854 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.789739 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n96rg_8f17a0e8-ce89-4a88-98f2-c7782260534d/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:41.872892 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.872859 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:41.873075 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.872912 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:41.873075 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.872954 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:41.873075 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.872992 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:41.873075 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:41.873016 4925 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 02:40:41.873075 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.873021 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:41.873305 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:41.873089 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls podName:ba886f04-386a-4cb7-aa00-b85f3921e6cc nodeName:}" failed. No retries permitted until 2026-10-30 02:40:43.873070836 +0000 UTC m=+4.886348932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls") pod "dns-default-sx7m8" (UID: "ba886f04-386a-4cb7-aa00-b85f3921e6cc") : secret "dns-default-metrics-tls" not found Oct 30 02:40:41.873305 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:41.873104 4925 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 02:40:41.873305 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:41.873117 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle podName:548e9394-45be-4ffa-937a-89fea1da3093 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:43.873108204 +0000 UTC m=+4.886386299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle") pod "router-default-6bffcf78f7-rplsj" (UID: "548e9394-45be-4ffa-937a-89fea1da3093") : configmap references non-existent config key: service-ca.crt Oct 30 02:40:41.873305 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:41.873135 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:41.873305 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:41.873159 4925 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 02:40:41.873305 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:41.873169 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls podName:c24d2a3f-bbb2-491e-9f25-608364102b2a nodeName:}" failed. No retries permitted until 2026-10-30 02:40:43.873151366 +0000 UTC m=+4.886429465 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j2rxp" (UID: "c24d2a3f-bbb2-491e-9f25-608364102b2a") : secret "insights-runtime-extractor-tls" not found Oct 30 02:40:41.873305 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:41.873104 4925 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 02:40:41.873305 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:41.873199 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls podName:bc230021-ead4-497f-9933-45fbdb9efc50 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:43.87318808 +0000 UTC m=+4.886466174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bshlg" (UID: "bc230021-ead4-497f-9933-45fbdb9efc50") : secret "cluster-monitoring-operator-tls" not found Oct 30 02:40:41.873305 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:41.873199 4925 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 02:40:41.873305 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:41.873219 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls podName:5ee44f66-3b88-4914-803d-9dcfbfb0efba nodeName:}" failed. No retries permitted until 2026-10-30 02:40:43.873209155 +0000 UTC m=+4.886487252 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-n74xk" (UID: "5ee44f66-3b88-4914-803d-9dcfbfb0efba") : secret "samples-operator-tls" not found Oct 30 02:40:41.873305 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:41.873237 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs podName:e17aabad-c1ca-490d-904d-ddd40b5fd741 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:43.873228425 +0000 UTC m=+4.886506541 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs") pod "network-metrics-daemon-9t4cc" (UID: "e17aabad-c1ca-490d-904d-ddd40b5fd741") : secret "metrics-daemon-secret" not found Oct 30 02:40:42.077136 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:42.077093 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r779s_02aed17c-f8e1-4fd0-bb90-4e972bd1a827/serve-healthcheck-canary/0.log" Oct 30 02:40:42.131264 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:42.131187 4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n96rg_8f17a0e8-ce89-4a88-98f2-c7782260534d/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:42.185050 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:42.185014 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-lttrq" event={"ID":"3fb3b957-3183-4c9e-a45f-c6930341197c","Type":"ContainerStarted","Data":"0a080a4ea53520dcdcfbab8d279019bbfd4dd94c984c986bfd8cbadd44d147d4"} Oct 30 02:40:42.186884 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:42.186860 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 02:40:42.187028 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:42.187013 4925 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 02:40:42.187098 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:42.187044 4925 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 02:40:42.214018 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:42.213989 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:42.273936 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:42.273901 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-nsf8s_c9becf8f-e251-42bc-a888-18c72145379d/migrator/0.log" Oct 30 02:40:42.675219 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:42.675186 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6_96981120-5aa0-43e1-9652-8a625d23b6ff/kube-storage-version-migrator-operator/0.log" Oct 30 02:40:43.192150 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:43.192117 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-lttrq" event={"ID":"3fb3b957-3183-4c9e-a45f-c6930341197c","Type":"ContainerStarted","Data":"fe8cd8939525154cf833a1cf0c4d5304ac9fcff8059fdd20e65056d88d1bea1c"} Oct 30 02:40:43.193573 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:43.193550 4925 generic.go:356] "Generic (PLEG): container finished" podID="96981120-5aa0-43e1-9652-8a625d23b6ff" containerID="3fd8a99f7e30739cb385ddfe3e6df1cc9e362163131c665d71f0f73d870ce4fd" exitCode=0 Oct 30 02:40:43.193670 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:43.193607 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" event={"ID":"96981120-5aa0-43e1-9652-8a625d23b6ff","Type":"ContainerDied","Data":"3fd8a99f7e30739cb385ddfe3e6df1cc9e362163131c665d71f0f73d870ce4fd"} Oct 30 02:40:43.193896 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:43.193883 4925 scope.go:122] "RemoveContainer" containerID="3fd8a99f7e30739cb385ddfe3e6df1cc9e362163131c665d71f0f73d870ce4fd" Oct 30 02:40:43.591393 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:43.591359 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:43.591566 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:43.591556 4925 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 02:40:43.892016 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:43.891924 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:43.892016 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:43.891993 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:43.892202 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:43.892024 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:43.892202 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:43.892053 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:43.892202 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:43.892081 4925 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 02:40:43.892202 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:43.892119 4925 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 02:40:43.892202 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:43.892086 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:43.892202 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:43.892156 4925 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 02:40:43.892202 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:43.892160 4925 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 02:40:43.892202 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:43.892158 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs podName:e17aabad-c1ca-490d-904d-ddd40b5fd741 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:47.892137316 +0000 UTC m=+8.905415413 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs") pod "network-metrics-daemon-9t4cc" (UID: "e17aabad-c1ca-490d-904d-ddd40b5fd741") : secret "metrics-daemon-secret" not found Oct 30 02:40:43.892474 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:43.892219 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls podName:ba886f04-386a-4cb7-aa00-b85f3921e6cc nodeName:}" failed. No retries permitted until 2026-10-30 02:40:47.892205612 +0000 UTC m=+8.905483693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls") pod "dns-default-sx7m8" (UID: "ba886f04-386a-4cb7-aa00-b85f3921e6cc") : secret "dns-default-metrics-tls" not found Oct 30 02:40:43.892474 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:43.892233 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls podName:bc230021-ead4-497f-9933-45fbdb9efc50 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:47.892227024 +0000 UTC m=+8.905505104 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bshlg" (UID: "bc230021-ead4-497f-9933-45fbdb9efc50") : secret "cluster-monitoring-operator-tls" not found Oct 30 02:40:43.892474 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:43.892256 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls podName:c24d2a3f-bbb2-491e-9f25-608364102b2a nodeName:}" failed. No retries permitted until 2026-10-30 02:40:47.89225107 +0000 UTC m=+8.905529151 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j2rxp" (UID: "c24d2a3f-bbb2-491e-9f25-608364102b2a") : secret "insights-runtime-extractor-tls" not found Oct 30 02:40:43.892474 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:43.892270 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:43.892848 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:43.892832 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/548e9394-45be-4ffa-937a-89fea1da3093-service-ca-bundle\") pod \"router-default-6bffcf78f7-rplsj\" (UID: \"548e9394-45be-4ffa-937a-89fea1da3093\") " pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:43.894500 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:43.894471 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ee44f66-3b88-4914-803d-9dcfbfb0efba-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-n74xk\" (UID: \"5ee44f66-3b88-4914-803d-9dcfbfb0efba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:44.036029 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:44.035988 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" Oct 30 02:40:44.038163 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:44.038138 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:44.199271 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:44.199231 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-nb4r6" event={"ID":"96981120-5aa0-43e1-9652-8a625d23b6ff","Type":"ContainerStarted","Data":"25ae83c0411d77e1abfa3ad34789cf825586d590279c10a6231f914d1eefe6ab"} Oct 30 02:40:44.295279 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:44.295247 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:40:44.350718 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:44.350686 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk"] Oct 30 02:40:44.393873 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:44.393817 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6bffcf78f7-rplsj"] Oct 30 02:40:44.396653 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:44.396630 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548e9394_45be_4ffa_937a_89fea1da3093.slice/crio-3c107147124027b790e59cd5cdc409a20043eeb1fb97342db05c830e2e9264aa WatchSource:0}: Error finding container 3c107147124027b790e59cd5cdc409a20043eeb1fb97342db05c830e2e9264aa: Status 404 returned error can't find the container with id 3c107147124027b790e59cd5cdc409a20043eeb1fb97342db05c830e2e9264aa Oct 30 02:40:45.204780 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:45.204735 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" event={"ID":"5ee44f66-3b88-4914-803d-9dcfbfb0efba","Type":"ContainerStarted","Data":"ab61a0ec5d31b5693f669b8117de871e25288f421762f6a11e47af48b52e7883"} Oct 30 02:40:45.206716 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:45.206690 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6bffcf78f7-rplsj" event={"ID":"548e9394-45be-4ffa-937a-89fea1da3093","Type":"ContainerStarted","Data":"aaeb66156b39c7df8fb80c2fd4cbac6a85608add8fc33a56751d7e6b0424ca0c"} Oct 30 02:40:45.206858 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:45.206738 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6bffcf78f7-rplsj" event={"ID":"548e9394-45be-4ffa-937a-89fea1da3093","Type":"ContainerStarted","Data":"3c107147124027b790e59cd5cdc409a20043eeb1fb97342db05c830e2e9264aa"} Oct 30 02:40:45.860025 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:45.859989 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:45.860348 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:45.860335 4925 scope.go:122] "RemoveContainer" containerID="5626e8723a67325e5de7b43358a3a1364529b3225c245de561a382755c0dc0ce" Oct 30 02:40:45.860530 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:45.860513 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-vb5mb_openshift-console-operator(21ee3aae-ce3d-4fc0-9419-60fb4c762ad3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" podUID="21ee3aae-ce3d-4fc0-9419-60fb4c762ad3" Oct 30 02:40:46.038362 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:46.038340 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:46.041115 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:46.041088 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:46.212067 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:46.211973 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" event={"ID":"5ee44f66-3b88-4914-803d-9dcfbfb0efba","Type":"ContainerStarted","Data":"cc92d921f167bd4e1484e9d17dc86474bb2fd1a699b21185b8f8228f560c12e6"} Oct 30 02:40:46.212067 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:46.212028 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-n74xk" event={"ID":"5ee44f66-3b88-4914-803d-9dcfbfb0efba","Type":"ContainerStarted","Data":"662e7b3c53bfe31db9141ea41d918866d2be059a40697ba27d0048de7fb30687"} Oct 30 02:40:46.491565 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:46.491486 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b995w" Oct 30 02:40:47.215199 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:47.215170 4925 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 02:40:47.237034 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:47.237005 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:47.238392 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:47.238372 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6bffcf78f7-rplsj" Oct 30 02:40:47.930727 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:47.930682 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:47.930929 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:47.930744 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:47.930929 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:47.930789 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:47.930929 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:47.930813 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:47.930929 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:47.930900 4925 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 02:40:47.931146 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:47.930968 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls podName:c24d2a3f-bbb2-491e-9f25-608364102b2a nodeName:}" failed. No retries permitted until 2026-10-30 02:40:55.930952667 +0000 UTC m=+16.944230749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j2rxp" (UID: "c24d2a3f-bbb2-491e-9f25-608364102b2a") : secret "insights-runtime-extractor-tls" not found Oct 30 02:40:47.931146 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:47.930902 4925 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 02:40:47.931146 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:47.931052 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs podName:e17aabad-c1ca-490d-904d-ddd40b5fd741 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:55.931040351 +0000 UTC m=+16.944318435 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs") pod "network-metrics-daemon-9t4cc" (UID: "e17aabad-c1ca-490d-904d-ddd40b5fd741") : secret "metrics-daemon-secret" not found Oct 30 02:40:47.931146 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:47.930902 4925 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 02:40:47.931146 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:47.931099 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls podName:bc230021-ead4-497f-9933-45fbdb9efc50 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:55.931089281 +0000 UTC m=+16.944367376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bshlg" (UID: "bc230021-ead4-497f-9933-45fbdb9efc50") : secret "cluster-monitoring-operator-tls" not found Oct 30 02:40:47.933159 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:47.933139 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba886f04-386a-4cb7-aa00-b85f3921e6cc-metrics-tls\") pod \"dns-default-sx7m8\" (UID: \"ba886f04-386a-4cb7-aa00-b85f3921e6cc\") " pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:47.950485 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:47.950461 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:48.251485 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:48.251460 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sx7m8"] Oct 30 02:40:48.253537 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:48.253507 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba886f04_386a_4cb7_aa00_b85f3921e6cc.slice/crio-23e8bcea4dbff225e1fc701b79d77647a3e4a4160b39fb1f089e2905ff89d0c3 WatchSource:0}: Error finding container 23e8bcea4dbff225e1fc701b79d77647a3e4a4160b39fb1f089e2905ff89d0c3: Status 404 returned error can't find the container with id 23e8bcea4dbff225e1fc701b79d77647a3e4a4160b39fb1f089e2905ff89d0c3 Oct 30 02:40:48.565604 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:48.565529 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:40:49.222253 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:49.222217 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sx7m8" event={"ID":"ba886f04-386a-4cb7-aa00-b85f3921e6cc","Type":"ContainerStarted","Data":"23e8bcea4dbff225e1fc701b79d77647a3e4a4160b39fb1f089e2905ff89d0c3"} Oct 30 02:40:50.036886 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:50.036807 4925 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:50.037247 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:50.037189 4925 scope.go:122] "RemoveContainer" containerID="5626e8723a67325e5de7b43358a3a1364529b3225c245de561a382755c0dc0ce" Oct 30 02:40:50.037394 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:40:50.037371 4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-vb5mb_openshift-console-operator(21ee3aae-ce3d-4fc0-9419-60fb4c762ad3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" podUID="21ee3aae-ce3d-4fc0-9419-60fb4c762ad3" Oct 30 02:40:50.100725 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:50.100691 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:40:50.227174 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:50.227137 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sx7m8" event={"ID":"ba886f04-386a-4cb7-aa00-b85f3921e6cc","Type":"ContainerStarted","Data":"e17497efbda0e30eeaa793815161fdd7bb4e59044d1e97596513b59919ac5669"} Oct 30 02:40:50.227360 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:50.227194 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sx7m8" event={"ID":"ba886f04-386a-4cb7-aa00-b85f3921e6cc","Type":"ContainerStarted","Data":"1d63038b2280fd9b416d2547c85ff05bdb5e576196dc3d4fcda96423e268f777"} Oct 30 02:40:50.227360 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:50.227283 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sx7m8" Oct 30 02:40:51.114012 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:51.113980 4925 scope.go:122] "RemoveContainer" containerID="5626e8723a67325e5de7b43358a3a1364529b3225c245de561a382755c0dc0ce" Oct 30 02:40:51.233008 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:51.232978 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 02:40:51.233181 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:51.233100 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" event={"ID":"21ee3aae-ce3d-4fc0-9419-60fb4c762ad3","Type":"ContainerStarted","Data":"6fa7d6ea7e91081d760d56aee3942836d05273e393c351108614704ece9077a3"} Oct 30 02:40:51.233419 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:51.233403 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:51.234386 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:51.234363 4925 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-vb5mb container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body= Oct 30 02:40:51.234472 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:51.234417 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" podUID="21ee3aae-ce3d-4fc0-9419-60fb4c762ad3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" Oct 30 02:40:52.241173 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:52.241148 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-vb5mb" Oct 30 02:40:55.997488 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:55.997432 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:55.998090 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:55.997500 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:55.998090 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:55.997539 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:55.999936 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:55.999913 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17aabad-c1ca-490d-904d-ddd40b5fd741-metrics-certs\") pod \"network-metrics-daemon-9t4cc\" (UID: \"e17aabad-c1ca-490d-904d-ddd40b5fd741\") " pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:56.000047 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:56.000013 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c24d2a3f-bbb2-491e-9f25-608364102b2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2rxp\" (UID: \"c24d2a3f-bbb2-491e-9f25-608364102b2a\") " pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:56.000111 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:56.000101 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc230021-ead4-497f-9933-45fbdb9efc50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bshlg\" (UID: \"bc230021-ead4-497f-9933-45fbdb9efc50\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:56.035961 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:56.035927 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" Oct 30 02:40:56.040132 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:56.040113 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t4cc" Oct 30 02:40:56.051489 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:56.051466 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j2rxp" Oct 30 02:40:56.355090 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:56.355059 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9t4cc"] Oct 30 02:40:56.357768 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:56.357720 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17aabad_c1ca_490d_904d_ddd40b5fd741.slice/crio-e5f165b4d03cc0f5c7527bffc42f53e8ef9a54cbc32f1c715f03b342fc08d88c WatchSource:0}: Error finding container e5f165b4d03cc0f5c7527bffc42f53e8ef9a54cbc32f1c715f03b342fc08d88c: Status 404 returned error can't find the container with id e5f165b4d03cc0f5c7527bffc42f53e8ef9a54cbc32f1c715f03b342fc08d88c Oct 30 02:40:56.368701 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:56.368675 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg"] Oct 30 02:40:56.369252 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:56.369234 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j2rxp"] Oct 30 02:40:56.370839 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:56.370819 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc230021_ead4_497f_9933_45fbdb9efc50.slice/crio-25b4a08393046454e55bde8827531a3978128e700a55656fbf670ddd71c12309 WatchSource:0}: Error finding container 25b4a08393046454e55bde8827531a3978128e700a55656fbf670ddd71c12309: Status 404 returned error can't find the container with id 25b4a08393046454e55bde8827531a3978128e700a55656fbf670ddd71c12309 Oct 30 02:40:56.371513 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:40:56.371490 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24d2a3f_bbb2_491e_9f25_608364102b2a.slice/crio-b6b0ae4e91f8fdd5be6c0d601304129b8363574cd0f899e996e1a059e13707b4 WatchSource:0}: Error finding container b6b0ae4e91f8fdd5be6c0d601304129b8363574cd0f899e996e1a059e13707b4: Status 404 returned error can't find the container with id b6b0ae4e91f8fdd5be6c0d601304129b8363574cd0f899e996e1a059e13707b4 Oct 30 02:40:56.400044 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:56.400015 4925 patch_prober.go:29] interesting pod/image-registry-65c57cd659-szcm2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 02:40:56.400153 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:56.400060 4925 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-65c57cd659-szcm2" podUID="547ef373-3835-4ec2-8f70-94954f34f452" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:40:57.261922 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:57.261873 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j2rxp" event={"ID":"c24d2a3f-bbb2-491e-9f25-608364102b2a","Type":"ContainerStarted","Data":"1d90a5e66e3fb842d10bc983afaa63049eceb282b7efcc783e4a05b4a1e8c0d0"} Oct 30 02:40:57.262346 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:57.261952 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j2rxp" event={"ID":"c24d2a3f-bbb2-491e-9f25-608364102b2a","Type":"ContainerStarted","Data":"4a19d59df359cc803248dddb53010d598c0ca7beda322060eddfe8aaf0871836"} Oct 30 02:40:57.262346 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:57.261985 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j2rxp" event={"ID":"c24d2a3f-bbb2-491e-9f25-608364102b2a","Type":"ContainerStarted","Data":"b6b0ae4e91f8fdd5be6c0d601304129b8363574cd0f899e996e1a059e13707b4"} Oct 30 02:40:57.263698 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:57.263668 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9t4cc" event={"ID":"e17aabad-c1ca-490d-904d-ddd40b5fd741","Type":"ContainerStarted","Data":"e5f165b4d03cc0f5c7527bffc42f53e8ef9a54cbc32f1c715f03b342fc08d88c"} Oct 30 02:40:57.265377 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:57.265348 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" event={"ID":"bc230021-ead4-497f-9933-45fbdb9efc50","Type":"ContainerStarted","Data":"25b4a08393046454e55bde8827531a3978128e700a55656fbf670ddd71c12309"} Oct 30 02:40:57.626135 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:57.625780 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb"] Oct 30 02:40:57.631992 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:57.631960 4925 patch_prober.go:29] interesting pod/image-registry-6bc5fd6d9d-w9mrb container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 02:40:57.632150 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:57.632011 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" podUID="11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:40:58.272043 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:58.272004 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bshlg" event={"ID":"bc230021-ead4-497f-9933-45fbdb9efc50","Type":"ContainerStarted","Data":"7e6d90dd2a2446a27a2e02c7810807cc163939d8dd12529b303ca76c18629e4f"} Oct 30 02:40:58.274098 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:58.274067 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9t4cc" event={"ID":"e17aabad-c1ca-490d-904d-ddd40b5fd741","Type":"ContainerStarted","Data":"26af74fa4a306c7a0cb2aa807dae72adb0a3030d9e04a3da88ed7b42402724d8"} Oct 30 02:40:58.274215 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:58.274119 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9t4cc" event={"ID":"e17aabad-c1ca-490d-904d-ddd40b5fd741","Type":"ContainerStarted","Data":"2e94725baa6816fe475b0110417a069f276566075ef5caa064e3c46c99ff0495"} Oct 30 02:40:58.570327 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:58.570291 4925 patch_prober.go:29] interesting pod/image-registry-65c57cd659-szcm2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 02:40:58.570513 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:58.570352 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-65c57cd659-szcm2" podUID="547ef373-3835-4ec2-8f70-94954f34f452" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:40:59.280114 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:40:59.280067 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j2rxp" event={"ID":"c24d2a3f-bbb2-491e-9f25-608364102b2a","Type":"ContainerStarted","Data":"0678d6bf51582dafa5882ac836f154a0c777ef169636d3d45a9e354bc7597020"} Oct 30 02:41:00.235848 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:00.235818 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sx7m8" Oct 30 02:41:00.809982 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:00.809954 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n96rg" Oct 30 02:41:07.164320 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.164291 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cc8q2"] Oct 30 02:41:07.199293 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.199264 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.216802 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.216775 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 02:41:07.216934 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.216879 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 02:41:07.216934 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.216886 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mmm8g\"" Oct 30 02:41:07.230855 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.230831 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 02:41:07.230855 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.230839 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 02:41:07.285390 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.285360 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrvtd\" (UniqueName: \"kubernetes.io/projected/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-kube-api-access-qrvtd\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.285571 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.285409 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-node-exporter-wtmp\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.285571 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.285498 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-sys\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.285571 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.285525 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.285721 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.285572 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-node-exporter-tls\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.285721 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.285595 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-node-exporter-textfile\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.285802 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.285767 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-node-exporter-accelerators-collector-config\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.285841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.285805 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-root\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.285841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.285823 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-metrics-client-ca\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.386300 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.386268 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-node-exporter-tls\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.386300 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.386304 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-node-exporter-textfile\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.386564 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.386339 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-node-exporter-accelerators-collector-config\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.386564 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.386364 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-root\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.386564 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.386388 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-metrics-client-ca\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.386564 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.386419 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrvtd\" (UniqueName: \"kubernetes.io/projected/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-kube-api-access-qrvtd\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.386564 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.386459 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-node-exporter-wtmp\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.386564 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.386486 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-sys\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.386564 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.386508 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.386964 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.386888 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-sys\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.386964 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.386914 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-node-exporter-wtmp\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.386964 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.386922 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-node-exporter-textfile\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.386964 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.386950 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-root\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.387560 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.387414 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-metrics-client-ca\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.387865 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.387782 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-node-exporter-accelerators-collector-config\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.389369 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.389349 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-node-exporter-tls\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.389517 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.389496 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.412020 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.411984 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrvtd\" (UniqueName: \"kubernetes.io/projected/6743f565-3d91-49c1-9c4a-9bc6ab5b434b-kube-api-access-qrvtd\") pod \"node-exporter-cc8q2\" (UID: \"6743f565-3d91-49c1-9c4a-9bc6ab5b434b\") " pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.548990 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.548904 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cc8q2" Oct 30 02:41:07.589117 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:41:07.589088 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6743f565_3d91_49c1_9c4a_9bc6ab5b434b.slice/crio-e53cf07407a2d2e950a409f03043472072f8cb16644b5d33c6e247f19f866155 WatchSource:0}: Error finding container e53cf07407a2d2e950a409f03043472072f8cb16644b5d33c6e247f19f866155: Status 404 returned error can't find the container with id e53cf07407a2d2e950a409f03043472072f8cb16644b5d33c6e247f19f866155 Oct 30 02:41:07.630436 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:07.630409 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:41:08.316002 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:08.315969 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cc8q2" event={"ID":"6743f565-3d91-49c1-9c4a-9bc6ab5b434b","Type":"ContainerStarted","Data":"e53cf07407a2d2e950a409f03043472072f8cb16644b5d33c6e247f19f866155"} Oct 30 02:41:08.570560 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:08.570490 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:41:09.321769 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:09.321662 4925 generic.go:356] "Generic (PLEG): container finished" podID="6743f565-3d91-49c1-9c4a-9bc6ab5b434b" containerID="18917319e0c374b0b783736fdab565f22743249d2865038cef76e4a229aad1d7" exitCode=0 Oct 30 02:41:09.321769 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:09.321734 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cc8q2" event={"ID":"6743f565-3d91-49c1-9c4a-9bc6ab5b434b","Type":"ContainerDied","Data":"18917319e0c374b0b783736fdab565f22743249d2865038cef76e4a229aad1d7"} Oct 30 02:41:10.328279 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:10.328238 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cc8q2" event={"ID":"6743f565-3d91-49c1-9c4a-9bc6ab5b434b","Type":"ContainerStarted","Data":"869e0cb0fecdf0b6db0449107cb0138b96b32f68bffd54b5441451e098912d17"} Oct 30 02:41:10.328667 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:10.328329 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cc8q2" event={"ID":"6743f565-3d91-49c1-9c4a-9bc6ab5b434b","Type":"ContainerStarted","Data":"96d98a2e008884393cf0a42258c137ef9b05a9bccfb2fbd83d70e2a08dd60e12"} Oct 30 02:41:10.377003 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:10.376935 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cc8q2" podStartSLOduration=2.033651802 podStartE2EDuration="3.376915843s" totalImagesPullingTime="1.271560158s" totalInitContainerRuntime="71.703883ms" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:10.375387078 +0000 UTC m=+31.388665182" watchObservedRunningTime="2026-10-30 02:41:10.376915843 +0000 UTC m=+31.390193947" Oct 30 02:41:14.298801 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:14.298768 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7rx9c" Oct 30 02:41:18.161364 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:41:18.161325 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:41:21.137461 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:21.137371 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65c57cd659-szcm2"] Oct 30 02:41:22.652093 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:22.652050 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" podUID="11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" containerName="registry" containerID="cri-o://9a782f2eee34b69429576128a8d9a987254570e7ef4a13e89e4000874f3f1fae" gracePeriod=30 Oct 30 02:41:22.889850 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:22.889825 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:41:23.021317 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.021231 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-trusted-ca\") pod \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " Oct 30 02:41:23.021317 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.021279 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-bound-sa-token\") pod \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " Oct 30 02:41:23.021317 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.021306 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-installation-pull-secrets\") pod \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " Oct 30 02:41:23.021594 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.021325 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-certificates\") pod \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " Oct 30 02:41:23.021594 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.021360 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls\") pod \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " Oct 30 02:41:23.021594 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.021393 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-ca-trust-extracted\") pod \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " Oct 30 02:41:23.021594 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.021445 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-image-registry-private-configuration\") pod \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " Oct 30 02:41:23.021594 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.021483 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-kube-api-access-66622\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-kube-api-access-66622\") pod \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\" (UID: \"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae\") " Oct 30 02:41:23.021892 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.021695 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-trusted-ca" pod "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" (UID: "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:23.022173 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.022148 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-certificates" pod "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" (UID: "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:23.024346 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.024318 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-installation-pull-secrets" pod "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" (UID: "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:23.024426 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.024396 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-kube-api-access-66622" pod "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" (UID: "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae"). InnerVolumeSpecName "kube-api-access-66622". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:23.024466 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.024439 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls" pod "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" (UID: "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:23.024563 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.024539 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-image-registry-private-configuration" pod "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" (UID: "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:23.024658 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.024572 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-bound-sa-token" pod "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" (UID: "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:23.028603 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.028563 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-ca-trust-extracted" pod "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" (UID: "11560ce6-6d4a-4c30-83eb-b1b7a3e50cae"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:23.122235 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.122206 4925 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-ca-trust-extracted\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:41:23.122235 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.122232 4925 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-image-registry-private-configuration\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:41:23.122425 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.122243 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66622\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-kube-api-access-66622\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:41:23.122425 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.122253 4925 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-trusted-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:41:23.122425 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.122262 4925 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-bound-sa-token\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:41:23.122425 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.122271 4925 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-installation-pull-secrets\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:41:23.122425 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.122279 4925 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-certificates\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:41:23.122425 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.122287 4925 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae-registry-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:41:23.379035 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.379001 4925 generic.go:356] "Generic (PLEG): container finished" podID="11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" containerID="9a782f2eee34b69429576128a8d9a987254570e7ef4a13e89e4000874f3f1fae" exitCode=0 Oct 30 02:41:23.379223 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.379064 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" Oct 30 02:41:23.379223 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.379090 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" event={"ID":"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae","Type":"ContainerDied","Data":"9a782f2eee34b69429576128a8d9a987254570e7ef4a13e89e4000874f3f1fae"} Oct 30 02:41:23.379223 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.379154 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb" event={"ID":"11560ce6-6d4a-4c30-83eb-b1b7a3e50cae","Type":"ContainerDied","Data":"8353eae6e45f0eb7533a8e09e5baa4280812964092d91db6d64f61335d7e7115"} Oct 30 02:41:23.379223 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.379180 4925 scope.go:122] "RemoveContainer" containerID="9a782f2eee34b69429576128a8d9a987254570e7ef4a13e89e4000874f3f1fae" Oct 30 02:41:23.389039 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.389021 4925 scope.go:122] "RemoveContainer" containerID="9a782f2eee34b69429576128a8d9a987254570e7ef4a13e89e4000874f3f1fae" Oct 30 02:41:23.389308 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:41:23.389288 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a782f2eee34b69429576128a8d9a987254570e7ef4a13e89e4000874f3f1fae\": container with ID starting with 9a782f2eee34b69429576128a8d9a987254570e7ef4a13e89e4000874f3f1fae not found: ID does not exist" containerID="9a782f2eee34b69429576128a8d9a987254570e7ef4a13e89e4000874f3f1fae" Oct 30 02:41:23.389384 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.389315 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a782f2eee34b69429576128a8d9a987254570e7ef4a13e89e4000874f3f1fae"} err="failed to get container status \"9a782f2eee34b69429576128a8d9a987254570e7ef4a13e89e4000874f3f1fae\": rpc error: code = NotFound desc = could not find container \"9a782f2eee34b69429576128a8d9a987254570e7ef4a13e89e4000874f3f1fae\": container with ID starting with 9a782f2eee34b69429576128a8d9a987254570e7ef4a13e89e4000874f3f1fae not found: ID does not exist" Oct 30 02:41:23.447220 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.447190 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb"] Oct 30 02:41:23.461144 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:23.461115 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6bc5fd6d9d-w9mrb"] Oct 30 02:41:25.119713 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:25.119678 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" path="/var/lib/kubelet/pods/11560ce6-6d4a-4c30-83eb-b1b7a3e50cae/volumes" Oct 30 02:41:46.157009 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.156948 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-65c57cd659-szcm2" podUID="547ef373-3835-4ec2-8f70-94954f34f452" containerName="registry" containerID="cri-o://f47d89273550df0a368ecb58f11bd9c421f36f43ab43a5ced156ab203f1f44f5" gracePeriod=30 Oct 30 02:41:46.407663 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.407606 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:41:46.470023 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.469990 4925 generic.go:356] "Generic (PLEG): container finished" podID="547ef373-3835-4ec2-8f70-94954f34f452" containerID="f47d89273550df0a368ecb58f11bd9c421f36f43ab43a5ced156ab203f1f44f5" exitCode=0 Oct 30 02:41:46.470184 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.470041 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65c57cd659-szcm2" Oct 30 02:41:46.470184 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.470047 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65c57cd659-szcm2" event={"ID":"547ef373-3835-4ec2-8f70-94954f34f452","Type":"ContainerDied","Data":"f47d89273550df0a368ecb58f11bd9c421f36f43ab43a5ced156ab203f1f44f5"} Oct 30 02:41:46.470184 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.470089 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65c57cd659-szcm2" event={"ID":"547ef373-3835-4ec2-8f70-94954f34f452","Type":"ContainerDied","Data":"a2d79f50696f747be59fe2864bcb00b5501135a5daf7d631ccdfbc53c806337a"} Oct 30 02:41:46.470184 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.470107 4925 scope.go:122] "RemoveContainer" containerID="f47d89273550df0a368ecb58f11bd9c421f36f43ab43a5ced156ab203f1f44f5" Oct 30 02:41:46.478126 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.478108 4925 scope.go:122] "RemoveContainer" containerID="f47d89273550df0a368ecb58f11bd9c421f36f43ab43a5ced156ab203f1f44f5" Oct 30 02:41:46.478378 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:41:46.478354 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f47d89273550df0a368ecb58f11bd9c421f36f43ab43a5ced156ab203f1f44f5\": container with ID starting with f47d89273550df0a368ecb58f11bd9c421f36f43ab43a5ced156ab203f1f44f5 not found: ID does not exist" containerID="f47d89273550df0a368ecb58f11bd9c421f36f43ab43a5ced156ab203f1f44f5" Oct 30 02:41:46.478442 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.478391 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f47d89273550df0a368ecb58f11bd9c421f36f43ab43a5ced156ab203f1f44f5"} err="failed to get container status \"f47d89273550df0a368ecb58f11bd9c421f36f43ab43a5ced156ab203f1f44f5\": rpc error: code = NotFound desc = could not find container \"f47d89273550df0a368ecb58f11bd9c421f36f43ab43a5ced156ab203f1f44f5\": container with ID starting with f47d89273550df0a368ecb58f11bd9c421f36f43ab43a5ced156ab203f1f44f5 not found: ID does not exist" Oct 30 02:41:46.517659 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.517629 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/547ef373-3835-4ec2-8f70-94954f34f452-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/547ef373-3835-4ec2-8f70-94954f34f452-installation-pull-secrets\") pod \"547ef373-3835-4ec2-8f70-94954f34f452\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " Oct 30 02:41:46.517830 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.517666 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-bound-sa-token\") pod \"547ef373-3835-4ec2-8f70-94954f34f452\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " Oct 30 02:41:46.517830 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.517705 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/547ef373-3835-4ec2-8f70-94954f34f452-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/547ef373-3835-4ec2-8f70-94954f34f452-ca-trust-extracted\") pod \"547ef373-3835-4ec2-8f70-94954f34f452\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " Oct 30 02:41:46.517830 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.517742 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-kube-api-access-9w87m\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-kube-api-access-9w87m\") pod \"547ef373-3835-4ec2-8f70-94954f34f452\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " Oct 30 02:41:46.517830 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.517787 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/547ef373-3835-4ec2-8f70-94954f34f452-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/547ef373-3835-4ec2-8f70-94954f34f452-registry-certificates\") pod \"547ef373-3835-4ec2-8f70-94954f34f452\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " Oct 30 02:41:46.517830 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.517819 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls\") pod \"547ef373-3835-4ec2-8f70-94954f34f452\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " Oct 30 02:41:46.518079 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.517842 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/547ef373-3835-4ec2-8f70-94954f34f452-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/547ef373-3835-4ec2-8f70-94954f34f452-trusted-ca\") pod \"547ef373-3835-4ec2-8f70-94954f34f452\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " Oct 30 02:41:46.518079 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.517893 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/547ef373-3835-4ec2-8f70-94954f34f452-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/547ef373-3835-4ec2-8f70-94954f34f452-image-registry-private-configuration\") pod \"547ef373-3835-4ec2-8f70-94954f34f452\" (UID: \"547ef373-3835-4ec2-8f70-94954f34f452\") " Oct 30 02:41:46.518347 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.518287 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/547ef373-3835-4ec2-8f70-94954f34f452-registry-certificates" pod "547ef373-3835-4ec2-8f70-94954f34f452" (UID: "547ef373-3835-4ec2-8f70-94954f34f452"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:46.518672 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.518639 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/547ef373-3835-4ec2-8f70-94954f34f452-trusted-ca" pod "547ef373-3835-4ec2-8f70-94954f34f452" (UID: "547ef373-3835-4ec2-8f70-94954f34f452"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:46.520087 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.520058 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls" pod "547ef373-3835-4ec2-8f70-94954f34f452" (UID: "547ef373-3835-4ec2-8f70-94954f34f452"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:46.520445 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.520424 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-bound-sa-token" pod "547ef373-3835-4ec2-8f70-94954f34f452" (UID: "547ef373-3835-4ec2-8f70-94954f34f452"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:46.520545 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.520479 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547ef373-3835-4ec2-8f70-94954f34f452-installation-pull-secrets" pod "547ef373-3835-4ec2-8f70-94954f34f452" (UID: "547ef373-3835-4ec2-8f70-94954f34f452"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:46.520586 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.520542 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547ef373-3835-4ec2-8f70-94954f34f452-image-registry-private-configuration" pod "547ef373-3835-4ec2-8f70-94954f34f452" (UID: "547ef373-3835-4ec2-8f70-94954f34f452"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:46.520586 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.520539 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-kube-api-access-9w87m" pod "547ef373-3835-4ec2-8f70-94954f34f452" (UID: "547ef373-3835-4ec2-8f70-94954f34f452"). InnerVolumeSpecName "kube-api-access-9w87m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:46.525089 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.525069 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547ef373-3835-4ec2-8f70-94954f34f452-ca-trust-extracted" pod "547ef373-3835-4ec2-8f70-94954f34f452" (UID: "547ef373-3835-4ec2-8f70-94954f34f452"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:46.619175 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.619133 4925 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/547ef373-3835-4ec2-8f70-94954f34f452-ca-trust-extracted\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:41:46.619175 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.619169 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w87m\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-kube-api-access-9w87m\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:41:46.619175 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.619179 4925 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/547ef373-3835-4ec2-8f70-94954f34f452-registry-certificates\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:41:46.619175 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.619189 4925 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-registry-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:41:46.619440 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.619198 4925 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/547ef373-3835-4ec2-8f70-94954f34f452-trusted-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:41:46.619440 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.619208 4925 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/547ef373-3835-4ec2-8f70-94954f34f452-image-registry-private-configuration\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:41:46.619440 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.619218 4925 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/547ef373-3835-4ec2-8f70-94954f34f452-installation-pull-secrets\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:41:46.619440 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.619226 4925 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/547ef373-3835-4ec2-8f70-94954f34f452-bound-sa-token\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:41:46.817640 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.817609 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65c57cd659-szcm2"] Oct 30 02:41:46.831371 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:46.831336 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-65c57cd659-szcm2"] Oct 30 02:41:47.118695 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:47.118667 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="547ef373-3835-4ec2-8f70-94954f34f452" path="/var/lib/kubelet/pods/547ef373-3835-4ec2-8f70-94954f34f452/volumes" Oct 30 02:41:48.161470 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:41:48.161433 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:41:49.486174 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:49.486142 4925 generic.go:356] "Generic (PLEG): container finished" podID="6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e" containerID="dedf8e40b1fa6b3767e3d6713659d1dfbdeb553ac31a306c0ec795f26b4aa1c5" exitCode=0 Oct 30 02:41:49.486543 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:49.486220 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" event={"ID":"6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e","Type":"ContainerDied","Data":"dedf8e40b1fa6b3767e3d6713659d1dfbdeb553ac31a306c0ec795f26b4aa1c5"} Oct 30 02:41:49.486543 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:49.486528 4925 scope.go:122] "RemoveContainer" containerID="dedf8e40b1fa6b3767e3d6713659d1dfbdeb553ac31a306c0ec795f26b4aa1c5" Oct 30 02:41:50.491407 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:41:50.491376 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-n22dv" event={"ID":"6aac72f9-3975-48ea-8dc0-f2cceaa0dc1e","Type":"ContainerStarted","Data":"c3ecd033ce8d1da6b2b1be6734ea0b641433106d4dbc6ba6c9fe212685c753b9"} Oct 30 02:42:08.553673 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:42:08.553635 4925 generic.go:356] "Generic (PLEG): container finished" podID="b0851f53-fa88-40f9-bfdb-beb956a5fb28" containerID="79ededb91e9281203b2625b1f49d55140e69f1a5729184264ae6b96e0e0c209d" exitCode=0 Oct 30 02:42:08.554246 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:42:08.553697 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zc4bx" event={"ID":"b0851f53-fa88-40f9-bfdb-beb956a5fb28","Type":"ContainerDied","Data":"79ededb91e9281203b2625b1f49d55140e69f1a5729184264ae6b96e0e0c209d"} Oct 30 02:42:08.554246 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:42:08.554137 4925 scope.go:122] "RemoveContainer" containerID="79ededb91e9281203b2625b1f49d55140e69f1a5729184264ae6b96e0e0c209d" Oct 30 02:42:09.558802 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:42:09.558769 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zc4bx" event={"ID":"b0851f53-fa88-40f9-bfdb-beb956a5fb28","Type":"ContainerStarted","Data":"c7c84b79be4b5948b51c3686a47f713bc9ba9d824e62c6039a74cbd41f7fab64"} Oct 30 02:42:18.161640 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:42:18.161603 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:42:48.161739 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:42:48.161698 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:43:18.162271 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:43:18.162227 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:43:48.161897 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:43:48.161857 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:44:18.160333 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:44:18.160294 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:44:48.161551 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:44:48.161513 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:45:18.161470 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:45:18.161431 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:45:39.128155 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:45:39.128127 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 02:45:39.129906 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:45:39.129880 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 02:45:39.136437 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:45:39.136421 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 02:45:39.138019 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:45:39.138002 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 02:45:48.160839 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:45:48.160798 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:46:18.161051 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:46:18.161002 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:46:48.160468 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:46:48.160428 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:47:18.161001 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:47:18.160965 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:47:48.161383 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:47:48.161299 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:48:18.160955 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:48:18.160917 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:48:48.161457 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:48:48.161423 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:49:18.161172 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:49:18.161111 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:49:48.161141 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:49:48.161094 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:50:18.161646 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:50:18.161535 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:50:39.150991 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:50:39.150911 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 02:50:39.152992 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:50:39.152969 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 02:50:39.159446 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:50:39.159428 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 02:50:39.161391 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:50:39.161373 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 02:50:48.161345 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:50:48.161299 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:51:18.160791 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:51:18.160738 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:51:48.161729 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:51:48.161696 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:52:18.160965 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:52:18.160929 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:52:48.162015 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:52:48.161976 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:53:13.943436 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:13.943405 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55"] Oct 30 02:53:13.943984 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:13.943682 4925 cpu_manager.go:414] "containerMap: removing container" podUID="11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" containerName="registry" Oct 30 02:53:13.943984 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:13.943691 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" containerName="registry" Oct 30 02:53:13.943984 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:13.943701 4925 cpu_manager.go:414] "containerMap: removing container" podUID="547ef373-3835-4ec2-8f70-94954f34f452" containerName="registry" Oct 30 02:53:13.943984 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:13.943705 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="547ef373-3835-4ec2-8f70-94954f34f452" containerName="registry" Oct 30 02:53:13.943984 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:13.943782 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="11560ce6-6d4a-4c30-83eb-b1b7a3e50cae" containerName="registry" Oct 30 02:53:13.943984 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:13.943790 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="547ef373-3835-4ec2-8f70-94954f34f452" containerName="registry" Oct 30 02:53:13.946806 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:13.946790 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55" Oct 30 02:53:13.974834 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:13.974810 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-f2z6c\"" Oct 30 02:53:13.974985 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:13.974822 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 02:53:13.974985 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:13.974902 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 02:53:13.976056 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:13.976036 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55"] Oct 30 02:53:14.051105 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:14.051070 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d717fdc-65b2-4216-83b0-aaf34f4c0093-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55\" (UID: \"7d717fdc-65b2-4216-83b0-aaf34f4c0093\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55" Oct 30 02:53:14.051310 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:14.051168 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d717fdc-65b2-4216-83b0-aaf34f4c0093-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55\" (UID: \"7d717fdc-65b2-4216-83b0-aaf34f4c0093\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55" Oct 30 02:53:14.051310 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:14.051224 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48s54\" (UniqueName: \"kubernetes.io/projected/7d717fdc-65b2-4216-83b0-aaf34f4c0093-kube-api-access-48s54\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55\" (UID: \"7d717fdc-65b2-4216-83b0-aaf34f4c0093\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55" Oct 30 02:53:14.152605 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:14.152564 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d717fdc-65b2-4216-83b0-aaf34f4c0093-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55\" (UID: \"7d717fdc-65b2-4216-83b0-aaf34f4c0093\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55" Oct 30 02:53:14.152605 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:14.152608 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48s54\" (UniqueName: \"kubernetes.io/projected/7d717fdc-65b2-4216-83b0-aaf34f4c0093-kube-api-access-48s54\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55\" (UID: \"7d717fdc-65b2-4216-83b0-aaf34f4c0093\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55" Oct 30 02:53:14.152893 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:14.152639 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d717fdc-65b2-4216-83b0-aaf34f4c0093-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55\" (UID: \"7d717fdc-65b2-4216-83b0-aaf34f4c0093\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55" Oct 30 02:53:14.152985 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:14.152965 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d717fdc-65b2-4216-83b0-aaf34f4c0093-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55\" (UID: \"7d717fdc-65b2-4216-83b0-aaf34f4c0093\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55" Oct 30 02:53:14.153040 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:14.152993 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d717fdc-65b2-4216-83b0-aaf34f4c0093-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55\" (UID: \"7d717fdc-65b2-4216-83b0-aaf34f4c0093\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55" Oct 30 02:53:14.176942 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:14.176912 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48s54\" (UniqueName: \"kubernetes.io/projected/7d717fdc-65b2-4216-83b0-aaf34f4c0093-kube-api-access-48s54\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55\" (UID: \"7d717fdc-65b2-4216-83b0-aaf34f4c0093\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55" Oct 30 02:53:14.261964 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:14.261884 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55" Oct 30 02:53:14.558088 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:14.558021 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55"] Oct 30 02:53:14.561186 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:53:14.561147 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d717fdc_65b2_4216_83b0_aaf34f4c0093.slice/crio-c1c4385be6e96ff863336d2850c92cdda550ef22793fd5e147852d6fe9d30632 WatchSource:0}: Error finding container c1c4385be6e96ff863336d2850c92cdda550ef22793fd5e147852d6fe9d30632: Status 404 returned error can't find the container with id c1c4385be6e96ff863336d2850c92cdda550ef22793fd5e147852d6fe9d30632 Oct 30 02:53:14.563261 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:14.563243 4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:53:14.865281 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:14.865214 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55" event={"ID":"7d717fdc-65b2-4216-83b0-aaf34f4c0093","Type":"ContainerStarted","Data":"c1c4385be6e96ff863336d2850c92cdda550ef22793fd5e147852d6fe9d30632"} Oct 30 02:53:18.165255 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:53:18.165215 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:53:19.886182 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:19.886095 4925 generic.go:356] "Generic (PLEG): container finished" podID="7d717fdc-65b2-4216-83b0-aaf34f4c0093" containerID="3ab9d0bd60ac5a643b6d0a3bc68564137347612f23a22144b834b9e7acc6d097" exitCode=0 Oct 30 02:53:19.886182 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:19.886142 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55" event={"ID":"7d717fdc-65b2-4216-83b0-aaf34f4c0093","Type":"ContainerDied","Data":"3ab9d0bd60ac5a643b6d0a3bc68564137347612f23a22144b834b9e7acc6d097"} Oct 30 02:53:22.899041 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:22.899007 4925 generic.go:356] "Generic (PLEG): container finished" podID="7d717fdc-65b2-4216-83b0-aaf34f4c0093" containerID="22ae1ec77f34fc16e4c08d6b23368021948a57ec06bbfbf09477ebeee26a9d01" exitCode=0 Oct 30 02:53:22.899359 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:22.899054 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55" event={"ID":"7d717fdc-65b2-4216-83b0-aaf34f4c0093","Type":"ContainerDied","Data":"22ae1ec77f34fc16e4c08d6b23368021948a57ec06bbfbf09477ebeee26a9d01"} Oct 30 02:53:28.925070 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:28.925025 4925 generic.go:356] "Generic (PLEG): container finished" podID="7d717fdc-65b2-4216-83b0-aaf34f4c0093" containerID="b4847f998a91a732116596413f63659bd2b1561e84329f0fd26c5f2de7aa3717" exitCode=0 Oct 30 02:53:28.925446 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:28.925076 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55" event={"ID":"7d717fdc-65b2-4216-83b0-aaf34f4c0093","Type":"ContainerDied","Data":"b4847f998a91a732116596413f63659bd2b1561e84329f0fd26c5f2de7aa3717"} Oct 30 02:53:30.055960 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:30.055933 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55" Oct 30 02:53:30.185382 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:30.185304 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d717fdc-65b2-4216-83b0-aaf34f4c0093-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d717fdc-65b2-4216-83b0-aaf34f4c0093-bundle\") pod \"7d717fdc-65b2-4216-83b0-aaf34f4c0093\" (UID: \"7d717fdc-65b2-4216-83b0-aaf34f4c0093\") " Oct 30 02:53:30.185382 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:30.185354 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d717fdc-65b2-4216-83b0-aaf34f4c0093-kube-api-access-48s54\" (UniqueName: \"kubernetes.io/projected/7d717fdc-65b2-4216-83b0-aaf34f4c0093-kube-api-access-48s54\") pod \"7d717fdc-65b2-4216-83b0-aaf34f4c0093\" (UID: \"7d717fdc-65b2-4216-83b0-aaf34f4c0093\") " Oct 30 02:53:30.185382 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:30.185384 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d717fdc-65b2-4216-83b0-aaf34f4c0093-util\" (UniqueName: \"kubernetes.io/empty-dir/7d717fdc-65b2-4216-83b0-aaf34f4c0093-util\") pod \"7d717fdc-65b2-4216-83b0-aaf34f4c0093\" (UID: \"7d717fdc-65b2-4216-83b0-aaf34f4c0093\") " Oct 30 02:53:30.185968 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:30.185944 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d717fdc-65b2-4216-83b0-aaf34f4c0093-bundle" pod "7d717fdc-65b2-4216-83b0-aaf34f4c0093" (UID: "7d717fdc-65b2-4216-83b0-aaf34f4c0093"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:53:30.187563 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:30.187537 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d717fdc-65b2-4216-83b0-aaf34f4c0093-kube-api-access-48s54" pod "7d717fdc-65b2-4216-83b0-aaf34f4c0093" (UID: "7d717fdc-65b2-4216-83b0-aaf34f4c0093"). InnerVolumeSpecName "kube-api-access-48s54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:53:30.189744 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:30.189724 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d717fdc-65b2-4216-83b0-aaf34f4c0093-util" pod "7d717fdc-65b2-4216-83b0-aaf34f4c0093" (UID: "7d717fdc-65b2-4216-83b0-aaf34f4c0093"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:53:30.286640 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:30.286610 4925 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d717fdc-65b2-4216-83b0-aaf34f4c0093-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:53:30.286640 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:30.286637 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48s54\" (UniqueName: \"kubernetes.io/projected/7d717fdc-65b2-4216-83b0-aaf34f4c0093-kube-api-access-48s54\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:53:30.286827 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:30.286651 4925 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d717fdc-65b2-4216-83b0-aaf34f4c0093-util\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:53:30.934332 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:30.934300 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55" Oct 30 02:53:30.934490 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:30.934306 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsz55" event={"ID":"7d717fdc-65b2-4216-83b0-aaf34f4c0093","Type":"ContainerDied","Data":"c1c4385be6e96ff863336d2850c92cdda550ef22793fd5e147852d6fe9d30632"} Oct 30 02:53:30.934490 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:30.934421 4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1c4385be6e96ff863336d2850c92cdda550ef22793fd5e147852d6fe9d30632" Oct 30 02:53:37.109201 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.109164 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-jhshn"] Oct 30 02:53:37.109574 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.109449 4925 cpu_manager.go:414] "containerMap: removing container" podUID="7d717fdc-65b2-4216-83b0-aaf34f4c0093" containerName="extract" Oct 30 02:53:37.109574 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.109460 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d717fdc-65b2-4216-83b0-aaf34f4c0093" containerName="extract" Oct 30 02:53:37.109574 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.109478 4925 cpu_manager.go:414] "containerMap: removing container" podUID="7d717fdc-65b2-4216-83b0-aaf34f4c0093" containerName="pull" Oct 30 02:53:37.109574 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.109484 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d717fdc-65b2-4216-83b0-aaf34f4c0093" containerName="pull" Oct 30 02:53:37.109574 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.109505 4925 cpu_manager.go:414] "containerMap: removing container" podUID="7d717fdc-65b2-4216-83b0-aaf34f4c0093" containerName="util" Oct 30 02:53:37.109574 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.109510 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d717fdc-65b2-4216-83b0-aaf34f4c0093" containerName="util" Oct 30 02:53:37.109574 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.109559 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d717fdc-65b2-4216-83b0-aaf34f4c0093" containerName="extract" Oct 30 02:53:37.144815 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.144783 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-jhshn"] Oct 30 02:53:37.144953 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.144887 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-jhshn" Oct 30 02:53:37.161383 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.161360 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-qbpj8\"" Oct 30 02:53:37.162130 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.162115 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:53:37.173973 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.173945 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 02:53:37.244446 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.244418 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abd9598b-8a7d-4bdf-87c8-05d0e3be964a-tmp\") pod \"cert-manager-operator-controller-manager-65fc875df4-jhshn\" (UID: \"abd9598b-8a7d-4bdf-87c8-05d0e3be964a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-jhshn" Oct 30 02:53:37.244594 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.244465 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd8kp\" (UniqueName: \"kubernetes.io/projected/abd9598b-8a7d-4bdf-87c8-05d0e3be964a-kube-api-access-sd8kp\") pod \"cert-manager-operator-controller-manager-65fc875df4-jhshn\" (UID: \"abd9598b-8a7d-4bdf-87c8-05d0e3be964a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-jhshn" Oct 30 02:53:37.345451 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.345419 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd8kp\" (UniqueName: \"kubernetes.io/projected/abd9598b-8a7d-4bdf-87c8-05d0e3be964a-kube-api-access-sd8kp\") pod \"cert-manager-operator-controller-manager-65fc875df4-jhshn\" (UID: \"abd9598b-8a7d-4bdf-87c8-05d0e3be964a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-jhshn" Oct 30 02:53:37.345622 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.345503 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abd9598b-8a7d-4bdf-87c8-05d0e3be964a-tmp\") pod \"cert-manager-operator-controller-manager-65fc875df4-jhshn\" (UID: \"abd9598b-8a7d-4bdf-87c8-05d0e3be964a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-jhshn" Oct 30 02:53:37.345931 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.345911 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abd9598b-8a7d-4bdf-87c8-05d0e3be964a-tmp\") pod \"cert-manager-operator-controller-manager-65fc875df4-jhshn\" (UID: \"abd9598b-8a7d-4bdf-87c8-05d0e3be964a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-jhshn" Oct 30 02:53:37.369612 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.369557 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd8kp\" (UniqueName: \"kubernetes.io/projected/abd9598b-8a7d-4bdf-87c8-05d0e3be964a-kube-api-access-sd8kp\") pod \"cert-manager-operator-controller-manager-65fc875df4-jhshn\" (UID: \"abd9598b-8a7d-4bdf-87c8-05d0e3be964a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-jhshn" Oct 30 02:53:37.466887 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.466843 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-jhshn" Oct 30 02:53:37.827920 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.827890 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-jhshn"] Oct 30 02:53:37.831454 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:53:37.831419 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd9598b_8a7d_4bdf_87c8_05d0e3be964a.slice/crio-7513e02911eed02316409d95859e3935da8466bd16e719c17c715fcc732be950 WatchSource:0}: Error finding container 7513e02911eed02316409d95859e3935da8466bd16e719c17c715fcc732be950: Status 404 returned error can't find the container with id 7513e02911eed02316409d95859e3935da8466bd16e719c17c715fcc732be950 Oct 30 02:53:37.961932 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:37.961896 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-jhshn" event={"ID":"abd9598b-8a7d-4bdf-87c8-05d0e3be964a","Type":"ContainerStarted","Data":"7513e02911eed02316409d95859e3935da8466bd16e719c17c715fcc732be950"} Oct 30 02:53:40.976100 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:40.976061 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-jhshn" event={"ID":"abd9598b-8a7d-4bdf-87c8-05d0e3be964a","Type":"ContainerStarted","Data":"a484d4faac4eac109b27009ad24e033f168771c80aff19a22c8c8f6c88786590"} Oct 30 02:53:41.023437 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:41.023387 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-jhshn" podStartSLOduration=1.868555362 podStartE2EDuration="4.023373812s" totalImagesPullingTime="2.15481845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:41.022838812 +0000 UTC m=+782.036116917" watchObservedRunningTime="2026-10-30 02:53:41.023373812 +0000 UTC m=+782.036651912" Oct 30 02:53:44.011040 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:44.011005 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4xhtl"] Oct 30 02:53:44.013917 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:44.013900 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4xhtl" Oct 30 02:53:44.040659 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:44.040633 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-ghwqq\"" Oct 30 02:53:44.043820 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:44.043796 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4xhtl"] Oct 30 02:53:44.053353 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:44.053332 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 02:53:44.053469 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:44.053371 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 02:53:44.098235 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:44.098200 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d640af6-b360-47ae-9fdc-d309506f44dc-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4xhtl\" (UID: \"8d640af6-b360-47ae-9fdc-d309506f44dc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4xhtl" Oct 30 02:53:44.098364 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:44.098260 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtptr\" (UniqueName: \"kubernetes.io/projected/8d640af6-b360-47ae-9fdc-d309506f44dc-kube-api-access-vtptr\") pod \"cert-manager-cainjector-5c57bbd4fc-4xhtl\" (UID: \"8d640af6-b360-47ae-9fdc-d309506f44dc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4xhtl" Oct 30 02:53:44.199647 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:44.199612 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d640af6-b360-47ae-9fdc-d309506f44dc-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4xhtl\" (UID: \"8d640af6-b360-47ae-9fdc-d309506f44dc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4xhtl" Oct 30 02:53:44.199817 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:44.199665 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtptr\" (UniqueName: \"kubernetes.io/projected/8d640af6-b360-47ae-9fdc-d309506f44dc-kube-api-access-vtptr\") pod \"cert-manager-cainjector-5c57bbd4fc-4xhtl\" (UID: \"8d640af6-b360-47ae-9fdc-d309506f44dc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4xhtl" Oct 30 02:53:44.223744 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:44.223716 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d640af6-b360-47ae-9fdc-d309506f44dc-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4xhtl\" (UID: \"8d640af6-b360-47ae-9fdc-d309506f44dc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4xhtl" Oct 30 02:53:44.223931 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:44.223908 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtptr\" (UniqueName: \"kubernetes.io/projected/8d640af6-b360-47ae-9fdc-d309506f44dc-kube-api-access-vtptr\") pod \"cert-manager-cainjector-5c57bbd4fc-4xhtl\" (UID: \"8d640af6-b360-47ae-9fdc-d309506f44dc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4xhtl" Oct 30 02:53:44.331398 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:44.331370 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4xhtl" Oct 30 02:53:44.626967 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:44.626884 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4xhtl"] Oct 30 02:53:44.630165 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:53:44.630138 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d640af6_b360_47ae_9fdc_d309506f44dc.slice/crio-5bf6bb49d16e1888e327c2ffe374699ccd8311f1f40015566cf25d3bfd11a8ce WatchSource:0}: Error finding container 5bf6bb49d16e1888e327c2ffe374699ccd8311f1f40015566cf25d3bfd11a8ce: Status 404 returned error can't find the container with id 5bf6bb49d16e1888e327c2ffe374699ccd8311f1f40015566cf25d3bfd11a8ce Oct 30 02:53:44.992796 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:44.992705 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4xhtl" event={"ID":"8d640af6-b360-47ae-9fdc-d309506f44dc","Type":"ContainerStarted","Data":"5bf6bb49d16e1888e327c2ffe374699ccd8311f1f40015566cf25d3bfd11a8ce"} Oct 30 02:53:48.005823 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:48.005784 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4xhtl" event={"ID":"8d640af6-b360-47ae-9fdc-d309506f44dc","Type":"ContainerStarted","Data":"292bfb3a59583c5061749740d73d192f8289248af9d252cf6fe28459614c0bcd"} Oct 30 02:53:48.048412 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:53:48.048357 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4xhtl" podStartSLOduration=2.7122383770000003 podStartE2EDuration="5.048344726s" totalImagesPullingTime="2.336106349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:48.047359617 +0000 UTC m=+789.060637720" watchObservedRunningTime="2026-10-30 02:53:48.048344726 +0000 UTC m=+789.061622829" Oct 30 02:53:48.160651 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:53:48.160618 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:54:18.161020 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:54:18.160973 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:54:28.687955 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:28.687925 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-observability-system/odh-observability-b9cbbfc76-6z5ws"] Oct 30 02:54:28.691044 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:28.691025 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-observability-system/odh-observability-b9cbbfc76-6z5ws" Oct 30 02:54:28.707016 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:28.706998 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-observability-system\"/\"openshift-service-ca.crt\"" Oct 30 02:54:28.719075 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:28.719047 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-observability-system\"/\"odh-observability-webhook-cert\"" Oct 30 02:54:28.719194 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:28.719122 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-observability-system\"/\"kube-root-ca.crt\"" Oct 30 02:54:28.719194 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:28.719156 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-observability-system\"/\"odh-observability-dockercfg-s79m7\"" Oct 30 02:54:28.719194 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:28.719190 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-observability-system\"/\"odh-observability-metrics-tls\"" Oct 30 02:54:28.720201 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:28.720182 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-observability-system/odh-observability-b9cbbfc76-6z5ws"] Oct 30 02:54:28.849073 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:28.849038 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9hlk\" (UniqueName: \"kubernetes.io/projected/dc3ab9cb-d80d-4ec7-9208-5d6510623867-kube-api-access-f9hlk\") pod \"odh-observability-b9cbbfc76-6z5ws\" (UID: \"dc3ab9cb-d80d-4ec7-9208-5d6510623867\") " pod="odh-observability-system/odh-observability-b9cbbfc76-6z5ws" Oct 30 02:54:28.849236 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:28.849101 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc3ab9cb-d80d-4ec7-9208-5d6510623867-metrics-tls\") pod \"odh-observability-b9cbbfc76-6z5ws\" (UID: \"dc3ab9cb-d80d-4ec7-9208-5d6510623867\") " pod="odh-observability-system/odh-observability-b9cbbfc76-6z5ws" Oct 30 02:54:28.849236 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:28.849185 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/dc3ab9cb-d80d-4ec7-9208-5d6510623867-webhook-certs\") pod \"odh-observability-b9cbbfc76-6z5ws\" (UID: \"dc3ab9cb-d80d-4ec7-9208-5d6510623867\") " pod="odh-observability-system/odh-observability-b9cbbfc76-6z5ws" Oct 30 02:54:28.950442 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:28.950366 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/dc3ab9cb-d80d-4ec7-9208-5d6510623867-webhook-certs\") pod \"odh-observability-b9cbbfc76-6z5ws\" (UID: \"dc3ab9cb-d80d-4ec7-9208-5d6510623867\") " pod="odh-observability-system/odh-observability-b9cbbfc76-6z5ws" Oct 30 02:54:28.950442 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:28.950417 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9hlk\" (UniqueName: \"kubernetes.io/projected/dc3ab9cb-d80d-4ec7-9208-5d6510623867-kube-api-access-f9hlk\") pod \"odh-observability-b9cbbfc76-6z5ws\" (UID: \"dc3ab9cb-d80d-4ec7-9208-5d6510623867\") " pod="odh-observability-system/odh-observability-b9cbbfc76-6z5ws" Oct 30 02:54:28.950806 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:28.950453 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc3ab9cb-d80d-4ec7-9208-5d6510623867-metrics-tls\") pod \"odh-observability-b9cbbfc76-6z5ws\" (UID: \"dc3ab9cb-d80d-4ec7-9208-5d6510623867\") " pod="odh-observability-system/odh-observability-b9cbbfc76-6z5ws" Oct 30 02:54:28.950806 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:54:28.950528 4925 secret.go:189] Couldn't get secret odh-observability-system/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 30 02:54:28.950806 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:54:28.950619 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3ab9cb-d80d-4ec7-9208-5d6510623867-webhook-certs podName:dc3ab9cb-d80d-4ec7-9208-5d6510623867 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:29.450597841 +0000 UTC m=+830.463875938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/dc3ab9cb-d80d-4ec7-9208-5d6510623867-webhook-certs") pod "odh-observability-b9cbbfc76-6z5ws" (UID: "dc3ab9cb-d80d-4ec7-9208-5d6510623867") : secret "odh-observability-webhook-cert" not found Oct 30 02:54:28.952804 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:28.952786 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc3ab9cb-d80d-4ec7-9208-5d6510623867-metrics-tls\") pod \"odh-observability-b9cbbfc76-6z5ws\" (UID: \"dc3ab9cb-d80d-4ec7-9208-5d6510623867\") " pod="odh-observability-system/odh-observability-b9cbbfc76-6z5ws" Oct 30 02:54:28.983209 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:28.983184 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9hlk\" (UniqueName: \"kubernetes.io/projected/dc3ab9cb-d80d-4ec7-9208-5d6510623867-kube-api-access-f9hlk\") pod \"odh-observability-b9cbbfc76-6z5ws\" (UID: \"dc3ab9cb-d80d-4ec7-9208-5d6510623867\") " pod="odh-observability-system/odh-observability-b9cbbfc76-6z5ws" Oct 30 02:54:29.454557 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:29.454523 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/dc3ab9cb-d80d-4ec7-9208-5d6510623867-webhook-certs\") pod \"odh-observability-b9cbbfc76-6z5ws\" (UID: \"dc3ab9cb-d80d-4ec7-9208-5d6510623867\") " pod="odh-observability-system/odh-observability-b9cbbfc76-6z5ws" Oct 30 02:54:29.456914 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:29.456891 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/dc3ab9cb-d80d-4ec7-9208-5d6510623867-webhook-certs\") pod \"odh-observability-b9cbbfc76-6z5ws\" (UID: \"dc3ab9cb-d80d-4ec7-9208-5d6510623867\") " pod="odh-observability-system/odh-observability-b9cbbfc76-6z5ws" Oct 30 02:54:29.603491 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:29.603457 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-observability-system/odh-observability-b9cbbfc76-6z5ws" Oct 30 02:54:29.904708 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:29.904676 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-observability-system/odh-observability-b9cbbfc76-6z5ws"] Oct 30 02:54:29.908013 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:54:29.907984 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3ab9cb_d80d_4ec7_9208_5d6510623867.slice/crio-065741fae4d83675c8a67bc79cdc2ddf55cf522a08180d32ec0b52cd143aef82 WatchSource:0}: Error finding container 065741fae4d83675c8a67bc79cdc2ddf55cf522a08180d32ec0b52cd143aef82: Status 404 returned error can't find the container with id 065741fae4d83675c8a67bc79cdc2ddf55cf522a08180d32ec0b52cd143aef82 Oct 30 02:54:30.164745 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:30.164662 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-observability-system/odh-observability-b9cbbfc76-6z5ws" event={"ID":"dc3ab9cb-d80d-4ec7-9208-5d6510623867","Type":"ContainerStarted","Data":"065741fae4d83675c8a67bc79cdc2ddf55cf522a08180d32ec0b52cd143aef82"} Oct 30 02:54:32.174657 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:32.174622 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-observability-system/odh-observability-b9cbbfc76-6z5ws" event={"ID":"dc3ab9cb-d80d-4ec7-9208-5d6510623867","Type":"ContainerStarted","Data":"aafab3a43c0f2c389cfb46cfb1846051d681179fe4ef09461c98f6f411e19e99"} Oct 30 02:54:32.175053 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:32.174738 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-observability-system/odh-observability-b9cbbfc76-6z5ws" Oct 30 02:54:32.218119 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:32.218065 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-observability-system/odh-observability-b9cbbfc76-6z5ws" podStartSLOduration=2.04669079 podStartE2EDuration="4.21805142s" totalImagesPullingTime="2.17136063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:32.217601832 +0000 UTC m=+833.230879934" watchObservedRunningTime="2026-10-30 02:54:32.21805142 +0000 UTC m=+833.231329523" Oct 30 02:54:43.180968 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:54:43.180937 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-observability-system/odh-observability-b9cbbfc76-6z5ws" Oct 30 02:54:48.161578 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:54:48.161538 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:55:18.160991 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:55:18.160956 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:55:39.173979 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:39.173948 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 02:55:39.176763 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:39.176725 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 02:55:39.182719 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:39.182699 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 02:55:39.185325 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:39.185306 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 02:55:48.160637 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:55:48.160596 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:55:53.403904 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:53.403866 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4"] Oct 30 02:55:53.407464 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:53.407440 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4" Oct 30 02:55:53.435410 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:53.435383 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 02:55:53.435410 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:53.435388 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 02:55:53.435836 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:53.435817 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e1df189bb31fb8c419c1e848482afda4dccb51-08dd003f-dockercfg-v56mc\"" Oct 30 02:55:53.436652 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:53.436636 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4"] Oct 30 02:55:53.501005 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:53.500981 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e32d564-a869-4b11-ac31-bf1935b1b1eb-util\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4\" (UID: \"3e32d564-a869-4b11-ac31-bf1935b1b1eb\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4" Oct 30 02:55:53.501152 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:53.501042 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9mxn\" (UniqueName: \"kubernetes.io/projected/3e32d564-a869-4b11-ac31-bf1935b1b1eb-kube-api-access-s9mxn\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4\" (UID: \"3e32d564-a869-4b11-ac31-bf1935b1b1eb\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4" Oct 30 02:55:53.501152 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:53.501131 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e32d564-a869-4b11-ac31-bf1935b1b1eb-bundle\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4\" (UID: \"3e32d564-a869-4b11-ac31-bf1935b1b1eb\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4" Oct 30 02:55:53.601689 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:53.601658 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9mxn\" (UniqueName: \"kubernetes.io/projected/3e32d564-a869-4b11-ac31-bf1935b1b1eb-kube-api-access-s9mxn\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4\" (UID: \"3e32d564-a869-4b11-ac31-bf1935b1b1eb\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4" Oct 30 02:55:53.601879 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:53.601701 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e32d564-a869-4b11-ac31-bf1935b1b1eb-bundle\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4\" (UID: \"3e32d564-a869-4b11-ac31-bf1935b1b1eb\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4" Oct 30 02:55:53.601879 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:53.601723 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e32d564-a869-4b11-ac31-bf1935b1b1eb-util\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4\" (UID: \"3e32d564-a869-4b11-ac31-bf1935b1b1eb\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4" Oct 30 02:55:53.602091 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:53.602069 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e32d564-a869-4b11-ac31-bf1935b1b1eb-bundle\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4\" (UID: \"3e32d564-a869-4b11-ac31-bf1935b1b1eb\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4" Oct 30 02:55:53.602128 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:53.602092 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e32d564-a869-4b11-ac31-bf1935b1b1eb-util\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4\" (UID: \"3e32d564-a869-4b11-ac31-bf1935b1b1eb\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4" Oct 30 02:55:53.626311 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:53.626285 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9mxn\" (UniqueName: \"kubernetes.io/projected/3e32d564-a869-4b11-ac31-bf1935b1b1eb-kube-api-access-s9mxn\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4\" (UID: \"3e32d564-a869-4b11-ac31-bf1935b1b1eb\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4" Oct 30 02:55:53.722391 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:53.722316 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4" Oct 30 02:55:53.974078 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:53.974001 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87"] Oct 30 02:55:53.977654 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:53.977636 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87" Oct 30 02:55:53.992842 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:53.992820 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6fb1a13d619f6ee73e221b33f4898cc831d48b-0ad9862f-dockercfg-xxvvb\"" Oct 30 02:55:53.998525 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:53.998501 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87"] Oct 30 02:55:54.004838 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:54.004813 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f66de600-b77d-4ec2-b9ee-c83ed413a55e-util\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87\" (UID: \"f66de600-b77d-4ec2-b9ee-c83ed413a55e\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87" Oct 30 02:55:54.004951 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:54.004842 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f66de600-b77d-4ec2-b9ee-c83ed413a55e-bundle\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87\" (UID: \"f66de600-b77d-4ec2-b9ee-c83ed413a55e\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87" Oct 30 02:55:54.004951 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:54.004860 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x44z5\" (UniqueName: \"kubernetes.io/projected/f66de600-b77d-4ec2-b9ee-c83ed413a55e-kube-api-access-x44z5\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87\" (UID: \"f66de600-b77d-4ec2-b9ee-c83ed413a55e\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87" Oct 30 02:55:54.032009 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:54.031989 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4"] Oct 30 02:55:54.034106 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:55:54.034084 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e32d564_a869_4b11_ac31_bf1935b1b1eb.slice/crio-5c630d3b1cb2973cd80092ad87adffcef76ea04634c763cd59e129d1b8e67e7a WatchSource:0}: Error finding container 5c630d3b1cb2973cd80092ad87adffcef76ea04634c763cd59e129d1b8e67e7a: Status 404 returned error can't find the container with id 5c630d3b1cb2973cd80092ad87adffcef76ea04634c763cd59e129d1b8e67e7a Oct 30 02:55:54.105603 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:54.105577 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f66de600-b77d-4ec2-b9ee-c83ed413a55e-util\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87\" (UID: \"f66de600-b77d-4ec2-b9ee-c83ed413a55e\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87" Oct 30 02:55:54.105713 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:54.105609 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f66de600-b77d-4ec2-b9ee-c83ed413a55e-bundle\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87\" (UID: \"f66de600-b77d-4ec2-b9ee-c83ed413a55e\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87" Oct 30 02:55:54.105713 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:54.105633 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x44z5\" (UniqueName: \"kubernetes.io/projected/f66de600-b77d-4ec2-b9ee-c83ed413a55e-kube-api-access-x44z5\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87\" (UID: \"f66de600-b77d-4ec2-b9ee-c83ed413a55e\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87" Oct 30 02:55:54.105996 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:54.105976 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f66de600-b77d-4ec2-b9ee-c83ed413a55e-util\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87\" (UID: \"f66de600-b77d-4ec2-b9ee-c83ed413a55e\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87" Oct 30 02:55:54.106064 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:54.106041 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f66de600-b77d-4ec2-b9ee-c83ed413a55e-bundle\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87\" (UID: \"f66de600-b77d-4ec2-b9ee-c83ed413a55e\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87" Oct 30 02:55:54.130185 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:54.130162 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x44z5\" (UniqueName: \"kubernetes.io/projected/f66de600-b77d-4ec2-b9ee-c83ed413a55e-kube-api-access-x44z5\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87\" (UID: \"f66de600-b77d-4ec2-b9ee-c83ed413a55e\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87" Oct 30 02:55:54.293733 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:54.293650 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87" Oct 30 02:55:54.498076 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:54.498046 4925 generic.go:356] "Generic (PLEG): container finished" podID="3e32d564-a869-4b11-ac31-bf1935b1b1eb" containerID="067d41c4c52ed06013679d505bbe9ea8d47501f337c4fdafadd97416b891b508" exitCode=0 Oct 30 02:55:54.498471 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:54.498140 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4" event={"ID":"3e32d564-a869-4b11-ac31-bf1935b1b1eb","Type":"ContainerDied","Data":"067d41c4c52ed06013679d505bbe9ea8d47501f337c4fdafadd97416b891b508"} Oct 30 02:55:54.498471 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:54.498199 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4" event={"ID":"3e32d564-a869-4b11-ac31-bf1935b1b1eb","Type":"ContainerStarted","Data":"5c630d3b1cb2973cd80092ad87adffcef76ea04634c763cd59e129d1b8e67e7a"} Oct 30 02:55:54.590710 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:54.590687 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87"] Oct 30 02:55:54.593116 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:55:54.593086 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66de600_b77d_4ec2_b9ee_c83ed413a55e.slice/crio-40f11b53e093e601fd725c99907f447158575cf786ef8c87e734f67229e116e9 WatchSource:0}: Error finding container 40f11b53e093e601fd725c99907f447158575cf786ef8c87e734f67229e116e9: Status 404 returned error can't find the container with id 40f11b53e093e601fd725c99907f447158575cf786ef8c87e734f67229e116e9 Oct 30 02:55:55.504416 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:55.504381 4925 generic.go:356] "Generic (PLEG): container finished" podID="3e32d564-a869-4b11-ac31-bf1935b1b1eb" containerID="cec2604693699962c38be2027676ae37ecdc6877cc946a95c6124c4086bf8e38" exitCode=0 Oct 30 02:55:55.504836 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:55.504462 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4" event={"ID":"3e32d564-a869-4b11-ac31-bf1935b1b1eb","Type":"ContainerDied","Data":"cec2604693699962c38be2027676ae37ecdc6877cc946a95c6124c4086bf8e38"} Oct 30 02:55:55.506019 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:55.505997 4925 generic.go:356] "Generic (PLEG): container finished" podID="f66de600-b77d-4ec2-b9ee-c83ed413a55e" containerID="15b2ba5b38f396f6e7344fcff8450efe7a231b0a5950d05995b7f31c18f169b6" exitCode=0 Oct 30 02:55:55.506123 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:55.506041 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87" event={"ID":"f66de600-b77d-4ec2-b9ee-c83ed413a55e","Type":"ContainerDied","Data":"15b2ba5b38f396f6e7344fcff8450efe7a231b0a5950d05995b7f31c18f169b6"} Oct 30 02:55:55.506123 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:55.506075 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87" event={"ID":"f66de600-b77d-4ec2-b9ee-c83ed413a55e","Type":"ContainerStarted","Data":"40f11b53e093e601fd725c99907f447158575cf786ef8c87e734f67229e116e9"} Oct 30 02:55:56.295836 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:56.295746 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c"] Oct 30 02:55:56.299257 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:56.299239 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c" Oct 30 02:55:56.319110 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:56.319089 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"07f0b9f07fd02dc1509b226b57404b9c198c3c-8adea440-dockercfg-46fzw\"" Oct 30 02:55:56.322887 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:56.322861 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h5tn\" (UniqueName: \"kubernetes.io/projected/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-kube-api-access-6h5tn\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c\" (UID: \"4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c" Oct 30 02:55:56.323037 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:56.323016 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-bundle\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c\" (UID: \"4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c" Oct 30 02:55:56.323101 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:56.323050 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-util\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c\" (UID: \"4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c" Oct 30 02:55:56.324036 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:56.324014 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c"] Oct 30 02:55:56.423624 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:56.423585 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-bundle\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c\" (UID: \"4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c" Oct 30 02:55:56.423624 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:56.423622 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-util\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c\" (UID: \"4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c" Oct 30 02:55:56.423903 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:56.423642 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h5tn\" (UniqueName: \"kubernetes.io/projected/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-kube-api-access-6h5tn\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c\" (UID: \"4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c" Oct 30 02:55:56.423984 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:56.423961 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-bundle\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c\" (UID: \"4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c" Oct 30 02:55:56.424067 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:56.424032 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-util\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c\" (UID: \"4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c" Oct 30 02:55:56.448145 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:56.448122 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h5tn\" (UniqueName: \"kubernetes.io/projected/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-kube-api-access-6h5tn\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c\" (UID: \"4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c" Oct 30 02:55:56.512880 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:56.512843 4925 generic.go:356] "Generic (PLEG): container finished" podID="f66de600-b77d-4ec2-b9ee-c83ed413a55e" containerID="6ea65dfc1483be21e42b2bb58167501891c57241e3f18f26a8f145767285dc40" exitCode=0 Oct 30 02:55:56.513271 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:56.512878 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87" event={"ID":"f66de600-b77d-4ec2-b9ee-c83ed413a55e","Type":"ContainerDied","Data":"6ea65dfc1483be21e42b2bb58167501891c57241e3f18f26a8f145767285dc40"} Oct 30 02:55:56.515349 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:56.515328 4925 generic.go:356] "Generic (PLEG): container finished" podID="3e32d564-a869-4b11-ac31-bf1935b1b1eb" containerID="45632ef734a1ece3145a3c075dbe2ef13b628da492f26056b98ceb568adf4e96" exitCode=0 Oct 30 02:55:56.515451 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:56.515378 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4" event={"ID":"3e32d564-a869-4b11-ac31-bf1935b1b1eb","Type":"ContainerDied","Data":"45632ef734a1ece3145a3c075dbe2ef13b628da492f26056b98ceb568adf4e96"} Oct 30 02:55:56.615087 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:56.615060 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c" Oct 30 02:55:56.925078 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:55:56.925048 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e20b7ae_cf5a_4f27_b3e0_4717ca9e7a9b.slice/crio-ffc8c7a263d6316f76ed8aba084f6e2a052565bec9317051425a16ba2f90d17c WatchSource:0}: Error finding container ffc8c7a263d6316f76ed8aba084f6e2a052565bec9317051425a16ba2f90d17c: Status 404 returned error can't find the container with id ffc8c7a263d6316f76ed8aba084f6e2a052565bec9317051425a16ba2f90d17c Oct 30 02:55:56.925248 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:56.925224 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c"] Oct 30 02:55:57.520353 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:57.520316 4925 generic.go:356] "Generic (PLEG): container finished" podID="4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b" containerID="6f0a2cc8f1211a19ddbb9a10e9a94a3cca9131c36d17f36d0980bdee3f179907" exitCode=0 Oct 30 02:55:57.520810 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:57.520403 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c" event={"ID":"4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b","Type":"ContainerDied","Data":"6f0a2cc8f1211a19ddbb9a10e9a94a3cca9131c36d17f36d0980bdee3f179907"} Oct 30 02:55:57.520810 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:57.520456 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c" event={"ID":"4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b","Type":"ContainerStarted","Data":"ffc8c7a263d6316f76ed8aba084f6e2a052565bec9317051425a16ba2f90d17c"} Oct 30 02:55:57.522729 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:57.522708 4925 generic.go:356] "Generic (PLEG): container finished" podID="f66de600-b77d-4ec2-b9ee-c83ed413a55e" containerID="9353aea1409c4ec6b0621c4a687c831c7209ce05a308b5123f1fc316b8d70e34" exitCode=0 Oct 30 02:55:57.522828 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:57.522786 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87" event={"ID":"f66de600-b77d-4ec2-b9ee-c83ed413a55e","Type":"ContainerDied","Data":"9353aea1409c4ec6b0621c4a687c831c7209ce05a308b5123f1fc316b8d70e34"} Oct 30 02:55:57.638603 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:57.638577 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4" Oct 30 02:55:57.734580 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:57.734540 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e32d564-a869-4b11-ac31-bf1935b1b1eb-util\" (UniqueName: \"kubernetes.io/empty-dir/3e32d564-a869-4b11-ac31-bf1935b1b1eb-util\") pod \"3e32d564-a869-4b11-ac31-bf1935b1b1eb\" (UID: \"3e32d564-a869-4b11-ac31-bf1935b1b1eb\") " Oct 30 02:55:57.734794 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:57.734637 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e32d564-a869-4b11-ac31-bf1935b1b1eb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e32d564-a869-4b11-ac31-bf1935b1b1eb-bundle\") pod \"3e32d564-a869-4b11-ac31-bf1935b1b1eb\" (UID: \"3e32d564-a869-4b11-ac31-bf1935b1b1eb\") " Oct 30 02:55:57.734794 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:57.734694 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e32d564-a869-4b11-ac31-bf1935b1b1eb-kube-api-access-s9mxn\" (UniqueName: \"kubernetes.io/projected/3e32d564-a869-4b11-ac31-bf1935b1b1eb-kube-api-access-s9mxn\") pod \"3e32d564-a869-4b11-ac31-bf1935b1b1eb\" (UID: \"3e32d564-a869-4b11-ac31-bf1935b1b1eb\") " Oct 30 02:55:57.735342 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:57.735315 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e32d564-a869-4b11-ac31-bf1935b1b1eb-bundle" pod "3e32d564-a869-4b11-ac31-bf1935b1b1eb" (UID: "3e32d564-a869-4b11-ac31-bf1935b1b1eb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:55:57.736809 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:57.736789 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e32d564-a869-4b11-ac31-bf1935b1b1eb-kube-api-access-s9mxn" pod "3e32d564-a869-4b11-ac31-bf1935b1b1eb" (UID: "3e32d564-a869-4b11-ac31-bf1935b1b1eb"). InnerVolumeSpecName "kube-api-access-s9mxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:55:57.740446 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:57.740423 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e32d564-a869-4b11-ac31-bf1935b1b1eb-util" pod "3e32d564-a869-4b11-ac31-bf1935b1b1eb" (UID: "3e32d564-a869-4b11-ac31-bf1935b1b1eb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:55:57.835829 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:57.835798 4925 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e32d564-a869-4b11-ac31-bf1935b1b1eb-util\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:55:57.835829 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:57.835824 4925 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e32d564-a869-4b11-ac31-bf1935b1b1eb-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:55:57.835829 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:57.835833 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9mxn\" (UniqueName: \"kubernetes.io/projected/3e32d564-a869-4b11-ac31-bf1935b1b1eb-kube-api-access-s9mxn\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:55:58.528653 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:58.528562 4925 generic.go:356] "Generic (PLEG): container finished" podID="4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b" containerID="a5dde5f2685bb007d0bc30f6ea7de2d1054eef95cc0bef7515eb994809ffbcdb" exitCode=0 Oct 30 02:55:58.529113 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:58.528651 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c" event={"ID":"4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b","Type":"ContainerDied","Data":"a5dde5f2685bb007d0bc30f6ea7de2d1054eef95cc0bef7515eb994809ffbcdb"} Oct 30 02:55:58.530663 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:58.530628 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4" event={"ID":"3e32d564-a869-4b11-ac31-bf1935b1b1eb","Type":"ContainerDied","Data":"5c630d3b1cb2973cd80092ad87adffcef76ea04634c763cd59e129d1b8e67e7a"} Oct 30 02:55:58.530663 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:58.530655 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974hqlk4" Oct 30 02:55:58.530797 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:58.530682 4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c630d3b1cb2973cd80092ad87adffcef76ea04634c763cd59e129d1b8e67e7a" Oct 30 02:55:58.654998 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:58.654976 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87" Oct 30 02:55:58.744231 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:58.744200 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66de600-b77d-4ec2-b9ee-c83ed413a55e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f66de600-b77d-4ec2-b9ee-c83ed413a55e-bundle\") pod \"f66de600-b77d-4ec2-b9ee-c83ed413a55e\" (UID: \"f66de600-b77d-4ec2-b9ee-c83ed413a55e\") " Oct 30 02:55:58.744372 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:58.744244 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66de600-b77d-4ec2-b9ee-c83ed413a55e-util\" (UniqueName: \"kubernetes.io/empty-dir/f66de600-b77d-4ec2-b9ee-c83ed413a55e-util\") pod \"f66de600-b77d-4ec2-b9ee-c83ed413a55e\" (UID: \"f66de600-b77d-4ec2-b9ee-c83ed413a55e\") " Oct 30 02:55:58.744372 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:58.744279 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f66de600-b77d-4ec2-b9ee-c83ed413a55e-kube-api-access-x44z5\" (UniqueName: \"kubernetes.io/projected/f66de600-b77d-4ec2-b9ee-c83ed413a55e-kube-api-access-x44z5\") pod \"f66de600-b77d-4ec2-b9ee-c83ed413a55e\" (UID: \"f66de600-b77d-4ec2-b9ee-c83ed413a55e\") " Oct 30 02:55:58.745615 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:58.745581 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66de600-b77d-4ec2-b9ee-c83ed413a55e-bundle" pod "f66de600-b77d-4ec2-b9ee-c83ed413a55e" (UID: "f66de600-b77d-4ec2-b9ee-c83ed413a55e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:55:58.746418 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:58.746391 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66de600-b77d-4ec2-b9ee-c83ed413a55e-kube-api-access-x44z5" pod "f66de600-b77d-4ec2-b9ee-c83ed413a55e" (UID: "f66de600-b77d-4ec2-b9ee-c83ed413a55e"). InnerVolumeSpecName "kube-api-access-x44z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:55:58.749990 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:58.749970 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66de600-b77d-4ec2-b9ee-c83ed413a55e-util" pod "f66de600-b77d-4ec2-b9ee-c83ed413a55e" (UID: "f66de600-b77d-4ec2-b9ee-c83ed413a55e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:55:58.845437 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:58.845416 4925 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f66de600-b77d-4ec2-b9ee-c83ed413a55e-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:55:58.845437 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:58.845438 4925 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f66de600-b77d-4ec2-b9ee-c83ed413a55e-util\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:55:58.845602 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:58.845448 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x44z5\" (UniqueName: \"kubernetes.io/projected/f66de600-b77d-4ec2-b9ee-c83ed413a55e-kube-api-access-x44z5\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:55:59.536152 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:59.536074 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87" Oct 30 02:55:59.536152 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:59.536101 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfkzc87" event={"ID":"f66de600-b77d-4ec2-b9ee-c83ed413a55e","Type":"ContainerDied","Data":"40f11b53e093e601fd725c99907f447158575cf786ef8c87e734f67229e116e9"} Oct 30 02:55:59.536624 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:59.536158 4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40f11b53e093e601fd725c99907f447158575cf786ef8c87e734f67229e116e9" Oct 30 02:55:59.538442 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:59.538414 4925 generic.go:356] "Generic (PLEG): container finished" podID="4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b" containerID="796880dbcae4aa2c5a43b5a2b8d1ea6086b231d5f0c7a5c667f606c57c49e8db" exitCode=0 Oct 30 02:55:59.538580 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:55:59.538464 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c" event={"ID":"4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b","Type":"ContainerDied","Data":"796880dbcae4aa2c5a43b5a2b8d1ea6086b231d5f0c7a5c667f606c57c49e8db"} Oct 30 02:56:00.666685 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:00.666660 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c" Oct 30 02:56:00.760738 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:00.760706 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-kube-api-access-6h5tn\" (UniqueName: \"kubernetes.io/projected/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-kube-api-access-6h5tn\") pod \"4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b\" (UID: \"4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b\") " Oct 30 02:56:00.760917 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:00.760763 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-bundle\") pod \"4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b\" (UID: \"4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b\") " Oct 30 02:56:00.760917 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:00.760835 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-util\" (UniqueName: \"kubernetes.io/empty-dir/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-util\") pod \"4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b\" (UID: \"4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b\") " Oct 30 02:56:00.761319 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:00.761291 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-bundle" pod "4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b" (UID: "4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:56:00.762838 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:00.762809 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-kube-api-access-6h5tn" pod "4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b" (UID: "4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b"). InnerVolumeSpecName "kube-api-access-6h5tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:56:00.766239 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:00.766212 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-util" pod "4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b" (UID: "4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:56:00.862363 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:00.862327 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h5tn\" (UniqueName: \"kubernetes.io/projected/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-kube-api-access-6h5tn\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:56:00.862363 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:00.862359 4925 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:56:00.862547 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:00.862371 4925 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b-util\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:56:01.552924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:01.552852 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c" event={"ID":"4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b","Type":"ContainerDied","Data":"ffc8c7a263d6316f76ed8aba084f6e2a052565bec9317051425a16ba2f90d17c"} Oct 30 02:56:01.552924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:01.552906 4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffc8c7a263d6316f76ed8aba084f6e2a052565bec9317051425a16ba2f90d17c" Oct 30 02:56:01.552924 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:01.552857 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054qpr6c" Oct 30 02:56:06.725533 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725496 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-llspn"] Oct 30 02:56:06.725913 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725814 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f66de600-b77d-4ec2-b9ee-c83ed413a55e" containerName="pull" Oct 30 02:56:06.725913 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725826 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f66de600-b77d-4ec2-b9ee-c83ed413a55e" containerName="pull" Oct 30 02:56:06.725913 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725843 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f66de600-b77d-4ec2-b9ee-c83ed413a55e" containerName="util" Oct 30 02:56:06.725913 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725848 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f66de600-b77d-4ec2-b9ee-c83ed413a55e" containerName="util" Oct 30 02:56:06.725913 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725857 4925 cpu_manager.go:414] "containerMap: removing container" podUID="3e32d564-a869-4b11-ac31-bf1935b1b1eb" containerName="pull" Oct 30 02:56:06.725913 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725862 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e32d564-a869-4b11-ac31-bf1935b1b1eb" containerName="pull" Oct 30 02:56:06.725913 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725870 4925 cpu_manager.go:414] "containerMap: removing container" podUID="4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b" containerName="util" Oct 30 02:56:06.725913 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725875 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b" containerName="util" Oct 30 02:56:06.725913 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725884 4925 cpu_manager.go:414] "containerMap: removing container" podUID="3e32d564-a869-4b11-ac31-bf1935b1b1eb" containerName="util" Oct 30 02:56:06.725913 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725889 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e32d564-a869-4b11-ac31-bf1935b1b1eb" containerName="util" Oct 30 02:56:06.725913 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725895 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f66de600-b77d-4ec2-b9ee-c83ed413a55e" containerName="extract" Oct 30 02:56:06.725913 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725902 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f66de600-b77d-4ec2-b9ee-c83ed413a55e" containerName="extract" Oct 30 02:56:06.725913 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725911 4925 cpu_manager.go:414] "containerMap: removing container" podUID="4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b" containerName="pull" Oct 30 02:56:06.725913 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725915 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b" containerName="pull" Oct 30 02:56:06.726307 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725925 4925 cpu_manager.go:414] "containerMap: removing container" podUID="4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b" containerName="extract" Oct 30 02:56:06.726307 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725929 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b" containerName="extract" Oct 30 02:56:06.726307 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725936 4925 cpu_manager.go:414] "containerMap: removing container" podUID="3e32d564-a869-4b11-ac31-bf1935b1b1eb" containerName="extract" Oct 30 02:56:06.726307 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725940 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e32d564-a869-4b11-ac31-bf1935b1b1eb" containerName="extract" Oct 30 02:56:06.726307 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725985 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e20b7ae-cf5a-4f27-b3e0-4717ca9e7a9b" containerName="extract" Oct 30 02:56:06.726307 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.725992 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e32d564-a869-4b11-ac31-bf1935b1b1eb" containerName="extract" Oct 30 02:56:06.726307 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.726002 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f66de600-b77d-4ec2-b9ee-c83ed413a55e" containerName="extract" Oct 30 02:56:06.729029 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.729005 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-llspn" Oct 30 02:56:06.761391 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.761363 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:56:06.761536 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.761478 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"kube-root-ca.crt\"" Oct 30 02:56:06.767484 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.767466 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-metrics\"" Oct 30 02:56:06.767876 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.767858 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-controller-manager-dockercfg-hdzf5\"" Oct 30 02:56:06.791082 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.791047 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-llspn"] Oct 30 02:56:06.813826 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.813785 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/57a72806-5efa-42da-b9a6-e730b85c54e2-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-llspn\" (UID: \"57a72806-5efa-42da-b9a6-e730b85c54e2\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-llspn" Oct 30 02:56:06.813978 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.813845 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxfs2\" (UniqueName: \"kubernetes.io/projected/57a72806-5efa-42da-b9a6-e730b85c54e2-kube-api-access-wxfs2\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-llspn\" (UID: \"57a72806-5efa-42da-b9a6-e730b85c54e2\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-llspn" Oct 30 02:56:06.914540 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.914504 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/57a72806-5efa-42da-b9a6-e730b85c54e2-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-llspn\" (UID: \"57a72806-5efa-42da-b9a6-e730b85c54e2\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-llspn" Oct 30 02:56:06.914699 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.914561 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxfs2\" (UniqueName: \"kubernetes.io/projected/57a72806-5efa-42da-b9a6-e730b85c54e2-kube-api-access-wxfs2\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-llspn\" (UID: \"57a72806-5efa-42da-b9a6-e730b85c54e2\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-llspn" Oct 30 02:56:06.917015 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.916987 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/57a72806-5efa-42da-b9a6-e730b85c54e2-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-llspn\" (UID: \"57a72806-5efa-42da-b9a6-e730b85c54e2\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-llspn" Oct 30 02:56:06.971965 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:06.971927 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxfs2\" (UniqueName: \"kubernetes.io/projected/57a72806-5efa-42da-b9a6-e730b85c54e2-kube-api-access-wxfs2\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-llspn\" (UID: \"57a72806-5efa-42da-b9a6-e730b85c54e2\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-llspn" Oct 30 02:56:07.040662 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:07.040574 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-llspn" Oct 30 02:56:07.397142 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:07.397105 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-llspn"] Oct 30 02:56:07.399185 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:56:07.399162 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a72806_5efa_42da_b9a6_e730b85c54e2.slice/crio-ef1b16a02239756b4f1ae0710638b8f7f7e686832114d63d7f8efc3af63625e1 WatchSource:0}: Error finding container ef1b16a02239756b4f1ae0710638b8f7f7e686832114d63d7f8efc3af63625e1: Status 404 returned error can't find the container with id ef1b16a02239756b4f1ae0710638b8f7f7e686832114d63d7f8efc3af63625e1 Oct 30 02:56:07.580638 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:07.580600 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-llspn" event={"ID":"57a72806-5efa-42da-b9a6-e730b85c54e2","Type":"ContainerStarted","Data":"ef1b16a02239756b4f1ae0710638b8f7f7e686832114d63d7f8efc3af63625e1"} Oct 30 02:56:08.314471 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:08.314436 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-dzwzw"] Oct 30 02:56:08.318122 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:08.318099 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-dzwzw" Oct 30 02:56:08.344878 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:08.344836 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-webhook-service-cert\"" Oct 30 02:56:08.374633 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:08.374558 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-dzwzw"] Oct 30 02:56:08.427653 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:08.427622 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jx5r\" (UniqueName: \"kubernetes.io/projected/e0bd62ab-ff52-4312-b2d3-ccd6e308f2bd-kube-api-access-8jx5r\") pod \"opentelemetry-operator-webhook-dd54bd85d-dzwzw\" (UID: \"e0bd62ab-ff52-4312-b2d3-ccd6e308f2bd\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-dzwzw" Oct 30 02:56:08.427841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:08.427662 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0bd62ab-ff52-4312-b2d3-ccd6e308f2bd-apiservice-cert\") pod \"opentelemetry-operator-webhook-dd54bd85d-dzwzw\" (UID: \"e0bd62ab-ff52-4312-b2d3-ccd6e308f2bd\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-dzwzw" Oct 30 02:56:08.427841 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:08.427721 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0bd62ab-ff52-4312-b2d3-ccd6e308f2bd-webhook-cert\") pod \"opentelemetry-operator-webhook-dd54bd85d-dzwzw\" (UID: \"e0bd62ab-ff52-4312-b2d3-ccd6e308f2bd\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-dzwzw" Oct 30 02:56:08.529774 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:08.528788 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jx5r\" (UniqueName: \"kubernetes.io/projected/e0bd62ab-ff52-4312-b2d3-ccd6e308f2bd-kube-api-access-8jx5r\") pod \"opentelemetry-operator-webhook-dd54bd85d-dzwzw\" (UID: \"e0bd62ab-ff52-4312-b2d3-ccd6e308f2bd\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-dzwzw" Oct 30 02:56:08.529774 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:08.528839 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0bd62ab-ff52-4312-b2d3-ccd6e308f2bd-apiservice-cert\") pod \"opentelemetry-operator-webhook-dd54bd85d-dzwzw\" (UID: \"e0bd62ab-ff52-4312-b2d3-ccd6e308f2bd\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-dzwzw" Oct 30 02:56:08.529774 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:08.528901 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0bd62ab-ff52-4312-b2d3-ccd6e308f2bd-webhook-cert\") pod \"opentelemetry-operator-webhook-dd54bd85d-dzwzw\" (UID: \"e0bd62ab-ff52-4312-b2d3-ccd6e308f2bd\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-dzwzw" Oct 30 02:56:08.532458 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:08.532394 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0bd62ab-ff52-4312-b2d3-ccd6e308f2bd-webhook-cert\") pod \"opentelemetry-operator-webhook-dd54bd85d-dzwzw\" (UID: \"e0bd62ab-ff52-4312-b2d3-ccd6e308f2bd\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-dzwzw" Oct 30 02:56:08.532584 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:08.532526 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0bd62ab-ff52-4312-b2d3-ccd6e308f2bd-apiservice-cert\") pod \"opentelemetry-operator-webhook-dd54bd85d-dzwzw\" (UID: \"e0bd62ab-ff52-4312-b2d3-ccd6e308f2bd\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-dzwzw" Oct 30 02:56:08.554310 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:08.554283 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jx5r\" (UniqueName: \"kubernetes.io/projected/e0bd62ab-ff52-4312-b2d3-ccd6e308f2bd-kube-api-access-8jx5r\") pod \"opentelemetry-operator-webhook-dd54bd85d-dzwzw\" (UID: \"e0bd62ab-ff52-4312-b2d3-ccd6e308f2bd\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-dzwzw" Oct 30 02:56:08.633486 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:08.633451 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-dzwzw" Oct 30 02:56:08.982367 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:08.982340 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-dzwzw"] Oct 30 02:56:08.984536 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:56:08.984509 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0bd62ab_ff52_4312_b2d3_ccd6e308f2bd.slice/crio-d8a8534d8c080185bf00794d0da88641fc8be14a93cc853d7200622cc11d5b1d WatchSource:0}: Error finding container d8a8534d8c080185bf00794d0da88641fc8be14a93cc853d7200622cc11d5b1d: Status 404 returned error can't find the container with id d8a8534d8c080185bf00794d0da88641fc8be14a93cc853d7200622cc11d5b1d Oct 30 02:56:09.591709 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:09.591673 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-llspn" event={"ID":"57a72806-5efa-42da-b9a6-e730b85c54e2","Type":"ContainerStarted","Data":"fbb79caa7e71ffe52e5deaf196a80287702c3064d6305d381e85480aba871e90"} Oct 30 02:56:09.592125 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:09.591737 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-llspn" Oct 30 02:56:09.593258 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:09.593235 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-dzwzw" event={"ID":"e0bd62ab-ff52-4312-b2d3-ccd6e308f2bd","Type":"ContainerStarted","Data":"e65eaa7760ac4ca75be826d596cf74d9ad5fdc3ef78709216f7b427909116c8f"} Oct 30 02:56:09.593407 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:09.593277 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-dzwzw" event={"ID":"e0bd62ab-ff52-4312-b2d3-ccd6e308f2bd","Type":"ContainerStarted","Data":"d8a8534d8c080185bf00794d0da88641fc8be14a93cc853d7200622cc11d5b1d"} Oct 30 02:56:09.593407 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:09.593303 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-dzwzw" Oct 30 02:56:09.719921 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:09.719813 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-llspn" podStartSLOduration=2.065648304 podStartE2EDuration="3.719797717s" totalImagesPullingTime="1.654149413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:09.714061133 +0000 UTC m=+930.727339241" watchObservedRunningTime="2026-10-30 02:56:09.719797717 +0000 UTC m=+930.733075818" Oct 30 02:56:09.833827 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:09.833682 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-dzwzw" podStartSLOduration=1.390784322 podStartE2EDuration="1.833669226s" totalImagesPullingTime="442.884904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:09.826884316 +0000 UTC m=+930.840162432" watchObservedRunningTime="2026-10-30 02:56:09.833669226 +0000 UTC m=+930.846947328" Oct 30 02:56:14.078909 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.078870 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2"] Oct 30 02:56:14.082441 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.082424 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2" Oct 30 02:56:14.108011 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.107984 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2"] Oct 30 02:56:14.112605 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.112578 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-tempo-operator\"/\"webhook-server-cert\"" Oct 30 02:56:14.112740 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.112581 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-tempo-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:56:14.112844 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.112817 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-tempo-operator\"/\"tempo-operator-controller-service-cert\"" Oct 30 02:56:14.112914 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.112846 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-tempo-operator\"/\"tempo-operator-metrics\"" Oct 30 02:56:14.112970 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.112914 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-tempo-operator\"/\"tempo-operator-controller-manager-dockercfg-mt8wp\"" Oct 30 02:56:14.112970 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.112956 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-tempo-operator\"/\"kube-root-ca.crt\"" Oct 30 02:56:14.174355 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.174325 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70492548-684c-40cb-aa32-fac377915861-tempo-operator-metrics-cert\") pod \"tempo-operator-controller-76699c7b66-d9kj2\" (UID: \"70492548-684c-40cb-aa32-fac377915861\") " pod="openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2" Oct 30 02:56:14.174478 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.174374 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/70492548-684c-40cb-aa32-fac377915861-apiservice-cert\") pod \"tempo-operator-controller-76699c7b66-d9kj2\" (UID: \"70492548-684c-40cb-aa32-fac377915861\") " pod="openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2" Oct 30 02:56:14.174548 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.174514 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70492548-684c-40cb-aa32-fac377915861-webhook-cert\") pod \"tempo-operator-controller-76699c7b66-d9kj2\" (UID: \"70492548-684c-40cb-aa32-fac377915861\") " pod="openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2" Oct 30 02:56:14.174584 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.174564 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzwck\" (UniqueName: \"kubernetes.io/projected/70492548-684c-40cb-aa32-fac377915861-kube-api-access-hzwck\") pod \"tempo-operator-controller-76699c7b66-d9kj2\" (UID: \"70492548-684c-40cb-aa32-fac377915861\") " pod="openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2" Oct 30 02:56:14.275694 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.275655 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/70492548-684c-40cb-aa32-fac377915861-apiservice-cert\") pod \"tempo-operator-controller-76699c7b66-d9kj2\" (UID: \"70492548-684c-40cb-aa32-fac377915861\") " pod="openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2" Oct 30 02:56:14.275918 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.275724 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70492548-684c-40cb-aa32-fac377915861-webhook-cert\") pod \"tempo-operator-controller-76699c7b66-d9kj2\" (UID: \"70492548-684c-40cb-aa32-fac377915861\") " pod="openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2" Oct 30 02:56:14.275918 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.275746 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwck\" (UniqueName: \"kubernetes.io/projected/70492548-684c-40cb-aa32-fac377915861-kube-api-access-hzwck\") pod \"tempo-operator-controller-76699c7b66-d9kj2\" (UID: \"70492548-684c-40cb-aa32-fac377915861\") " pod="openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2" Oct 30 02:56:14.275918 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.275794 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70492548-684c-40cb-aa32-fac377915861-tempo-operator-metrics-cert\") pod \"tempo-operator-controller-76699c7b66-d9kj2\" (UID: \"70492548-684c-40cb-aa32-fac377915861\") " pod="openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2" Oct 30 02:56:14.278219 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.278190 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/70492548-684c-40cb-aa32-fac377915861-apiservice-cert\") pod \"tempo-operator-controller-76699c7b66-d9kj2\" (UID: \"70492548-684c-40cb-aa32-fac377915861\") " pod="openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2" Oct 30 02:56:14.278343 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.278230 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70492548-684c-40cb-aa32-fac377915861-tempo-operator-metrics-cert\") pod \"tempo-operator-controller-76699c7b66-d9kj2\" (UID: \"70492548-684c-40cb-aa32-fac377915861\") " pod="openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2" Oct 30 02:56:14.278383 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.278361 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70492548-684c-40cb-aa32-fac377915861-webhook-cert\") pod \"tempo-operator-controller-76699c7b66-d9kj2\" (UID: \"70492548-684c-40cb-aa32-fac377915861\") " pod="openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2" Oct 30 02:56:14.301522 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.301492 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwck\" (UniqueName: \"kubernetes.io/projected/70492548-684c-40cb-aa32-fac377915861-kube-api-access-hzwck\") pod \"tempo-operator-controller-76699c7b66-d9kj2\" (UID: \"70492548-684c-40cb-aa32-fac377915861\") " pod="openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2" Oct 30 02:56:14.394136 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.394039 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2" Oct 30 02:56:14.725832 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:14.725809 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2"] Oct 30 02:56:14.728298 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:56:14.728277 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70492548_684c_40cb_aa32_fac377915861.slice/crio-00fac1a3fa14cc53b4188e7459ae39b6eef5058cc6c341964a5fe8e0e7c7769b WatchSource:0}: Error finding container 00fac1a3fa14cc53b4188e7459ae39b6eef5058cc6c341964a5fe8e0e7c7769b: Status 404 returned error can't find the container with id 00fac1a3fa14cc53b4188e7459ae39b6eef5058cc6c341964a5fe8e0e7c7769b Oct 30 02:56:15.621406 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:15.621370 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2" event={"ID":"70492548-684c-40cb-aa32-fac377915861","Type":"ContainerStarted","Data":"00fac1a3fa14cc53b4188e7459ae39b6eef5058cc6c341964a5fe8e0e7c7769b"} Oct 30 02:56:17.632862 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:17.632821 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2" event={"ID":"70492548-684c-40cb-aa32-fac377915861","Type":"ContainerStarted","Data":"b90e3aec97d93943421feea2965e6efa7d0236517ad27edd6ee5157a9442ee22"} Oct 30 02:56:17.633294 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:17.633019 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2" Oct 30 02:56:17.722179 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:17.722121 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2" podStartSLOduration=1.408984156 podStartE2EDuration="3.722102562s" totalImagesPullingTime="2.313118406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:17.707940434 +0000 UTC m=+938.721218531" watchObservedRunningTime="2026-10-30 02:56:17.722102562 +0000 UTC m=+938.735380665" Oct 30 02:56:17.932801 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:17.932696 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-nz9tt"] Oct 30 02:56:17.936226 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:17.936203 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-nz9tt" Oct 30 02:56:17.955616 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:17.955586 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-admission-webhook-dockercfg-t48zq\"" Oct 30 02:56:17.955879 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:17.955864 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-admission-webhook-service-cert\"" Oct 30 02:56:17.986568 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:17.986536 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-nz9tt"] Oct 30 02:56:18.008314 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:18.008283 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/550e337d-ca82-47cf-b79b-ab63e506c3d9-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-7795c9f77c-nz9tt\" (UID: \"550e337d-ca82-47cf-b79b-ab63e506c3d9\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-nz9tt" Oct 30 02:56:18.008434 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:18.008336 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/550e337d-ca82-47cf-b79b-ab63e506c3d9-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-7795c9f77c-nz9tt\" (UID: \"550e337d-ca82-47cf-b79b-ab63e506c3d9\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-nz9tt" Oct 30 02:56:18.109023 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:18.108995 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/550e337d-ca82-47cf-b79b-ab63e506c3d9-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-7795c9f77c-nz9tt\" (UID: \"550e337d-ca82-47cf-b79b-ab63e506c3d9\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-nz9tt" Oct 30 02:56:18.109180 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:18.109057 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/550e337d-ca82-47cf-b79b-ab63e506c3d9-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-7795c9f77c-nz9tt\" (UID: \"550e337d-ca82-47cf-b79b-ab63e506c3d9\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-nz9tt" Oct 30 02:56:18.111388 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:18.111367 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/550e337d-ca82-47cf-b79b-ab63e506c3d9-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-7795c9f77c-nz9tt\" (UID: \"550e337d-ca82-47cf-b79b-ab63e506c3d9\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-nz9tt" Oct 30 02:56:18.111476 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:18.111433 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/550e337d-ca82-47cf-b79b-ab63e506c3d9-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-7795c9f77c-nz9tt\" (UID: \"550e337d-ca82-47cf-b79b-ab63e506c3d9\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-nz9tt" Oct 30 02:56:18.161674 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:56:18.161634 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:56:18.248878 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:18.248810 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-nz9tt" Oct 30 02:56:18.579504 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:18.579472 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-nz9tt"] Oct 30 02:56:18.580715 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:56:18.580684 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550e337d_ca82_47cf_b79b_ab63e506c3d9.slice/crio-de40f43789ed689844f375c856a9bf0a5456a11e2b527d2a9ab7a29c8bea7a4d WatchSource:0}: Error finding container de40f43789ed689844f375c856a9bf0a5456a11e2b527d2a9ab7a29c8bea7a4d: Status 404 returned error can't find the container with id de40f43789ed689844f375c856a9bf0a5456a11e2b527d2a9ab7a29c8bea7a4d Oct 30 02:56:18.645790 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:18.644966 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-nz9tt" event={"ID":"550e337d-ca82-47cf-b79b-ab63e506c3d9","Type":"ContainerStarted","Data":"de40f43789ed689844f375c856a9bf0a5456a11e2b527d2a9ab7a29c8bea7a4d"} Oct 30 02:56:20.602548 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:20.602516 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-llspn" Oct 30 02:56:20.603790 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:20.603461 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-dzwzw" Oct 30 02:56:21.664356 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:21.664317 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-nz9tt" event={"ID":"550e337d-ca82-47cf-b79b-ab63e506c3d9","Type":"ContainerStarted","Data":"774b6b675cba995ffb10be7614a8a6dba29f24c532d3971ba24369b79b05820c"} Oct 30 02:56:28.648868 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:28.648831 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-tempo-operator/tempo-operator-controller-76699c7b66-d9kj2" Oct 30 02:56:28.730145 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:28.730093 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7795c9f77c-nz9tt" podStartSLOduration=9.579760323 podStartE2EDuration="11.730078964s" totalImagesPullingTime="2.150318641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:21.738815132 +0000 UTC m=+942.752093234" watchObservedRunningTime="2026-10-30 02:56:28.730078964 +0000 UTC m=+949.743357067" Oct 30 02:56:45.747070 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:45.747032 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns"] Oct 30 02:56:45.752149 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:45.752128 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns" Oct 30 02:56:45.802162 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:45.802128 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns"] Oct 30 02:56:45.846216 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:45.846177 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7f3fff6-40ff-460c-8fa5-2327854bb23a-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-78c8655555-9r8ns\" (UID: \"f7f3fff6-40ff-460c-8fa5-2327854bb23a\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns" Oct 30 02:56:45.846385 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:45.846243 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzg65\" (UniqueName: \"kubernetes.io/projected/f7f3fff6-40ff-460c-8fa5-2327854bb23a-kube-api-access-wzg65\") pod \"opentelemetry-operator-controller-manager-78c8655555-9r8ns\" (UID: \"f7f3fff6-40ff-460c-8fa5-2327854bb23a\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns" Oct 30 02:56:45.947256 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:45.947214 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzg65\" (UniqueName: \"kubernetes.io/projected/f7f3fff6-40ff-460c-8fa5-2327854bb23a-kube-api-access-wzg65\") pod \"opentelemetry-operator-controller-manager-78c8655555-9r8ns\" (UID: \"f7f3fff6-40ff-460c-8fa5-2327854bb23a\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns" Oct 30 02:56:45.947432 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:45.947317 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7f3fff6-40ff-460c-8fa5-2327854bb23a-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-78c8655555-9r8ns\" (UID: \"f7f3fff6-40ff-460c-8fa5-2327854bb23a\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns" Oct 30 02:56:45.949921 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:45.949893 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7f3fff6-40ff-460c-8fa5-2327854bb23a-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-78c8655555-9r8ns\" (UID: \"f7f3fff6-40ff-460c-8fa5-2327854bb23a\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns" Oct 30 02:56:45.976280 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:45.976256 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzg65\" (UniqueName: \"kubernetes.io/projected/f7f3fff6-40ff-460c-8fa5-2327854bb23a-kube-api-access-wzg65\") pod \"opentelemetry-operator-controller-manager-78c8655555-9r8ns\" (UID: \"f7f3fff6-40ff-460c-8fa5-2327854bb23a\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns" Oct 30 02:56:46.063670 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:46.063578 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns" Oct 30 02:56:46.391806 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:46.391777 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns"] Oct 30 02:56:46.394560 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:56:46.394525 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f3fff6_40ff_460c_8fa5_2327854bb23a.slice/crio-5f8f93755f429b20212b87cc38781f6c14d16283c9c30a9022df90246762b467 WatchSource:0}: Error finding container 5f8f93755f429b20212b87cc38781f6c14d16283c9c30a9022df90246762b467: Status 404 returned error can't find the container with id 5f8f93755f429b20212b87cc38781f6c14d16283c9c30a9022df90246762b467 Oct 30 02:56:46.781054 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:46.780965 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns" event={"ID":"f7f3fff6-40ff-460c-8fa5-2327854bb23a","Type":"ContainerStarted","Data":"5c37e8ef523030bc00752d0a8322b4b5953edd79b3d88bcc797e10506b7ebd66"} Oct 30 02:56:46.781054 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:46.781023 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns" event={"ID":"f7f3fff6-40ff-460c-8fa5-2327854bb23a","Type":"ContainerStarted","Data":"5f8f93755f429b20212b87cc38781f6c14d16283c9c30a9022df90246762b467"} Oct 30 02:56:46.781465 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:46.781099 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns" Oct 30 02:56:47.135625 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:47.135574 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns" podStartSLOduration=2.135560311 podStartE2EDuration="2.135560311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:46.85134578 +0000 UTC m=+967.864623884" watchObservedRunningTime="2026-10-30 02:56:47.135560311 +0000 UTC m=+968.148838414" Oct 30 02:56:47.136637 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:47.136619 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt"] Oct 30 02:56:47.140460 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:47.140437 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt" Oct 30 02:56:47.164539 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:47.164509 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 02:56:47.164821 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:47.164799 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 02:56:47.166170 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:47.166148 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt"] Oct 30 02:56:47.168159 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:47.168138 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"d28b81ce69fab5e58d2cd73462dde4041bbc20-487d35f4-dockercfg-dkbqm\"" Oct 30 02:56:47.258352 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:47.258320 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv2bg\" (UniqueName: \"kubernetes.io/projected/e3257394-53c4-466f-a3a7-a6b650520d50-kube-api-access-hv2bg\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt\" (UID: \"e3257394-53c4-466f-a3a7-a6b650520d50\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt" Oct 30 02:56:47.258506 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:47.258390 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3257394-53c4-466f-a3a7-a6b650520d50-util\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt\" (UID: \"e3257394-53c4-466f-a3a7-a6b650520d50\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt" Oct 30 02:56:47.258506 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:47.258453 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3257394-53c4-466f-a3a7-a6b650520d50-bundle\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt\" (UID: \"e3257394-53c4-466f-a3a7-a6b650520d50\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt" Oct 30 02:56:47.358936 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:47.358905 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3257394-53c4-466f-a3a7-a6b650520d50-util\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt\" (UID: \"e3257394-53c4-466f-a3a7-a6b650520d50\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt" Oct 30 02:56:47.359135 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:47.358947 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3257394-53c4-466f-a3a7-a6b650520d50-bundle\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt\" (UID: \"e3257394-53c4-466f-a3a7-a6b650520d50\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt" Oct 30 02:56:47.359135 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:47.358976 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv2bg\" (UniqueName: \"kubernetes.io/projected/e3257394-53c4-466f-a3a7-a6b650520d50-kube-api-access-hv2bg\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt\" (UID: \"e3257394-53c4-466f-a3a7-a6b650520d50\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt" Oct 30 02:56:47.359363 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:47.359341 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3257394-53c4-466f-a3a7-a6b650520d50-bundle\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt\" (UID: \"e3257394-53c4-466f-a3a7-a6b650520d50\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt" Oct 30 02:56:47.359430 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:47.359363 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3257394-53c4-466f-a3a7-a6b650520d50-util\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt\" (UID: \"e3257394-53c4-466f-a3a7-a6b650520d50\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt" Oct 30 02:56:47.385568 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:47.385537 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv2bg\" (UniqueName: \"kubernetes.io/projected/e3257394-53c4-466f-a3a7-a6b650520d50-kube-api-access-hv2bg\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt\" (UID: \"e3257394-53c4-466f-a3a7-a6b650520d50\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt" Oct 30 02:56:47.402198 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:47.402130 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns"] Oct 30 02:56:47.456587 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:47.456559 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt" Oct 30 02:56:47.758108 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:47.758079 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt"] Oct 30 02:56:47.760333 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:56:47.760299 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3257394_53c4_466f_a3a7_a6b650520d50.slice/crio-a6f772c6abe9a9241b3a93ab728efc46167ee1ba9c149b030097ca575ae5f2e2 WatchSource:0}: Error finding container a6f772c6abe9a9241b3a93ab728efc46167ee1ba9c149b030097ca575ae5f2e2: Status 404 returned error can't find the container with id a6f772c6abe9a9241b3a93ab728efc46167ee1ba9c149b030097ca575ae5f2e2 Oct 30 02:56:47.787229 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:47.787199 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt" event={"ID":"e3257394-53c4-466f-a3a7-a6b650520d50","Type":"ContainerStarted","Data":"a6f772c6abe9a9241b3a93ab728efc46167ee1ba9c149b030097ca575ae5f2e2"} Oct 30 02:56:48.161691 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:56:48.161655 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:56:48.793220 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:48.793185 4925 generic.go:356] "Generic (PLEG): container finished" podID="e3257394-53c4-466f-a3a7-a6b650520d50" containerID="6d73f146b53a1829d59d17b49810ef0ca0e262eb5cbacf8554e0cfc0a5bcae69" exitCode=0 Oct 30 02:56:48.793674 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:48.793276 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt" event={"ID":"e3257394-53c4-466f-a3a7-a6b650520d50","Type":"ContainerDied","Data":"6d73f146b53a1829d59d17b49810ef0ca0e262eb5cbacf8554e0cfc0a5bcae69"} Oct 30 02:56:48.793674 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:48.793458 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns" podUID="f7f3fff6-40ff-460c-8fa5-2327854bb23a" containerName="manager" containerID="cri-o://5c37e8ef523030bc00752d0a8322b4b5953edd79b3d88bcc797e10506b7ebd66" gracePeriod=10 Oct 30 02:56:49.037206 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:49.037182 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns" Oct 30 02:56:49.173965 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:49.173931 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7f3fff6-40ff-460c-8fa5-2327854bb23a-opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7f3fff6-40ff-460c-8fa5-2327854bb23a-opentelemetry-operator-metrics-cert\") pod \"f7f3fff6-40ff-460c-8fa5-2327854bb23a\" (UID: \"f7f3fff6-40ff-460c-8fa5-2327854bb23a\") " Oct 30 02:56:49.174157 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:49.173981 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f3fff6-40ff-460c-8fa5-2327854bb23a-kube-api-access-wzg65\" (UniqueName: \"kubernetes.io/projected/f7f3fff6-40ff-460c-8fa5-2327854bb23a-kube-api-access-wzg65\") pod \"f7f3fff6-40ff-460c-8fa5-2327854bb23a\" (UID: \"f7f3fff6-40ff-460c-8fa5-2327854bb23a\") " Oct 30 02:56:49.176245 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:49.176216 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f3fff6-40ff-460c-8fa5-2327854bb23a-opentelemetry-operator-metrics-cert" pod "f7f3fff6-40ff-460c-8fa5-2327854bb23a" (UID: "f7f3fff6-40ff-460c-8fa5-2327854bb23a"). InnerVolumeSpecName "opentelemetry-operator-metrics-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:56:49.176349 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:49.176283 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f3fff6-40ff-460c-8fa5-2327854bb23a-kube-api-access-wzg65" pod "f7f3fff6-40ff-460c-8fa5-2327854bb23a" (UID: "f7f3fff6-40ff-460c-8fa5-2327854bb23a"). InnerVolumeSpecName "kube-api-access-wzg65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:56:49.274742 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:49.274707 4925 reconciler_common.go:299] "Volume detached for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7f3fff6-40ff-460c-8fa5-2327854bb23a-opentelemetry-operator-metrics-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:56:49.274742 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:49.274735 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzg65\" (UniqueName: \"kubernetes.io/projected/f7f3fff6-40ff-460c-8fa5-2327854bb23a-kube-api-access-wzg65\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:56:49.799347 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:49.799314 4925 generic.go:356] "Generic (PLEG): container finished" podID="f7f3fff6-40ff-460c-8fa5-2327854bb23a" containerID="5c37e8ef523030bc00752d0a8322b4b5953edd79b3d88bcc797e10506b7ebd66" exitCode=0 Oct 30 02:56:49.799347 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:49.799352 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns" event={"ID":"f7f3fff6-40ff-460c-8fa5-2327854bb23a","Type":"ContainerDied","Data":"5c37e8ef523030bc00752d0a8322b4b5953edd79b3d88bcc797e10506b7ebd66"} Oct 30 02:56:49.799946 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:49.799356 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns" Oct 30 02:56:49.799946 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:49.799417 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns" event={"ID":"f7f3fff6-40ff-460c-8fa5-2327854bb23a","Type":"ContainerDied","Data":"5f8f93755f429b20212b87cc38781f6c14d16283c9c30a9022df90246762b467"} Oct 30 02:56:49.799946 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:49.799438 4925 scope.go:122] "RemoveContainer" containerID="5c37e8ef523030bc00752d0a8322b4b5953edd79b3d88bcc797e10506b7ebd66" Oct 30 02:56:49.810084 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:49.810056 4925 scope.go:122] "RemoveContainer" containerID="5c37e8ef523030bc00752d0a8322b4b5953edd79b3d88bcc797e10506b7ebd66" Oct 30 02:56:49.810379 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:56:49.810355 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c37e8ef523030bc00752d0a8322b4b5953edd79b3d88bcc797e10506b7ebd66\": container with ID starting with 5c37e8ef523030bc00752d0a8322b4b5953edd79b3d88bcc797e10506b7ebd66 not found: ID does not exist" containerID="5c37e8ef523030bc00752d0a8322b4b5953edd79b3d88bcc797e10506b7ebd66" Oct 30 02:56:49.810446 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:49.810387 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c37e8ef523030bc00752d0a8322b4b5953edd79b3d88bcc797e10506b7ebd66"} err="failed to get container status \"5c37e8ef523030bc00752d0a8322b4b5953edd79b3d88bcc797e10506b7ebd66\": rpc error: code = NotFound desc = could not find container \"5c37e8ef523030bc00752d0a8322b4b5953edd79b3d88bcc797e10506b7ebd66\": container with ID starting with 5c37e8ef523030bc00752d0a8322b4b5953edd79b3d88bcc797e10506b7ebd66 not found: ID does not exist" Oct 30 02:56:49.864452 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:49.864421 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns"] Oct 30 02:56:49.877265 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:49.877236 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-78c8655555-9r8ns"] Oct 30 02:56:50.808112 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:50.808073 4925 generic.go:356] "Generic (PLEG): container finished" podID="e3257394-53c4-466f-a3a7-a6b650520d50" containerID="f33d4d155941fe1f3105a16c3e1ee923e6cb7984084a7edbd0490a9f6edabd4c" exitCode=0 Oct 30 02:56:50.808501 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:50.808156 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt" event={"ID":"e3257394-53c4-466f-a3a7-a6b650520d50","Type":"ContainerDied","Data":"f33d4d155941fe1f3105a16c3e1ee923e6cb7984084a7edbd0490a9f6edabd4c"} Oct 30 02:56:51.119656 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:51.119615 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f3fff6-40ff-460c-8fa5-2327854bb23a" path="/var/lib/kubelet/pods/f7f3fff6-40ff-460c-8fa5-2327854bb23a/volumes" Oct 30 02:56:51.815745 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:51.815707 4925 generic.go:356] "Generic (PLEG): container finished" podID="e3257394-53c4-466f-a3a7-a6b650520d50" containerID="5e791c1ed33d378c0621da8d3f47e8cfc6adb1b472c6b0119959f7426160664c" exitCode=0 Oct 30 02:56:51.816190 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:51.815789 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt" event={"ID":"e3257394-53c4-466f-a3a7-a6b650520d50","Type":"ContainerDied","Data":"5e791c1ed33d378c0621da8d3f47e8cfc6adb1b472c6b0119959f7426160664c"} Oct 30 02:56:52.949119 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:52.949093 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt" Oct 30 02:56:53.108212 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:53.108184 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3257394-53c4-466f-a3a7-a6b650520d50-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3257394-53c4-466f-a3a7-a6b650520d50-bundle\") pod \"e3257394-53c4-466f-a3a7-a6b650520d50\" (UID: \"e3257394-53c4-466f-a3a7-a6b650520d50\") " Oct 30 02:56:53.108379 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:53.108263 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3257394-53c4-466f-a3a7-a6b650520d50-kube-api-access-hv2bg\" (UniqueName: \"kubernetes.io/projected/e3257394-53c4-466f-a3a7-a6b650520d50-kube-api-access-hv2bg\") pod \"e3257394-53c4-466f-a3a7-a6b650520d50\" (UID: \"e3257394-53c4-466f-a3a7-a6b650520d50\") " Oct 30 02:56:53.108379 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:53.108281 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3257394-53c4-466f-a3a7-a6b650520d50-util\" (UniqueName: \"kubernetes.io/empty-dir/e3257394-53c4-466f-a3a7-a6b650520d50-util\") pod \"e3257394-53c4-466f-a3a7-a6b650520d50\" (UID: \"e3257394-53c4-466f-a3a7-a6b650520d50\") " Oct 30 02:56:53.108789 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:53.108735 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3257394-53c4-466f-a3a7-a6b650520d50-bundle" pod "e3257394-53c4-466f-a3a7-a6b650520d50" (UID: "e3257394-53c4-466f-a3a7-a6b650520d50"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:56:53.110370 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:53.110347 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3257394-53c4-466f-a3a7-a6b650520d50-kube-api-access-hv2bg" pod "e3257394-53c4-466f-a3a7-a6b650520d50" (UID: "e3257394-53c4-466f-a3a7-a6b650520d50"). InnerVolumeSpecName "kube-api-access-hv2bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:56:53.161344 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:53.161312 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3257394-53c4-466f-a3a7-a6b650520d50-util" pod "e3257394-53c4-466f-a3a7-a6b650520d50" (UID: "e3257394-53c4-466f-a3a7-a6b650520d50"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:56:53.212884 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:53.212852 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv2bg\" (UniqueName: \"kubernetes.io/projected/e3257394-53c4-466f-a3a7-a6b650520d50-kube-api-access-hv2bg\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:56:53.212884 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:53.212883 4925 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3257394-53c4-466f-a3a7-a6b650520d50-util\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:56:53.213056 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:53.212896 4925 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3257394-53c4-466f-a3a7-a6b650520d50-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:56:53.827146 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:53.827104 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt" event={"ID":"e3257394-53c4-466f-a3a7-a6b650520d50","Type":"ContainerDied","Data":"a6f772c6abe9a9241b3a93ab728efc46167ee1ba9c149b030097ca575ae5f2e2"} Oct 30 02:56:53.827146 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:53.827134 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadvm9rt" Oct 30 02:56:53.827348 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:56:53.827193 4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6f772c6abe9a9241b3a93ab728efc46167ee1ba9c149b030097ca575ae5f2e2" Oct 30 02:57:18.161974 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:57:18.161936 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:57:22.694189 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.694154 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-xx96s"] Oct 30 02:57:22.694600 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.694485 4925 cpu_manager.go:414] "containerMap: removing container" podUID="e3257394-53c4-466f-a3a7-a6b650520d50" containerName="util" Oct 30 02:57:22.694600 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.694501 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3257394-53c4-466f-a3a7-a6b650520d50" containerName="util" Oct 30 02:57:22.694600 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.694515 4925 cpu_manager.go:414] "containerMap: removing container" podUID="e3257394-53c4-466f-a3a7-a6b650520d50" containerName="pull" Oct 30 02:57:22.694600 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.694521 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3257394-53c4-466f-a3a7-a6b650520d50" containerName="pull" Oct 30 02:57:22.694600 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.694528 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f7f3fff6-40ff-460c-8fa5-2327854bb23a" containerName="manager" Oct 30 02:57:22.694600 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.694533 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f3fff6-40ff-460c-8fa5-2327854bb23a" containerName="manager" Oct 30 02:57:22.694600 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.694549 4925 cpu_manager.go:414] "containerMap: removing container" podUID="e3257394-53c4-466f-a3a7-a6b650520d50" containerName="extract" Oct 30 02:57:22.694600 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.694554 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3257394-53c4-466f-a3a7-a6b650520d50" containerName="extract" Oct 30 02:57:22.694877 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.694614 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f3fff6-40ff-460c-8fa5-2327854bb23a" containerName="manager" Oct 30 02:57:22.694877 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.694629 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3257394-53c4-466f-a3a7-a6b650520d50" containerName="extract" Oct 30 02:57:22.698570 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.698550 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" Oct 30 02:57:22.718859 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.718829 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-xx96s"] Oct 30 02:57:22.729707 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.729674 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 02:57:22.729707 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.729703 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-6wgd2\"" Oct 30 02:57:22.729910 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.729834 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 02:57:22.729910 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.729844 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 02:57:22.729910 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.729898 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 02:57:22.736329 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.736297 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2t7m\" (UniqueName: \"kubernetes.io/projected/56a7bf6b-ca85-45b1-9a22-4611f680d376-kube-api-access-n2t7m\") pod \"korrel8r-6644769df5-xx96s\" (UID: \"56a7bf6b-ca85-45b1-9a22-4611f680d376\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" Oct 30 02:57:22.736417 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.736339 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/56a7bf6b-ca85-45b1-9a22-4611f680d376-tls\") pod \"korrel8r-6644769df5-xx96s\" (UID: \"56a7bf6b-ca85-45b1-9a22-4611f680d376\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" Oct 30 02:57:22.736417 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.736362 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56a7bf6b-ca85-45b1-9a22-4611f680d376-config\") pod \"korrel8r-6644769df5-xx96s\" (UID: \"56a7bf6b-ca85-45b1-9a22-4611f680d376\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" Oct 30 02:57:22.837309 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.837264 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2t7m\" (UniqueName: \"kubernetes.io/projected/56a7bf6b-ca85-45b1-9a22-4611f680d376-kube-api-access-n2t7m\") pod \"korrel8r-6644769df5-xx96s\" (UID: \"56a7bf6b-ca85-45b1-9a22-4611f680d376\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" Oct 30 02:57:22.837309 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.837309 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/56a7bf6b-ca85-45b1-9a22-4611f680d376-tls\") pod \"korrel8r-6644769df5-xx96s\" (UID: \"56a7bf6b-ca85-45b1-9a22-4611f680d376\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" Oct 30 02:57:22.837539 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.837359 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56a7bf6b-ca85-45b1-9a22-4611f680d376-config\") pod \"korrel8r-6644769df5-xx96s\" (UID: \"56a7bf6b-ca85-45b1-9a22-4611f680d376\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" Oct 30 02:57:22.838226 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.838204 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56a7bf6b-ca85-45b1-9a22-4611f680d376-config\") pod \"korrel8r-6644769df5-xx96s\" (UID: \"56a7bf6b-ca85-45b1-9a22-4611f680d376\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" Oct 30 02:57:22.839700 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.839682 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/56a7bf6b-ca85-45b1-9a22-4611f680d376-tls\") pod \"korrel8r-6644769df5-xx96s\" (UID: \"56a7bf6b-ca85-45b1-9a22-4611f680d376\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" Oct 30 02:57:22.862235 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:22.862200 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2t7m\" (UniqueName: \"kubernetes.io/projected/56a7bf6b-ca85-45b1-9a22-4611f680d376-kube-api-access-n2t7m\") pod \"korrel8r-6644769df5-xx96s\" (UID: \"56a7bf6b-ca85-45b1-9a22-4611f680d376\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" Oct 30 02:57:23.011746 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:23.011659 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" Oct 30 02:57:23.313108 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:23.313082 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-xx96s"] Oct 30 02:57:23.315554 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:57:23.315522 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a7bf6b_ca85_45b1_9a22_4611f680d376.slice/crio-2ff6b20e3bcb4fc37d34acdfdec2714e7b734eef8ba6421740bc89b879701794 WatchSource:0}: Error finding container 2ff6b20e3bcb4fc37d34acdfdec2714e7b734eef8ba6421740bc89b879701794: Status 404 returned error can't find the container with id 2ff6b20e3bcb4fc37d34acdfdec2714e7b734eef8ba6421740bc89b879701794 Oct 30 02:57:23.638477 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:23.638445 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 02:57:23.643212 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:23.643182 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:23.666332 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:23.666300 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-qm2pj\"" Oct 30 02:57:23.666332 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:23.666308 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 02:57:23.673426 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:23.673400 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 02:57:23.712200 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:23.712175 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-645168e1\"" Oct 30 02:57:23.745651 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:23.745621 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9379d931-3f51-4133-8f29-2bca67714232-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9379d931-3f51-4133-8f29-2bca67714232\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:23.745823 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:23.745668 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9379d931-3f51-4133-8f29-2bca67714232-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9379d931-3f51-4133-8f29-2bca67714232\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:23.745823 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:23.745812 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88qg8\" (UniqueName: \"kubernetes.io/projected/9379d931-3f51-4133-8f29-2bca67714232-kube-api-access-88qg8\") pod \"data-science-collector-collector-0\" (UID: \"9379d931-3f51-4133-8f29-2bca67714232\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:23.846423 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:23.846382 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88qg8\" (UniqueName: \"kubernetes.io/projected/9379d931-3f51-4133-8f29-2bca67714232-kube-api-access-88qg8\") pod \"data-science-collector-collector-0\" (UID: \"9379d931-3f51-4133-8f29-2bca67714232\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:23.846600 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:23.846442 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9379d931-3f51-4133-8f29-2bca67714232-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9379d931-3f51-4133-8f29-2bca67714232\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:23.846600 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:23.846510 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9379d931-3f51-4133-8f29-2bca67714232-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9379d931-3f51-4133-8f29-2bca67714232\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:23.847537 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:23.847507 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9379d931-3f51-4133-8f29-2bca67714232-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9379d931-3f51-4133-8f29-2bca67714232\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:23.848984 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:23.848960 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9379d931-3f51-4133-8f29-2bca67714232-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9379d931-3f51-4133-8f29-2bca67714232\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:23.870534 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:23.870504 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88qg8\" (UniqueName: \"kubernetes.io/projected/9379d931-3f51-4133-8f29-2bca67714232-kube-api-access-88qg8\") pod \"data-science-collector-collector-0\" (UID: \"9379d931-3f51-4133-8f29-2bca67714232\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:23.958390 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:23.958313 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:23.971247 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:23.971011 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" event={"ID":"56a7bf6b-ca85-45b1-9a22-4611f680d376","Type":"ContainerStarted","Data":"2ff6b20e3bcb4fc37d34acdfdec2714e7b734eef8ba6421740bc89b879701794"} Oct 30 02:57:24.271260 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:24.271225 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 02:57:24.273253 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:57:24.273217 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9379d931_3f51_4133_8f29_2bca67714232.slice/crio-3ab51693928d75a1d70c2dc0d97a6f462db979a0f14d82f824bbc0c2242b35cf WatchSource:0}: Error finding container 3ab51693928d75a1d70c2dc0d97a6f462db979a0f14d82f824bbc0c2242b35cf: Status 404 returned error can't find the container with id 3ab51693928d75a1d70c2dc0d97a6f462db979a0f14d82f824bbc0c2242b35cf Oct 30 02:57:24.977774 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:24.977716 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9379d931-3f51-4133-8f29-2bca67714232","Type":"ContainerStarted","Data":"3ab51693928d75a1d70c2dc0d97a6f462db979a0f14d82f824bbc0c2242b35cf"} Oct 30 02:57:25.984741 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:25.984689 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" event={"ID":"56a7bf6b-ca85-45b1-9a22-4611f680d376","Type":"ContainerStarted","Data":"0296fd881ef1e56294d48aa562f7020d71513dd34bb1ffb90dace792310c1fb8"} Oct 30 02:57:25.985218 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:25.984821 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" Oct 30 02:57:26.028608 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:26.028556 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" podStartSLOduration=2.007197294 podStartE2EDuration="4.028543008s" totalImagesPullingTime="2.021345714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:26.027493963 +0000 UTC m=+1007.040772067" watchObservedRunningTime="2026-10-30 02:57:26.028543008 +0000 UTC m=+1007.041821110" Oct 30 02:57:28.369291 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:28.369251 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:57:28.369774 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:28.369328 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:57:29.001565 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:29.001525 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9379d931-3f51-4133-8f29-2bca67714232","Type":"ContainerStarted","Data":"c40f5af1d50544f5b84f414cfbf5d5d0292da962303f7c1d5448c5c0cd17fac1"} Oct 30 02:57:29.046091 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:29.046038 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.953896659 podStartE2EDuration="6.046021983s" totalImagesPullingTime="4.092125324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:29.044108672 +0000 UTC m=+1010.057386785" watchObservedRunningTime="2026-10-30 02:57:29.046021983 +0000 UTC m=+1010.059300087" Oct 30 02:57:36.992655 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:36.992623 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" Oct 30 02:57:40.574581 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:40.574542 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-xx96s"] Oct 30 02:57:40.575031 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:40.574730 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" podUID="56a7bf6b-ca85-45b1-9a22-4611f680d376" containerName="korrel8r" containerID="cri-o://0296fd881ef1e56294d48aa562f7020d71513dd34bb1ffb90dace792310c1fb8" gracePeriod=30 Oct 30 02:57:40.817763 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:40.817730 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" Oct 30 02:57:40.886928 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:40.886896 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a7bf6b-ca85-45b1-9a22-4611f680d376-kube-api-access-n2t7m\" (UniqueName: \"kubernetes.io/projected/56a7bf6b-ca85-45b1-9a22-4611f680d376-kube-api-access-n2t7m\") pod \"56a7bf6b-ca85-45b1-9a22-4611f680d376\" (UID: \"56a7bf6b-ca85-45b1-9a22-4611f680d376\") " Oct 30 02:57:40.887119 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:40.886996 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a7bf6b-ca85-45b1-9a22-4611f680d376-tls\" (UniqueName: \"kubernetes.io/secret/56a7bf6b-ca85-45b1-9a22-4611f680d376-tls\") pod \"56a7bf6b-ca85-45b1-9a22-4611f680d376\" (UID: \"56a7bf6b-ca85-45b1-9a22-4611f680d376\") " Oct 30 02:57:40.887119 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:40.887085 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a7bf6b-ca85-45b1-9a22-4611f680d376-config\" (UniqueName: \"kubernetes.io/configmap/56a7bf6b-ca85-45b1-9a22-4611f680d376-config\") pod \"56a7bf6b-ca85-45b1-9a22-4611f680d376\" (UID: \"56a7bf6b-ca85-45b1-9a22-4611f680d376\") " Oct 30 02:57:40.887483 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:40.887451 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a7bf6b-ca85-45b1-9a22-4611f680d376-config" pod "56a7bf6b-ca85-45b1-9a22-4611f680d376" (UID: "56a7bf6b-ca85-45b1-9a22-4611f680d376"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:57:40.889258 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:40.889228 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a7bf6b-ca85-45b1-9a22-4611f680d376-kube-api-access-n2t7m" pod "56a7bf6b-ca85-45b1-9a22-4611f680d376" (UID: "56a7bf6b-ca85-45b1-9a22-4611f680d376"). InnerVolumeSpecName "kube-api-access-n2t7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:57:40.889385 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:40.889300 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a7bf6b-ca85-45b1-9a22-4611f680d376-tls" pod "56a7bf6b-ca85-45b1-9a22-4611f680d376" (UID: "56a7bf6b-ca85-45b1-9a22-4611f680d376"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:57:40.988026 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:40.987987 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/56a7bf6b-ca85-45b1-9a22-4611f680d376-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:57:40.988026 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:40.988026 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56a7bf6b-ca85-45b1-9a22-4611f680d376-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:57:40.988278 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:40.988040 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2t7m\" (UniqueName: \"kubernetes.io/projected/56a7bf6b-ca85-45b1-9a22-4611f680d376-kube-api-access-n2t7m\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:57:41.008097 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.008057 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 02:57:41.008357 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.008328 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="9379d931-3f51-4133-8f29-2bca67714232" containerName="otc-container" containerID="cri-o://c40f5af1d50544f5b84f414cfbf5d5d0292da962303f7c1d5448c5c0cd17fac1" gracePeriod=30 Oct 30 02:57:41.058931 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.058891 4925 generic.go:356] "Generic (PLEG): container finished" podID="56a7bf6b-ca85-45b1-9a22-4611f680d376" containerID="0296fd881ef1e56294d48aa562f7020d71513dd34bb1ffb90dace792310c1fb8" exitCode=0 Oct 30 02:57:41.059123 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.058933 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" Oct 30 02:57:41.059123 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.058968 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" event={"ID":"56a7bf6b-ca85-45b1-9a22-4611f680d376","Type":"ContainerDied","Data":"0296fd881ef1e56294d48aa562f7020d71513dd34bb1ffb90dace792310c1fb8"} Oct 30 02:57:41.059123 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.059026 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-xx96s" event={"ID":"56a7bf6b-ca85-45b1-9a22-4611f680d376","Type":"ContainerDied","Data":"2ff6b20e3bcb4fc37d34acdfdec2714e7b734eef8ba6421740bc89b879701794"} Oct 30 02:57:41.059123 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.059051 4925 scope.go:122] "RemoveContainer" containerID="0296fd881ef1e56294d48aa562f7020d71513dd34bb1ffb90dace792310c1fb8" Oct 30 02:57:41.121598 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.121577 4925 scope.go:122] "RemoveContainer" containerID="0296fd881ef1e56294d48aa562f7020d71513dd34bb1ffb90dace792310c1fb8" Oct 30 02:57:41.121916 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:57:41.121891 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0296fd881ef1e56294d48aa562f7020d71513dd34bb1ffb90dace792310c1fb8\": container with ID starting with 0296fd881ef1e56294d48aa562f7020d71513dd34bb1ffb90dace792310c1fb8 not found: ID does not exist" containerID="0296fd881ef1e56294d48aa562f7020d71513dd34bb1ffb90dace792310c1fb8" Oct 30 02:57:41.121983 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.121925 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0296fd881ef1e56294d48aa562f7020d71513dd34bb1ffb90dace792310c1fb8"} err="failed to get container status \"0296fd881ef1e56294d48aa562f7020d71513dd34bb1ffb90dace792310c1fb8\": rpc error: code = NotFound desc = could not find container \"0296fd881ef1e56294d48aa562f7020d71513dd34bb1ffb90dace792310c1fb8\": container with ID starting with 0296fd881ef1e56294d48aa562f7020d71513dd34bb1ffb90dace792310c1fb8 not found: ID does not exist" Oct 30 02:57:41.122071 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.122050 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-xx96s"] Oct 30 02:57:41.122106 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.122098 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-xx96s"] Oct 30 02:57:41.261468 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.261438 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:41.391427 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.391390 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9379d931-3f51-4133-8f29-2bca67714232-kube-api-access-88qg8\" (UniqueName: \"kubernetes.io/projected/9379d931-3f51-4133-8f29-2bca67714232-kube-api-access-88qg8\") pod \"9379d931-3f51-4133-8f29-2bca67714232\" (UID: \"9379d931-3f51-4133-8f29-2bca67714232\") " Oct 30 02:57:41.391618 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.391463 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9379d931-3f51-4133-8f29-2bca67714232-otc-internal\" (UniqueName: \"kubernetes.io/configmap/9379d931-3f51-4133-8f29-2bca67714232-otc-internal\") pod \"9379d931-3f51-4133-8f29-2bca67714232\" (UID: \"9379d931-3f51-4133-8f29-2bca67714232\") " Oct 30 02:57:41.391618 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.391482 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9379d931-3f51-4133-8f29-2bca67714232-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9379d931-3f51-4133-8f29-2bca67714232-monitoring-tls-certs\") pod \"9379d931-3f51-4133-8f29-2bca67714232\" (UID: \"9379d931-3f51-4133-8f29-2bca67714232\") " Oct 30 02:57:41.391879 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.391853 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9379d931-3f51-4133-8f29-2bca67714232-otc-internal" pod "9379d931-3f51-4133-8f29-2bca67714232" (UID: "9379d931-3f51-4133-8f29-2bca67714232"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:57:41.393593 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.393569 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9379d931-3f51-4133-8f29-2bca67714232-monitoring-tls-certs" pod "9379d931-3f51-4133-8f29-2bca67714232" (UID: "9379d931-3f51-4133-8f29-2bca67714232"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:57:41.393698 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.393569 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9379d931-3f51-4133-8f29-2bca67714232-kube-api-access-88qg8" pod "9379d931-3f51-4133-8f29-2bca67714232" (UID: "9379d931-3f51-4133-8f29-2bca67714232"). InnerVolumeSpecName "kube-api-access-88qg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:57:41.492642 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.492553 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88qg8\" (UniqueName: \"kubernetes.io/projected/9379d931-3f51-4133-8f29-2bca67714232-kube-api-access-88qg8\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:57:41.492642 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.492584 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9379d931-3f51-4133-8f29-2bca67714232-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:57:41.492642 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:41.492593 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9379d931-3f51-4133-8f29-2bca67714232-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 02:57:42.065692 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:42.065655 4925 generic.go:356] "Generic (PLEG): container finished" podID="9379d931-3f51-4133-8f29-2bca67714232" containerID="c40f5af1d50544f5b84f414cfbf5d5d0292da962303f7c1d5448c5c0cd17fac1" exitCode=0 Oct 30 02:57:42.066199 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:42.065725 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:42.066199 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:42.065736 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9379d931-3f51-4133-8f29-2bca67714232","Type":"ContainerDied","Data":"c40f5af1d50544f5b84f414cfbf5d5d0292da962303f7c1d5448c5c0cd17fac1"} Oct 30 02:57:42.066199 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:42.065826 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9379d931-3f51-4133-8f29-2bca67714232","Type":"ContainerDied","Data":"3ab51693928d75a1d70c2dc0d97a6f462db979a0f14d82f824bbc0c2242b35cf"} Oct 30 02:57:42.066199 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:42.065848 4925 scope.go:122] "RemoveContainer" containerID="c40f5af1d50544f5b84f414cfbf5d5d0292da962303f7c1d5448c5c0cd17fac1" Oct 30 02:57:42.074581 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:42.074560 4925 scope.go:122] "RemoveContainer" containerID="c40f5af1d50544f5b84f414cfbf5d5d0292da962303f7c1d5448c5c0cd17fac1" Oct 30 02:57:42.074905 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:57:42.074883 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c40f5af1d50544f5b84f414cfbf5d5d0292da962303f7c1d5448c5c0cd17fac1\": container with ID starting with c40f5af1d50544f5b84f414cfbf5d5d0292da962303f7c1d5448c5c0cd17fac1 not found: ID does not exist" containerID="c40f5af1d50544f5b84f414cfbf5d5d0292da962303f7c1d5448c5c0cd17fac1" Oct 30 02:57:42.074976 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:42.074915 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c40f5af1d50544f5b84f414cfbf5d5d0292da962303f7c1d5448c5c0cd17fac1"} err="failed to get container status \"c40f5af1d50544f5b84f414cfbf5d5d0292da962303f7c1d5448c5c0cd17fac1\": rpc error: code = NotFound desc = could not find container \"c40f5af1d50544f5b84f414cfbf5d5d0292da962303f7c1d5448c5c0cd17fac1\": container with ID starting with c40f5af1d50544f5b84f414cfbf5d5d0292da962303f7c1d5448c5c0cd17fac1 not found: ID does not exist" Oct 30 02:57:42.118611 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:42.118575 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 02:57:42.131256 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:42.131223 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 02:57:43.119789 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:43.119736 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a7bf6b-ca85-45b1-9a22-4611f680d376" path="/var/lib/kubelet/pods/56a7bf6b-ca85-45b1-9a22-4611f680d376/volumes" Oct 30 02:57:43.120164 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:43.120148 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9379d931-3f51-4133-8f29-2bca67714232" path="/var/lib/kubelet/pods/9379d931-3f51-4133-8f29-2bca67714232/volumes" Oct 30 02:57:46.160158 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.160120 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq"] Oct 30 02:57:46.160545 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.160460 4925 cpu_manager.go:414] "containerMap: removing container" podUID="56a7bf6b-ca85-45b1-9a22-4611f680d376" containerName="korrel8r" Oct 30 02:57:46.160545 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.160470 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a7bf6b-ca85-45b1-9a22-4611f680d376" containerName="korrel8r" Oct 30 02:57:46.160545 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.160493 4925 cpu_manager.go:414] "containerMap: removing container" podUID="9379d931-3f51-4133-8f29-2bca67714232" containerName="otc-container" Oct 30 02:57:46.160545 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.160498 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9379d931-3f51-4133-8f29-2bca67714232" containerName="otc-container" Oct 30 02:57:46.160669 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.160562 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a7bf6b-ca85-45b1-9a22-4611f680d376" containerName="korrel8r" Oct 30 02:57:46.160669 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.160574 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9379d931-3f51-4133-8f29-2bca67714232" containerName="otc-container" Oct 30 02:57:46.164884 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.164861 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" Oct 30 02:57:46.194692 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.194659 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 02:57:46.194692 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.194699 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 02:57:46.194922 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.194698 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 02:57:46.194922 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.194796 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-wf6pr\"" Oct 30 02:57:46.194922 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.194866 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 02:57:46.196194 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.196174 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq"] Oct 30 02:57:46.235146 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.235110 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p67h\" (UniqueName: \"kubernetes.io/projected/acfec3e0-9e55-4f88-b724-6c7db0176d10-kube-api-access-5p67h\") pod \"korrel8r-84f5b7859-vjmzq\" (UID: \"acfec3e0-9e55-4f88-b724-6c7db0176d10\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" Oct 30 02:57:46.235310 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.235162 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/acfec3e0-9e55-4f88-b724-6c7db0176d10-tls\") pod \"korrel8r-84f5b7859-vjmzq\" (UID: \"acfec3e0-9e55-4f88-b724-6c7db0176d10\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" Oct 30 02:57:46.235310 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.235182 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acfec3e0-9e55-4f88-b724-6c7db0176d10-config\") pod \"korrel8r-84f5b7859-vjmzq\" (UID: \"acfec3e0-9e55-4f88-b724-6c7db0176d10\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" Oct 30 02:57:46.335902 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.335861 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p67h\" (UniqueName: \"kubernetes.io/projected/acfec3e0-9e55-4f88-b724-6c7db0176d10-kube-api-access-5p67h\") pod \"korrel8r-84f5b7859-vjmzq\" (UID: \"acfec3e0-9e55-4f88-b724-6c7db0176d10\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" Oct 30 02:57:46.336080 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.335918 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/acfec3e0-9e55-4f88-b724-6c7db0176d10-tls\") pod \"korrel8r-84f5b7859-vjmzq\" (UID: \"acfec3e0-9e55-4f88-b724-6c7db0176d10\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" Oct 30 02:57:46.336080 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.335939 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acfec3e0-9e55-4f88-b724-6c7db0176d10-config\") pod \"korrel8r-84f5b7859-vjmzq\" (UID: \"acfec3e0-9e55-4f88-b724-6c7db0176d10\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" Oct 30 02:57:46.336627 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.336608 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acfec3e0-9e55-4f88-b724-6c7db0176d10-config\") pod \"korrel8r-84f5b7859-vjmzq\" (UID: \"acfec3e0-9e55-4f88-b724-6c7db0176d10\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" Oct 30 02:57:46.338431 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.338412 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/acfec3e0-9e55-4f88-b724-6c7db0176d10-tls\") pod \"korrel8r-84f5b7859-vjmzq\" (UID: \"acfec3e0-9e55-4f88-b724-6c7db0176d10\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" Oct 30 02:57:46.360198 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.360165 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p67h\" (UniqueName: \"kubernetes.io/projected/acfec3e0-9e55-4f88-b724-6c7db0176d10-kube-api-access-5p67h\") pod \"korrel8r-84f5b7859-vjmzq\" (UID: \"acfec3e0-9e55-4f88-b724-6c7db0176d10\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" Oct 30 02:57:46.477627 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.477552 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" Oct 30 02:57:46.776287 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:46.776262 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq"] Oct 30 02:57:46.778543 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:57:46.778512 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacfec3e0_9e55_4f88_b724_6c7db0176d10.slice/crio-24cca1e4a7d3b65f8fe62a65bcad0d468b6e249a80de4112fdaa7d16c0776e64 WatchSource:0}: Error finding container 24cca1e4a7d3b65f8fe62a65bcad0d468b6e249a80de4112fdaa7d16c0776e64: Status 404 returned error can't find the container with id 24cca1e4a7d3b65f8fe62a65bcad0d468b6e249a80de4112fdaa7d16c0776e64 Oct 30 02:57:47.093827 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.093794 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" event={"ID":"acfec3e0-9e55-4f88-b724-6c7db0176d10","Type":"ContainerStarted","Data":"3f66d9ef6fd9db447a70442d899589eed1dd16669a8afa7a9245ab2dcd7e5ef6"} Oct 30 02:57:47.093983 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.093846 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" event={"ID":"acfec3e0-9e55-4f88-b724-6c7db0176d10","Type":"ContainerStarted","Data":"24cca1e4a7d3b65f8fe62a65bcad0d468b6e249a80de4112fdaa7d16c0776e64"} Oct 30 02:57:47.093983 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.093891 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" Oct 30 02:57:47.141739 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.141692 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" podStartSLOduration=1.141677487 podStartE2EDuration="1.141677487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:47.139463006 +0000 UTC m=+1028.152741108" watchObservedRunningTime="2026-10-30 02:57:47.141677487 +0000 UTC m=+1028.154955612" Oct 30 02:57:47.221918 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.221880 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 02:57:47.225829 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.225806 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:47.246554 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.246519 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-0df4277d\"" Oct 30 02:57:47.246722 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.246614 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 02:57:47.248552 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.248526 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 02:57:47.256130 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.256106 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 02:57:47.256130 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.256117 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-2sftv\"" Oct 30 02:57:47.347984 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.347905 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:47.347984 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.347948 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:47.348152 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.348031 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cq6b\" (UniqueName: \"kubernetes.io/projected/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-kube-api-access-8cq6b\") pod \"data-science-collector-collector-0\" (UID: \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:47.348152 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.348074 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:47.448527 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.448492 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:47.448722 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.448589 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:47.448722 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.448637 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:47.448722 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.448662 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cq6b\" (UniqueName: \"kubernetes.io/projected/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-kube-api-access-8cq6b\") pod \"data-science-collector-collector-0\" (UID: \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:47.448942 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:57:47.448741 4925 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found Oct 30 02:57:47.448942 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:57:47.448858 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-data-science-collector-ta-client-cert podName:5a0932f9-5a95-4666-81d3-2f4f1ca5e334 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:47.94883584 +0000 UTC m=+1028.962113920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-data-science-collector-ta-client-cert") pod "data-science-collector-collector-0" (UID: "5a0932f9-5a95-4666-81d3-2f4f1ca5e334") : secret "data-science-collector-ta-client-cert" not found Oct 30 02:57:47.449347 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.449327 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:47.451039 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.451022 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:47.473343 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.473315 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cq6b\" (UniqueName: \"kubernetes.io/projected/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-kube-api-access-8cq6b\") pod \"data-science-collector-collector-0\" (UID: \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:47.952569 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:47.952528 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:47.952739 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:57:47.952650 4925 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found Oct 30 02:57:47.952739 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:57:47.952703 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-data-science-collector-ta-client-cert podName:5a0932f9-5a95-4666-81d3-2f4f1ca5e334 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:48.952689941 +0000 UTC m=+1029.965968021 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-data-science-collector-ta-client-cert") pod "data-science-collector-collector-0" (UID: "5a0932f9-5a95-4666-81d3-2f4f1ca5e334") : secret "data-science-collector-ta-client-cert" not found Oct 30 02:57:48.161720 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:57:48.161684 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:57:48.962935 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:48.962894 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:48.965558 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:48.965529 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:49.047333 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:49.047278 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 02:57:49.344489 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:49.344461 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 02:57:49.346585 ip-10-0-139-19 kubenswrapper[4925]: W1030 02:57:49.346556 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0932f9_5a95_4666_81d3_2f4f1ca5e334.slice/crio-fd324a577e951d2a6b7408e9c051d2ace4e8117d68d6530cf9c9e10f7c36c55a WatchSource:0}: Error finding container fd324a577e951d2a6b7408e9c051d2ace4e8117d68d6530cf9c9e10f7c36c55a: Status 404 returned error can't find the container with id fd324a577e951d2a6b7408e9c051d2ace4e8117d68d6530cf9c9e10f7c36c55a Oct 30 02:57:49.348903 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:49.348869 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:57:49.348986 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:49.348931 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:57:50.112937 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:50.112896 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5a0932f9-5a95-4666-81d3-2f4f1ca5e334","Type":"ContainerStarted","Data":"a35a58665989c0b64375ef450be2f7af057bc80c75d5773c72101da0a809471b"} Oct 30 02:57:50.113376 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:50.113000 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5a0932f9-5a95-4666-81d3-2f4f1ca5e334","Type":"ContainerStarted","Data":"fd324a577e951d2a6b7408e9c051d2ace4e8117d68d6530cf9c9e10f7c36c55a"} Oct 30 02:57:50.156699 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:50.156624 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=3.156610739 podStartE2EDuration="3.156610739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:50.155736156 +0000 UTC m=+1031.169014261" watchObservedRunningTime="2026-10-30 02:57:50.156610739 +0000 UTC m=+1031.169888841" Oct 30 02:57:58.102481 ip-10-0-139-19 kubenswrapper[4925]: I1030 02:57:58.102406 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" Oct 30 02:58:18.162104 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:58:18.162063 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:58:48.161319 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:58:48.161281 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:59:18.162292 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:59:18.162249 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 02:59:48.161317 ip-10-0-139-19 kubenswrapper[4925]: E1030 02:59:48.161280 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:00:18.161335 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:00:18.161293 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:00:39.207909 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:00:39.207877 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:00:39.210474 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:00:39.210453 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:00:39.216366 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:00:39.216348 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:00:39.218992 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:00:39.218976 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:00:48.161221 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:00:48.161184 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:01:18.161516 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:01:18.161476 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:01:48.162926 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:01:48.162886 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:02:18.161561 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:02:18.161514 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:02:48.162314 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:02:48.162276 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:03:18.162315 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:03:18.162276 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:03:48.162130 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:03:48.162088 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:04:18.161626 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:04:18.161587 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:04:48.161882 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:04:48.161842 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:05:18.161655 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:05:18.161608 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:05:39.237908 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:05:39.237880 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:05:39.241219 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:05:39.241197 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:05:39.246271 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:05:39.246255 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:05:39.251791 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:05:39.249999 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:05:48.160938 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:05:48.160902 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:06:18.161609 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:06:18.161564 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:06:48.161334 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:06:48.161296 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:07:18.161799 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:07:18.161741 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:07:48.160476 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:07:48.160442 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:08:01.235359 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:01.235324 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq"] Oct 30 03:08:01.235893 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:01.235510 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" podUID="acfec3e0-9e55-4f88-b724-6c7db0176d10" containerName="korrel8r" containerID="cri-o://3f66d9ef6fd9db447a70442d899589eed1dd16669a8afa7a9245ab2dcd7e5ef6" gracePeriod=30 Oct 30 03:08:01.478547 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:01.478522 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" Oct 30 03:08:01.537790 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:01.537707 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acfec3e0-9e55-4f88-b724-6c7db0176d10-tls\" (UniqueName: \"kubernetes.io/secret/acfec3e0-9e55-4f88-b724-6c7db0176d10-tls\") pod \"acfec3e0-9e55-4f88-b724-6c7db0176d10\" (UID: \"acfec3e0-9e55-4f88-b724-6c7db0176d10\") " Oct 30 03:08:01.537918 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:01.537791 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acfec3e0-9e55-4f88-b724-6c7db0176d10-kube-api-access-5p67h\" (UniqueName: \"kubernetes.io/projected/acfec3e0-9e55-4f88-b724-6c7db0176d10-kube-api-access-5p67h\") pod \"acfec3e0-9e55-4f88-b724-6c7db0176d10\" (UID: \"acfec3e0-9e55-4f88-b724-6c7db0176d10\") " Oct 30 03:08:01.537918 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:01.537868 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acfec3e0-9e55-4f88-b724-6c7db0176d10-config\" (UniqueName: \"kubernetes.io/configmap/acfec3e0-9e55-4f88-b724-6c7db0176d10-config\") pod \"acfec3e0-9e55-4f88-b724-6c7db0176d10\" (UID: \"acfec3e0-9e55-4f88-b724-6c7db0176d10\") " Oct 30 03:08:01.538290 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:01.538260 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acfec3e0-9e55-4f88-b724-6c7db0176d10-config" pod "acfec3e0-9e55-4f88-b724-6c7db0176d10" (UID: "acfec3e0-9e55-4f88-b724-6c7db0176d10"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:08:01.539930 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:01.539902 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acfec3e0-9e55-4f88-b724-6c7db0176d10-tls" pod "acfec3e0-9e55-4f88-b724-6c7db0176d10" (UID: "acfec3e0-9e55-4f88-b724-6c7db0176d10"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:08:01.540064 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:01.540044 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acfec3e0-9e55-4f88-b724-6c7db0176d10-kube-api-access-5p67h" pod "acfec3e0-9e55-4f88-b724-6c7db0176d10" (UID: "acfec3e0-9e55-4f88-b724-6c7db0176d10"). InnerVolumeSpecName "kube-api-access-5p67h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:08:01.638906 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:01.638873 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acfec3e0-9e55-4f88-b724-6c7db0176d10-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:08:01.638906 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:01.638899 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/acfec3e0-9e55-4f88-b724-6c7db0176d10-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:08:01.638906 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:01.638909 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p67h\" (UniqueName: \"kubernetes.io/projected/acfec3e0-9e55-4f88-b724-6c7db0176d10-kube-api-access-5p67h\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:08:01.970545 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:01.970508 4925 generic.go:356] "Generic (PLEG): container finished" podID="acfec3e0-9e55-4f88-b724-6c7db0176d10" containerID="3f66d9ef6fd9db447a70442d899589eed1dd16669a8afa7a9245ab2dcd7e5ef6" exitCode=0 Oct 30 03:08:01.970715 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:01.970556 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" event={"ID":"acfec3e0-9e55-4f88-b724-6c7db0176d10","Type":"ContainerDied","Data":"3f66d9ef6fd9db447a70442d899589eed1dd16669a8afa7a9245ab2dcd7e5ef6"} Oct 30 03:08:01.970715 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:01.970565 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" Oct 30 03:08:01.970715 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:01.970628 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq" event={"ID":"acfec3e0-9e55-4f88-b724-6c7db0176d10","Type":"ContainerDied","Data":"24cca1e4a7d3b65f8fe62a65bcad0d468b6e249a80de4112fdaa7d16c0776e64"} Oct 30 03:08:01.970715 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:01.970652 4925 scope.go:122] "RemoveContainer" containerID="3f66d9ef6fd9db447a70442d899589eed1dd16669a8afa7a9245ab2dcd7e5ef6" Oct 30 03:08:01.979540 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:01.979516 4925 scope.go:122] "RemoveContainer" containerID="3f66d9ef6fd9db447a70442d899589eed1dd16669a8afa7a9245ab2dcd7e5ef6" Oct 30 03:08:01.979808 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:08:01.979786 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f66d9ef6fd9db447a70442d899589eed1dd16669a8afa7a9245ab2dcd7e5ef6\": container with ID starting with 3f66d9ef6fd9db447a70442d899589eed1dd16669a8afa7a9245ab2dcd7e5ef6 not found: ID does not exist" containerID="3f66d9ef6fd9db447a70442d899589eed1dd16669a8afa7a9245ab2dcd7e5ef6" Oct 30 03:08:01.979894 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:01.979814 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f66d9ef6fd9db447a70442d899589eed1dd16669a8afa7a9245ab2dcd7e5ef6"} err="failed to get container status \"3f66d9ef6fd9db447a70442d899589eed1dd16669a8afa7a9245ab2dcd7e5ef6\": rpc error: code = NotFound desc = could not find container \"3f66d9ef6fd9db447a70442d899589eed1dd16669a8afa7a9245ab2dcd7e5ef6\": container with ID starting with 3f66d9ef6fd9db447a70442d899589eed1dd16669a8afa7a9245ab2dcd7e5ef6 not found: ID does not exist" Oct 30 03:08:02.025437 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.025413 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq"] Oct 30 03:08:02.038008 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.037985 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-84f5b7859-vjmzq"] Oct 30 03:08:02.218887 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.218854 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:08:02.219137 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.219112 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="5a0932f9-5a95-4666-81d3-2f4f1ca5e334" containerName="otc-container" containerID="cri-o://a35a58665989c0b64375ef450be2f7af057bc80c75d5773c72101da0a809471b" gracePeriod=30 Oct 30 03:08:02.471029 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.471008 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:02.545234 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.545146 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-monitoring-tls-certs\") pod \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\" (UID: \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\") " Oct 30 03:08:02.545234 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.545185 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-data-science-collector-ta-client-cert\") pod \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\" (UID: \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\") " Oct 30 03:08:02.545234 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.545208 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-kube-api-access-8cq6b\" (UniqueName: \"kubernetes.io/projected/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-kube-api-access-8cq6b\") pod \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\" (UID: \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\") " Oct 30 03:08:02.545501 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.545238 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-otc-internal\") pod \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\" (UID: \"5a0932f9-5a95-4666-81d3-2f4f1ca5e334\") " Oct 30 03:08:02.545692 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.545655 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-otc-internal" pod "5a0932f9-5a95-4666-81d3-2f4f1ca5e334" (UID: "5a0932f9-5a95-4666-81d3-2f4f1ca5e334"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:08:02.547384 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.547358 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-kube-api-access-8cq6b" pod "5a0932f9-5a95-4666-81d3-2f4f1ca5e334" (UID: "5a0932f9-5a95-4666-81d3-2f4f1ca5e334"). InnerVolumeSpecName "kube-api-access-8cq6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:08:02.547662 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.547637 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-data-science-collector-ta-client-cert" pod "5a0932f9-5a95-4666-81d3-2f4f1ca5e334" (UID: "5a0932f9-5a95-4666-81d3-2f4f1ca5e334"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:08:02.547841 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.547816 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-monitoring-tls-certs" pod "5a0932f9-5a95-4666-81d3-2f4f1ca5e334" (UID: "5a0932f9-5a95-4666-81d3-2f4f1ca5e334"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:08:02.646557 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.646511 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:08:02.646557 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.646552 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:08:02.646557 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.646563 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cq6b\" (UniqueName: \"kubernetes.io/projected/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-kube-api-access-8cq6b\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:08:02.646557 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.646573 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5a0932f9-5a95-4666-81d3-2f4f1ca5e334-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:08:02.976558 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.976525 4925 generic.go:356] "Generic (PLEG): container finished" podID="5a0932f9-5a95-4666-81d3-2f4f1ca5e334" containerID="a35a58665989c0b64375ef450be2f7af057bc80c75d5773c72101da0a809471b" exitCode=0 Oct 30 03:08:02.976712 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.976575 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:02.976712 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.976614 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5a0932f9-5a95-4666-81d3-2f4f1ca5e334","Type":"ContainerDied","Data":"a35a58665989c0b64375ef450be2f7af057bc80c75d5773c72101da0a809471b"} Oct 30 03:08:02.976712 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.976665 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5a0932f9-5a95-4666-81d3-2f4f1ca5e334","Type":"ContainerDied","Data":"fd324a577e951d2a6b7408e9c051d2ace4e8117d68d6530cf9c9e10f7c36c55a"} Oct 30 03:08:02.976712 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.976693 4925 scope.go:122] "RemoveContainer" containerID="a35a58665989c0b64375ef450be2f7af057bc80c75d5773c72101da0a809471b" Oct 30 03:08:02.985505 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.985489 4925 scope.go:122] "RemoveContainer" containerID="a35a58665989c0b64375ef450be2f7af057bc80c75d5773c72101da0a809471b" Oct 30 03:08:02.985777 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:08:02.985741 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a35a58665989c0b64375ef450be2f7af057bc80c75d5773c72101da0a809471b\": container with ID starting with a35a58665989c0b64375ef450be2f7af057bc80c75d5773c72101da0a809471b not found: ID does not exist" containerID="a35a58665989c0b64375ef450be2f7af057bc80c75d5773c72101da0a809471b" Oct 30 03:08:02.985831 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:02.985785 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35a58665989c0b64375ef450be2f7af057bc80c75d5773c72101da0a809471b"} err="failed to get container status \"a35a58665989c0b64375ef450be2f7af057bc80c75d5773c72101da0a809471b\": rpc error: code = NotFound desc = could not find container \"a35a58665989c0b64375ef450be2f7af057bc80c75d5773c72101da0a809471b\": container with ID starting with a35a58665989c0b64375ef450be2f7af057bc80c75d5773c72101da0a809471b not found: ID does not exist" Oct 30 03:08:03.030287 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:03.030254 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:08:03.042944 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:03.042916 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:08:03.119712 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:03.119682 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a0932f9-5a95-4666-81d3-2f4f1ca5e334" path="/var/lib/kubelet/pods/5a0932f9-5a95-4666-81d3-2f4f1ca5e334/volumes" Oct 30 03:08:03.120077 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:03.120063 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acfec3e0-9e55-4f88-b724-6c7db0176d10" path="/var/lib/kubelet/pods/acfec3e0-9e55-4f88-b724-6c7db0176d10/volumes" Oct 30 03:08:06.903055 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:06.903015 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6"] Oct 30 03:08:06.903525 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:06.903509 4925 cpu_manager.go:414] "containerMap: removing container" podUID="acfec3e0-9e55-4f88-b724-6c7db0176d10" containerName="korrel8r" Oct 30 03:08:06.903593 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:06.903530 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acfec3e0-9e55-4f88-b724-6c7db0176d10" containerName="korrel8r" Oct 30 03:08:06.903593 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:06.903543 4925 cpu_manager.go:414] "containerMap: removing container" podUID="5a0932f9-5a95-4666-81d3-2f4f1ca5e334" containerName="otc-container" Oct 30 03:08:06.903593 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:06.903551 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a0932f9-5a95-4666-81d3-2f4f1ca5e334" containerName="otc-container" Oct 30 03:08:06.903737 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:06.903637 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="acfec3e0-9e55-4f88-b724-6c7db0176d10" containerName="korrel8r" Oct 30 03:08:06.903737 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:06.903649 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a0932f9-5a95-4666-81d3-2f4f1ca5e334" containerName="otc-container" Oct 30 03:08:06.907717 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:06.907697 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" Oct 30 03:08:06.927562 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:06.927538 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6"] Oct 30 03:08:06.938463 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:06.938441 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 03:08:06.938604 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:06.938518 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-jc49t\"" Oct 30 03:08:06.938604 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:06.938569 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 03:08:06.938604 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:06.938588 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 03:08:06.938604 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:06.938596 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 03:08:06.981692 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:06.981664 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/07c908aa-dab6-486e-83ff-6bd52e2e15a4-tls\") pod \"korrel8r-58bbdb5dcb-bjdn6\" (UID: \"07c908aa-dab6-486e-83ff-6bd52e2e15a4\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" Oct 30 03:08:06.981840 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:06.981710 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp59d\" (UniqueName: \"kubernetes.io/projected/07c908aa-dab6-486e-83ff-6bd52e2e15a4-kube-api-access-qp59d\") pod \"korrel8r-58bbdb5dcb-bjdn6\" (UID: \"07c908aa-dab6-486e-83ff-6bd52e2e15a4\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" Oct 30 03:08:06.981915 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:06.981853 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07c908aa-dab6-486e-83ff-6bd52e2e15a4-config\") pod \"korrel8r-58bbdb5dcb-bjdn6\" (UID: \"07c908aa-dab6-486e-83ff-6bd52e2e15a4\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" Oct 30 03:08:07.089142 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.082993 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/07c908aa-dab6-486e-83ff-6bd52e2e15a4-tls\") pod \"korrel8r-58bbdb5dcb-bjdn6\" (UID: \"07c908aa-dab6-486e-83ff-6bd52e2e15a4\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" Oct 30 03:08:07.089142 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.083277 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp59d\" (UniqueName: \"kubernetes.io/projected/07c908aa-dab6-486e-83ff-6bd52e2e15a4-kube-api-access-qp59d\") pod \"korrel8r-58bbdb5dcb-bjdn6\" (UID: \"07c908aa-dab6-486e-83ff-6bd52e2e15a4\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" Oct 30 03:08:07.089142 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.083377 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07c908aa-dab6-486e-83ff-6bd52e2e15a4-config\") pod \"korrel8r-58bbdb5dcb-bjdn6\" (UID: \"07c908aa-dab6-486e-83ff-6bd52e2e15a4\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" Oct 30 03:08:07.089142 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.084311 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07c908aa-dab6-486e-83ff-6bd52e2e15a4-config\") pod \"korrel8r-58bbdb5dcb-bjdn6\" (UID: \"07c908aa-dab6-486e-83ff-6bd52e2e15a4\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" Oct 30 03:08:07.089142 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.086196 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/07c908aa-dab6-486e-83ff-6bd52e2e15a4-tls\") pod \"korrel8r-58bbdb5dcb-bjdn6\" (UID: \"07c908aa-dab6-486e-83ff-6bd52e2e15a4\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" Oct 30 03:08:07.108684 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.108656 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp59d\" (UniqueName: \"kubernetes.io/projected/07c908aa-dab6-486e-83ff-6bd52e2e15a4-kube-api-access-qp59d\") pod \"korrel8r-58bbdb5dcb-bjdn6\" (UID: \"07c908aa-dab6-486e-83ff-6bd52e2e15a4\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" Oct 30 03:08:07.220666 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.220596 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" Oct 30 03:08:07.525781 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.525666 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6"] Oct 30 03:08:07.529872 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:08:07.529839 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c908aa_dab6_486e_83ff_6bd52e2e15a4.slice/crio-66bfc7ed82de2cb295dc524c438bea1de20ef7136380da983a49a80e1ba24b67 WatchSource:0}: Error finding container 66bfc7ed82de2cb295dc524c438bea1de20ef7136380da983a49a80e1ba24b67: Status 404 returned error can't find the container with id 66bfc7ed82de2cb295dc524c438bea1de20ef7136380da983a49a80e1ba24b67 Oct 30 03:08:07.557872 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.557847 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:08:07.562811 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.562788 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:07.579806 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.579780 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-638a775f\"" Oct 30 03:08:07.587525 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.586240 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:08:07.592442 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.592420 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-x9jdm\"" Oct 30 03:08:07.592499 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.592424 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:08:07.689227 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.689192 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6eb8d74b-2985-4368-ab80-41e090965e5a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6eb8d74b-2985-4368-ab80-41e090965e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:07.689227 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.689229 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5dlw\" (UniqueName: \"kubernetes.io/projected/6eb8d74b-2985-4368-ab80-41e090965e5a-kube-api-access-z5dlw\") pod \"data-science-collector-collector-0\" (UID: \"6eb8d74b-2985-4368-ab80-41e090965e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:07.689439 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.689257 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6eb8d74b-2985-4368-ab80-41e090965e5a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6eb8d74b-2985-4368-ab80-41e090965e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:07.790167 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.790079 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6eb8d74b-2985-4368-ab80-41e090965e5a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6eb8d74b-2985-4368-ab80-41e090965e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:07.790167 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.790119 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5dlw\" (UniqueName: \"kubernetes.io/projected/6eb8d74b-2985-4368-ab80-41e090965e5a-kube-api-access-z5dlw\") pod \"data-science-collector-collector-0\" (UID: \"6eb8d74b-2985-4368-ab80-41e090965e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:07.790352 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.790237 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6eb8d74b-2985-4368-ab80-41e090965e5a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6eb8d74b-2985-4368-ab80-41e090965e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:07.790904 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.790882 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6eb8d74b-2985-4368-ab80-41e090965e5a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6eb8d74b-2985-4368-ab80-41e090965e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:07.792723 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.792704 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6eb8d74b-2985-4368-ab80-41e090965e5a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6eb8d74b-2985-4368-ab80-41e090965e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:07.814278 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.814251 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5dlw\" (UniqueName: \"kubernetes.io/projected/6eb8d74b-2985-4368-ab80-41e090965e5a-kube-api-access-z5dlw\") pod \"data-science-collector-collector-0\" (UID: \"6eb8d74b-2985-4368-ab80-41e090965e5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:07.893326 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:07.893295 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:08.009700 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:08.009654 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" event={"ID":"07c908aa-dab6-486e-83ff-6bd52e2e15a4","Type":"ContainerStarted","Data":"cc9ff68eb083537ffd8527c6615c4934bd7a71a49881941a2ed082b858e706c4"} Oct 30 03:08:08.010092 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:08.009716 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" event={"ID":"07c908aa-dab6-486e-83ff-6bd52e2e15a4","Type":"ContainerStarted","Data":"66bfc7ed82de2cb295dc524c438bea1de20ef7136380da983a49a80e1ba24b67"} Oct 30 03:08:08.010092 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:08.009875 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" Oct 30 03:08:08.053226 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:08.053144 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" podStartSLOduration=2.053132301 podStartE2EDuration="2.053132301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:08.052559516 +0000 UTC m=+1649.065837619" watchObservedRunningTime="2026-10-30 03:08:08.053132301 +0000 UTC m=+1649.066410403" Oct 30 03:08:08.192304 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:08.192267 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:08:08.194949 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:08:08.194920 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb8d74b_2985_4368_ab80_41e090965e5a.slice/crio-d9da1778ae603c0d983e26924b54b60bc2eb026b58b10a36792c21460ca75c77 WatchSource:0}: Error finding container d9da1778ae603c0d983e26924b54b60bc2eb026b58b10a36792c21460ca75c77: Status 404 returned error can't find the container with id d9da1778ae603c0d983e26924b54b60bc2eb026b58b10a36792c21460ca75c77 Oct 30 03:08:08.197083 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:08.197055 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:08:08.197157 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:08.197126 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:08:09.016847 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:09.016812 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6eb8d74b-2985-4368-ab80-41e090965e5a","Type":"ContainerStarted","Data":"722b9cdfc938d86c8505da17851bac114c77f8c4264ab17bfa563b5b705cb26f"} Oct 30 03:08:09.017256 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:09.016869 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6eb8d74b-2985-4368-ab80-41e090965e5a","Type":"ContainerStarted","Data":"d9da1778ae603c0d983e26924b54b60bc2eb026b58b10a36792c21460ca75c77"} Oct 30 03:08:09.060519 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:09.060469 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.060457003 podStartE2EDuration="2.060457003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:09.059936077 +0000 UTC m=+1650.073214182" watchObservedRunningTime="2026-10-30 03:08:09.060457003 +0000 UTC m=+1650.073735106" Oct 30 03:08:18.160348 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:08:18.160301 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:08:19.019557 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:19.019523 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" Oct 30 03:08:22.237395 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:22.237311 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6"] Oct 30 03:08:22.237858 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:22.237571 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" podUID="07c908aa-dab6-486e-83ff-6bd52e2e15a4" containerName="korrel8r" containerID="cri-o://cc9ff68eb083537ffd8527c6615c4934bd7a71a49881941a2ed082b858e706c4" gracePeriod=30 Oct 30 03:08:22.477419 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:22.477391 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" Oct 30 03:08:22.616823 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:22.616793 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07c908aa-dab6-486e-83ff-6bd52e2e15a4-tls\" (UniqueName: \"kubernetes.io/secret/07c908aa-dab6-486e-83ff-6bd52e2e15a4-tls\") pod \"07c908aa-dab6-486e-83ff-6bd52e2e15a4\" (UID: \"07c908aa-dab6-486e-83ff-6bd52e2e15a4\") " Oct 30 03:08:22.617003 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:22.616846 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07c908aa-dab6-486e-83ff-6bd52e2e15a4-kube-api-access-qp59d\" (UniqueName: \"kubernetes.io/projected/07c908aa-dab6-486e-83ff-6bd52e2e15a4-kube-api-access-qp59d\") pod \"07c908aa-dab6-486e-83ff-6bd52e2e15a4\" (UID: \"07c908aa-dab6-486e-83ff-6bd52e2e15a4\") " Oct 30 03:08:22.617003 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:22.616881 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07c908aa-dab6-486e-83ff-6bd52e2e15a4-config\" (UniqueName: \"kubernetes.io/configmap/07c908aa-dab6-486e-83ff-6bd52e2e15a4-config\") pod \"07c908aa-dab6-486e-83ff-6bd52e2e15a4\" (UID: \"07c908aa-dab6-486e-83ff-6bd52e2e15a4\") " Oct 30 03:08:22.617399 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:22.617368 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07c908aa-dab6-486e-83ff-6bd52e2e15a4-config" pod "07c908aa-dab6-486e-83ff-6bd52e2e15a4" (UID: "07c908aa-dab6-486e-83ff-6bd52e2e15a4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:08:22.619985 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:22.619951 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07c908aa-dab6-486e-83ff-6bd52e2e15a4-tls" pod "07c908aa-dab6-486e-83ff-6bd52e2e15a4" (UID: "07c908aa-dab6-486e-83ff-6bd52e2e15a4"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:08:22.619985 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:22.619968 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c908aa-dab6-486e-83ff-6bd52e2e15a4-kube-api-access-qp59d" pod "07c908aa-dab6-486e-83ff-6bd52e2e15a4" (UID: "07c908aa-dab6-486e-83ff-6bd52e2e15a4"). InnerVolumeSpecName "kube-api-access-qp59d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:08:22.629900 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:22.629875 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:08:22.630130 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:22.630093 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="6eb8d74b-2985-4368-ab80-41e090965e5a" containerName="otc-container" containerID="cri-o://722b9cdfc938d86c8505da17851bac114c77f8c4264ab17bfa563b5b705cb26f" gracePeriod=30 Oct 30 03:08:22.717602 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:22.717576 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/07c908aa-dab6-486e-83ff-6bd52e2e15a4-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:08:22.717602 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:22.717600 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp59d\" (UniqueName: \"kubernetes.io/projected/07c908aa-dab6-486e-83ff-6bd52e2e15a4-kube-api-access-qp59d\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:08:22.717811 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:22.717612 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07c908aa-dab6-486e-83ff-6bd52e2e15a4-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:08:22.861028 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:22.861005 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:23.019669 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.019590 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eb8d74b-2985-4368-ab80-41e090965e5a-otc-internal\" (UniqueName: \"kubernetes.io/configmap/6eb8d74b-2985-4368-ab80-41e090965e5a-otc-internal\") pod \"6eb8d74b-2985-4368-ab80-41e090965e5a\" (UID: \"6eb8d74b-2985-4368-ab80-41e090965e5a\") " Oct 30 03:08:23.019669 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.019626 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eb8d74b-2985-4368-ab80-41e090965e5a-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6eb8d74b-2985-4368-ab80-41e090965e5a-monitoring-tls-certs\") pod \"6eb8d74b-2985-4368-ab80-41e090965e5a\" (UID: \"6eb8d74b-2985-4368-ab80-41e090965e5a\") " Oct 30 03:08:23.019928 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.019699 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb8d74b-2985-4368-ab80-41e090965e5a-kube-api-access-z5dlw\" (UniqueName: \"kubernetes.io/projected/6eb8d74b-2985-4368-ab80-41e090965e5a-kube-api-access-z5dlw\") pod \"6eb8d74b-2985-4368-ab80-41e090965e5a\" (UID: \"6eb8d74b-2985-4368-ab80-41e090965e5a\") " Oct 30 03:08:23.019986 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.019943 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb8d74b-2985-4368-ab80-41e090965e5a-otc-internal" pod "6eb8d74b-2985-4368-ab80-41e090965e5a" (UID: "6eb8d74b-2985-4368-ab80-41e090965e5a"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:08:23.021762 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.021729 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb8d74b-2985-4368-ab80-41e090965e5a-kube-api-access-z5dlw" pod "6eb8d74b-2985-4368-ab80-41e090965e5a" (UID: "6eb8d74b-2985-4368-ab80-41e090965e5a"). InnerVolumeSpecName "kube-api-access-z5dlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:08:23.021842 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.021792 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eb8d74b-2985-4368-ab80-41e090965e5a-monitoring-tls-certs" pod "6eb8d74b-2985-4368-ab80-41e090965e5a" (UID: "6eb8d74b-2985-4368-ab80-41e090965e5a"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:08:23.086450 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.086414 4925 generic.go:356] "Generic (PLEG): container finished" podID="6eb8d74b-2985-4368-ab80-41e090965e5a" containerID="722b9cdfc938d86c8505da17851bac114c77f8c4264ab17bfa563b5b705cb26f" exitCode=0 Oct 30 03:08:23.086620 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.086474 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:23.086723 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.086693 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6eb8d74b-2985-4368-ab80-41e090965e5a","Type":"ContainerDied","Data":"722b9cdfc938d86c8505da17851bac114c77f8c4264ab17bfa563b5b705cb26f"} Oct 30 03:08:23.086870 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.086776 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6eb8d74b-2985-4368-ab80-41e090965e5a","Type":"ContainerDied","Data":"d9da1778ae603c0d983e26924b54b60bc2eb026b58b10a36792c21460ca75c77"} Oct 30 03:08:23.086870 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.086806 4925 scope.go:122] "RemoveContainer" containerID="722b9cdfc938d86c8505da17851bac114c77f8c4264ab17bfa563b5b705cb26f" Oct 30 03:08:23.088979 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.088957 4925 generic.go:356] "Generic (PLEG): container finished" podID="07c908aa-dab6-486e-83ff-6bd52e2e15a4" containerID="cc9ff68eb083537ffd8527c6615c4934bd7a71a49881941a2ed082b858e706c4" exitCode=0 Oct 30 03:08:23.089069 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.088994 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" event={"ID":"07c908aa-dab6-486e-83ff-6bd52e2e15a4","Type":"ContainerDied","Data":"cc9ff68eb083537ffd8527c6615c4934bd7a71a49881941a2ed082b858e706c4"} Oct 30 03:08:23.089069 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.089032 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" event={"ID":"07c908aa-dab6-486e-83ff-6bd52e2e15a4","Type":"ContainerDied","Data":"66bfc7ed82de2cb295dc524c438bea1de20ef7136380da983a49a80e1ba24b67"} Oct 30 03:08:23.089147 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.089112 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6" Oct 30 03:08:23.100021 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.099954 4925 scope.go:122] "RemoveContainer" containerID="722b9cdfc938d86c8505da17851bac114c77f8c4264ab17bfa563b5b705cb26f" Oct 30 03:08:23.100407 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:08:23.100371 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"722b9cdfc938d86c8505da17851bac114c77f8c4264ab17bfa563b5b705cb26f\": container with ID starting with 722b9cdfc938d86c8505da17851bac114c77f8c4264ab17bfa563b5b705cb26f not found: ID does not exist" containerID="722b9cdfc938d86c8505da17851bac114c77f8c4264ab17bfa563b5b705cb26f" Oct 30 03:08:23.100502 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.100435 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"722b9cdfc938d86c8505da17851bac114c77f8c4264ab17bfa563b5b705cb26f"} err="failed to get container status \"722b9cdfc938d86c8505da17851bac114c77f8c4264ab17bfa563b5b705cb26f\": rpc error: code = NotFound desc = could not find container \"722b9cdfc938d86c8505da17851bac114c77f8c4264ab17bfa563b5b705cb26f\": container with ID starting with 722b9cdfc938d86c8505da17851bac114c77f8c4264ab17bfa563b5b705cb26f not found: ID does not exist" Oct 30 03:08:23.100502 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.100455 4925 scope.go:122] "RemoveContainer" containerID="cc9ff68eb083537ffd8527c6615c4934bd7a71a49881941a2ed082b858e706c4" Oct 30 03:08:23.109403 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.109379 4925 scope.go:122] "RemoveContainer" containerID="cc9ff68eb083537ffd8527c6615c4934bd7a71a49881941a2ed082b858e706c4" Oct 30 03:08:23.109694 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:08:23.109676 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc9ff68eb083537ffd8527c6615c4934bd7a71a49881941a2ed082b858e706c4\": container with ID starting with cc9ff68eb083537ffd8527c6615c4934bd7a71a49881941a2ed082b858e706c4 not found: ID does not exist" containerID="cc9ff68eb083537ffd8527c6615c4934bd7a71a49881941a2ed082b858e706c4" Oct 30 03:08:23.109791 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.109699 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc9ff68eb083537ffd8527c6615c4934bd7a71a49881941a2ed082b858e706c4"} err="failed to get container status \"cc9ff68eb083537ffd8527c6615c4934bd7a71a49881941a2ed082b858e706c4\": rpc error: code = NotFound desc = could not find container \"cc9ff68eb083537ffd8527c6615c4934bd7a71a49881941a2ed082b858e706c4\": container with ID starting with cc9ff68eb083537ffd8527c6615c4934bd7a71a49881941a2ed082b858e706c4 not found: ID does not exist" Oct 30 03:08:23.121140 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.121116 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5dlw\" (UniqueName: \"kubernetes.io/projected/6eb8d74b-2985-4368-ab80-41e090965e5a-kube-api-access-z5dlw\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:08:23.121230 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.121146 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6eb8d74b-2985-4368-ab80-41e090965e5a-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:08:23.121230 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.121163 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6eb8d74b-2985-4368-ab80-41e090965e5a-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:08:23.139442 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.139421 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6"] Oct 30 03:08:23.143983 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.143964 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-58bbdb5dcb-bjdn6"] Oct 30 03:08:23.181337 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.181313 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:08:23.187285 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:23.187256 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:08:25.118597 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:25.118561 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07c908aa-dab6-486e-83ff-6bd52e2e15a4" path="/var/lib/kubelet/pods/07c908aa-dab6-486e-83ff-6bd52e2e15a4/volumes" Oct 30 03:08:25.118991 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:25.118962 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eb8d74b-2985-4368-ab80-41e090965e5a" path="/var/lib/kubelet/pods/6eb8d74b-2985-4368-ab80-41e090965e5a/volumes" Oct 30 03:08:26.048152 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.048108 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5c7449969d-84wnm"] Oct 30 03:08:26.048476 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.048463 4925 cpu_manager.go:414] "containerMap: removing container" podUID="07c908aa-dab6-486e-83ff-6bd52e2e15a4" containerName="korrel8r" Oct 30 03:08:26.048520 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.048477 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07c908aa-dab6-486e-83ff-6bd52e2e15a4" containerName="korrel8r" Oct 30 03:08:26.048520 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.048495 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6eb8d74b-2985-4368-ab80-41e090965e5a" containerName="otc-container" Oct 30 03:08:26.048520 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.048507 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb8d74b-2985-4368-ab80-41e090965e5a" containerName="otc-container" Oct 30 03:08:26.048606 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.048593 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="07c908aa-dab6-486e-83ff-6bd52e2e15a4" containerName="korrel8r" Oct 30 03:08:26.048606 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.048603 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb8d74b-2985-4368-ab80-41e090965e5a" containerName="otc-container" Oct 30 03:08:26.053183 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.053161 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" Oct 30 03:08:26.072293 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.072264 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5c7449969d-84wnm"] Oct 30 03:08:26.083064 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.083041 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 03:08:26.083215 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.083181 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 03:08:26.083215 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.083195 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-b7mb8\"" Oct 30 03:08:26.083305 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.083219 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 03:08:26.083305 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.083279 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 03:08:26.146728 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.146687 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-tls\") pod \"korrel8r-5c7449969d-84wnm\" (UID: \"5dd04619-86fd-4bd2-ad4b-1e81fbbeb190\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" Oct 30 03:08:26.147135 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.146731 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhx75\" (UniqueName: \"kubernetes.io/projected/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-kube-api-access-jhx75\") pod \"korrel8r-5c7449969d-84wnm\" (UID: \"5dd04619-86fd-4bd2-ad4b-1e81fbbeb190\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" Oct 30 03:08:26.147135 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.146880 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-config\") pod \"korrel8r-5c7449969d-84wnm\" (UID: \"5dd04619-86fd-4bd2-ad4b-1e81fbbeb190\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" Oct 30 03:08:26.247953 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.247907 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-config\") pod \"korrel8r-5c7449969d-84wnm\" (UID: \"5dd04619-86fd-4bd2-ad4b-1e81fbbeb190\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" Oct 30 03:08:26.248154 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.247971 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-tls\") pod \"korrel8r-5c7449969d-84wnm\" (UID: \"5dd04619-86fd-4bd2-ad4b-1e81fbbeb190\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" Oct 30 03:08:26.248154 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.248006 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhx75\" (UniqueName: \"kubernetes.io/projected/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-kube-api-access-jhx75\") pod \"korrel8r-5c7449969d-84wnm\" (UID: \"5dd04619-86fd-4bd2-ad4b-1e81fbbeb190\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" Oct 30 03:08:26.248812 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.248790 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-config\") pod \"korrel8r-5c7449969d-84wnm\" (UID: \"5dd04619-86fd-4bd2-ad4b-1e81fbbeb190\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" Oct 30 03:08:26.250432 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.250403 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-tls\") pod \"korrel8r-5c7449969d-84wnm\" (UID: \"5dd04619-86fd-4bd2-ad4b-1e81fbbeb190\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" Oct 30 03:08:26.272175 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.272142 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhx75\" (UniqueName: \"kubernetes.io/projected/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-kube-api-access-jhx75\") pod \"korrel8r-5c7449969d-84wnm\" (UID: \"5dd04619-86fd-4bd2-ad4b-1e81fbbeb190\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" Oct 30 03:08:26.365078 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.365033 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" Oct 30 03:08:26.667415 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:26.667362 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5c7449969d-84wnm"] Oct 30 03:08:26.670314 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:08:26.670282 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd04619_86fd_4bd2_ad4b_1e81fbbeb190.slice/crio-624a43ec558567dff7b6ec12613ceb5444bab502874debabf703df6eb0221d00 WatchSource:0}: Error finding container 624a43ec558567dff7b6ec12613ceb5444bab502874debabf703df6eb0221d00: Status 404 returned error can't find the container with id 624a43ec558567dff7b6ec12613ceb5444bab502874debabf703df6eb0221d00 Oct 30 03:08:27.111146 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.111107 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" event={"ID":"5dd04619-86fd-4bd2-ad4b-1e81fbbeb190","Type":"ContainerStarted","Data":"f06616e68e5ae17850b7f7f03c95e4337a076eab4d8612f8ba68a8c8e895bd15"} Oct 30 03:08:27.111362 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.111173 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" event={"ID":"5dd04619-86fd-4bd2-ad4b-1e81fbbeb190","Type":"ContainerStarted","Data":"624a43ec558567dff7b6ec12613ceb5444bab502874debabf703df6eb0221d00"} Oct 30 03:08:27.111362 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.111240 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" Oct 30 03:08:27.120182 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.120145 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:08:27.124258 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.124233 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:27.143187 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.143157 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:08:27.145727 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.145588 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-zlrvj\"" Oct 30 03:08:27.145727 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.145689 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 03:08:27.155932 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.155898 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9fbc563c-bde6-4c2b-8c32-570ec33285d3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9fbc563c-bde6-4c2b-8c32-570ec33285d3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:27.156476 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.155971 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwb8l\" (UniqueName: \"kubernetes.io/projected/9fbc563c-bde6-4c2b-8c32-570ec33285d3-kube-api-access-rwb8l\") pod \"data-science-collector-collector-0\" (UID: \"9fbc563c-bde6-4c2b-8c32-570ec33285d3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:27.156476 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.156010 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9fbc563c-bde6-4c2b-8c32-570ec33285d3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9fbc563c-bde6-4c2b-8c32-570ec33285d3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:27.156476 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.156041 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9fbc563c-bde6-4c2b-8c32-570ec33285d3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"9fbc563c-bde6-4c2b-8c32-570ec33285d3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:27.156476 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.156215 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-bbb0deae\"" Oct 30 03:08:27.158214 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.158191 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:08:27.159956 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.159911 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" podStartSLOduration=1.159897936 podStartE2EDuration="1.159897936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:27.158740272 +0000 UTC m=+1668.172018374" watchObservedRunningTime="2026-10-30 03:08:27.159897936 +0000 UTC m=+1668.173176038" Oct 30 03:08:27.257167 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.257125 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwb8l\" (UniqueName: \"kubernetes.io/projected/9fbc563c-bde6-4c2b-8c32-570ec33285d3-kube-api-access-rwb8l\") pod \"data-science-collector-collector-0\" (UID: \"9fbc563c-bde6-4c2b-8c32-570ec33285d3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:27.257297 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.257175 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9fbc563c-bde6-4c2b-8c32-570ec33285d3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9fbc563c-bde6-4c2b-8c32-570ec33285d3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:27.257297 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.257209 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9fbc563c-bde6-4c2b-8c32-570ec33285d3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"9fbc563c-bde6-4c2b-8c32-570ec33285d3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:27.257297 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.257286 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9fbc563c-bde6-4c2b-8c32-570ec33285d3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9fbc563c-bde6-4c2b-8c32-570ec33285d3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:27.257858 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.257832 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9fbc563c-bde6-4c2b-8c32-570ec33285d3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9fbc563c-bde6-4c2b-8c32-570ec33285d3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:27.259725 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.259701 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9fbc563c-bde6-4c2b-8c32-570ec33285d3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9fbc563c-bde6-4c2b-8c32-570ec33285d3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:27.259901 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.259883 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9fbc563c-bde6-4c2b-8c32-570ec33285d3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"9fbc563c-bde6-4c2b-8c32-570ec33285d3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:27.286123 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.286093 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwb8l\" (UniqueName: \"kubernetes.io/projected/9fbc563c-bde6-4c2b-8c32-570ec33285d3-kube-api-access-rwb8l\") pod \"data-science-collector-collector-0\" (UID: \"9fbc563c-bde6-4c2b-8c32-570ec33285d3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:27.438922 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.438839 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:08:27.749724 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.749644 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:08:27.753009 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:08:27.752978 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbc563c_bde6_4c2b_8c32_570ec33285d3.slice/crio-6c27933a18d2512f4e8a5211dc5f26542310161c12f59880f671b4d84f5a1bb1 WatchSource:0}: Error finding container 6c27933a18d2512f4e8a5211dc5f26542310161c12f59880f671b4d84f5a1bb1: Status 404 returned error can't find the container with id 6c27933a18d2512f4e8a5211dc5f26542310161c12f59880f671b4d84f5a1bb1 Oct 30 03:08:27.755190 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.755159 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:08:27.755256 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:27.755228 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:08:28.118075 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:28.118037 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9fbc563c-bde6-4c2b-8c32-570ec33285d3","Type":"ContainerStarted","Data":"975cc315a46a9ac5f0daca6764280be0d01ff1439384cd39dec933aa6d23cfee"} Oct 30 03:08:28.118245 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:28.118101 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9fbc563c-bde6-4c2b-8c32-570ec33285d3","Type":"ContainerStarted","Data":"6c27933a18d2512f4e8a5211dc5f26542310161c12f59880f671b4d84f5a1bb1"} Oct 30 03:08:28.161627 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:28.161568 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.16155419 podStartE2EDuration="1.16155419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:28.160969376 +0000 UTC m=+1669.174247482" watchObservedRunningTime="2026-10-30 03:08:28.16155419 +0000 UTC m=+1669.174832293" Oct 30 03:08:38.120203 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:08:38.120168 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" Oct 30 03:08:48.161640 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:08:48.161593 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:09:18.164192 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:09:18.164153 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:09:48.161764 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:09:48.161709 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:10:18.161704 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:10:18.161658 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:10:39.267654 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:10:39.267544 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:10:39.276459 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:10:39.276438 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:10:39.278116 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:10:39.278097 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:10:39.287354 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:10:39.287338 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:10:48.160296 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:10:48.160263 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:11:18.161494 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:11:18.161455 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:11:48.162072 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:11:48.162030 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:12:18.161501 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:12:18.161459 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:12:48.161788 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:12:48.161719 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:13:18.160953 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:13:18.160913 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:13:48.160805 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:13:48.160769 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:14:18.160987 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:14:18.160953 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:14:48.161082 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:14:48.160997 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:15:18.161350 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:15:18.161308 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:15:39.295929 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:15:39.295816 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:15:39.311179 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:15:39.311151 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:15:39.315990 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:15:39.315971 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:15:39.324122 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:15:39.324105 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:15:48.161213 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:15:48.161151 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:16:18.162097 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:16:18.162051 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:16:48.160701 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:16:48.160661 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:17:18.161220 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:17:18.161185 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:17:48.161336 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:17:48.161258 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:18:18.161317 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:18:18.161278 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:18:40.892325 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:40.892290 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5c7449969d-84wnm"] Oct 30 03:18:40.892922 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:40.892488 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" podUID="5dd04619-86fd-4bd2-ad4b-1e81fbbeb190" containerName="korrel8r" containerID="cri-o://f06616e68e5ae17850b7f7f03c95e4337a076eab4d8612f8ba68a8c8e895bd15" gracePeriod=30 Oct 30 03:18:41.136480 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:41.136455 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" Oct 30 03:18:41.230069 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:41.229975 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-kube-api-access-jhx75\" (UniqueName: \"kubernetes.io/projected/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-kube-api-access-jhx75\") pod \"5dd04619-86fd-4bd2-ad4b-1e81fbbeb190\" (UID: \"5dd04619-86fd-4bd2-ad4b-1e81fbbeb190\") " Oct 30 03:18:41.230069 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:41.230046 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-tls\" (UniqueName: \"kubernetes.io/secret/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-tls\") pod \"5dd04619-86fd-4bd2-ad4b-1e81fbbeb190\" (UID: \"5dd04619-86fd-4bd2-ad4b-1e81fbbeb190\") " Oct 30 03:18:41.230298 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:41.230140 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-config\" (UniqueName: \"kubernetes.io/configmap/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-config\") pod \"5dd04619-86fd-4bd2-ad4b-1e81fbbeb190\" (UID: \"5dd04619-86fd-4bd2-ad4b-1e81fbbeb190\") " Oct 30 03:18:41.230507 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:41.230477 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-config" pod "5dd04619-86fd-4bd2-ad4b-1e81fbbeb190" (UID: "5dd04619-86fd-4bd2-ad4b-1e81fbbeb190"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:18:41.232114 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:41.232086 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-tls" pod "5dd04619-86fd-4bd2-ad4b-1e81fbbeb190" (UID: "5dd04619-86fd-4bd2-ad4b-1e81fbbeb190"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:18:41.232506 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:41.232481 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-kube-api-access-jhx75" pod "5dd04619-86fd-4bd2-ad4b-1e81fbbeb190" (UID: "5dd04619-86fd-4bd2-ad4b-1e81fbbeb190"). InnerVolumeSpecName "kube-api-access-jhx75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:18:41.330722 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:41.330682 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:18:41.330722 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:41.330714 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhx75\" (UniqueName: \"kubernetes.io/projected/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-kube-api-access-jhx75\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:18:41.330722 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:41.330724 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:18:41.883518 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:41.883484 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:18:41.883791 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:41.883724 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="9fbc563c-bde6-4c2b-8c32-570ec33285d3" containerName="otc-container" containerID="cri-o://975cc315a46a9ac5f0daca6764280be0d01ff1439384cd39dec933aa6d23cfee" gracePeriod=30 Oct 30 03:18:41.959057 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:41.959021 4925 generic.go:356] "Generic (PLEG): container finished" podID="5dd04619-86fd-4bd2-ad4b-1e81fbbeb190" containerID="f06616e68e5ae17850b7f7f03c95e4337a076eab4d8612f8ba68a8c8e895bd15" exitCode=0 Oct 30 03:18:41.959429 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:41.959068 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" Oct 30 03:18:41.959429 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:41.959103 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" event={"ID":"5dd04619-86fd-4bd2-ad4b-1e81fbbeb190","Type":"ContainerDied","Data":"f06616e68e5ae17850b7f7f03c95e4337a076eab4d8612f8ba68a8c8e895bd15"} Oct 30 03:18:41.959429 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:41.959161 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5c7449969d-84wnm" event={"ID":"5dd04619-86fd-4bd2-ad4b-1e81fbbeb190","Type":"ContainerDied","Data":"624a43ec558567dff7b6ec12613ceb5444bab502874debabf703df6eb0221d00"} Oct 30 03:18:41.959429 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:41.959185 4925 scope.go:122] "RemoveContainer" containerID="f06616e68e5ae17850b7f7f03c95e4337a076eab4d8612f8ba68a8c8e895bd15" Oct 30 03:18:41.996497 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:41.996472 4925 scope.go:122] "RemoveContainer" containerID="f06616e68e5ae17850b7f7f03c95e4337a076eab4d8612f8ba68a8c8e895bd15" Oct 30 03:18:41.996775 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:18:41.996740 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f06616e68e5ae17850b7f7f03c95e4337a076eab4d8612f8ba68a8c8e895bd15\": container with ID starting with f06616e68e5ae17850b7f7f03c95e4337a076eab4d8612f8ba68a8c8e895bd15 not found: ID does not exist" containerID="f06616e68e5ae17850b7f7f03c95e4337a076eab4d8612f8ba68a8c8e895bd15" Oct 30 03:18:41.996835 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:41.996785 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f06616e68e5ae17850b7f7f03c95e4337a076eab4d8612f8ba68a8c8e895bd15"} err="failed to get container status \"f06616e68e5ae17850b7f7f03c95e4337a076eab4d8612f8ba68a8c8e895bd15\": rpc error: code = NotFound desc = could not find container \"f06616e68e5ae17850b7f7f03c95e4337a076eab4d8612f8ba68a8c8e895bd15\": container with ID starting with f06616e68e5ae17850b7f7f03c95e4337a076eab4d8612f8ba68a8c8e895bd15 not found: ID does not exist" Oct 30 03:18:42.018359 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.018327 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5c7449969d-84wnm"] Oct 30 03:18:42.021386 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.021363 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5c7449969d-84wnm"] Oct 30 03:18:42.125034 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.125008 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:18:42.236363 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.236295 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fbc563c-bde6-4c2b-8c32-570ec33285d3-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9fbc563c-bde6-4c2b-8c32-570ec33285d3-monitoring-tls-certs\") pod \"9fbc563c-bde6-4c2b-8c32-570ec33285d3\" (UID: \"9fbc563c-bde6-4c2b-8c32-570ec33285d3\") " Oct 30 03:18:42.236363 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.236359 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fbc563c-bde6-4c2b-8c32-570ec33285d3-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9fbc563c-bde6-4c2b-8c32-570ec33285d3-data-science-collector-ta-client-cert\") pod \"9fbc563c-bde6-4c2b-8c32-570ec33285d3\" (UID: \"9fbc563c-bde6-4c2b-8c32-570ec33285d3\") " Oct 30 03:18:42.236616 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.236404 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fbc563c-bde6-4c2b-8c32-570ec33285d3-otc-internal\" (UniqueName: \"kubernetes.io/configmap/9fbc563c-bde6-4c2b-8c32-570ec33285d3-otc-internal\") pod \"9fbc563c-bde6-4c2b-8c32-570ec33285d3\" (UID: \"9fbc563c-bde6-4c2b-8c32-570ec33285d3\") " Oct 30 03:18:42.236616 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.236444 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbc563c-bde6-4c2b-8c32-570ec33285d3-kube-api-access-rwb8l\" (UniqueName: \"kubernetes.io/projected/9fbc563c-bde6-4c2b-8c32-570ec33285d3-kube-api-access-rwb8l\") pod \"9fbc563c-bde6-4c2b-8c32-570ec33285d3\" (UID: \"9fbc563c-bde6-4c2b-8c32-570ec33285d3\") " Oct 30 03:18:42.236889 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.236860 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbc563c-bde6-4c2b-8c32-570ec33285d3-otc-internal" pod "9fbc563c-bde6-4c2b-8c32-570ec33285d3" (UID: "9fbc563c-bde6-4c2b-8c32-570ec33285d3"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:18:42.238681 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.238654 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbc563c-bde6-4c2b-8c32-570ec33285d3-data-science-collector-ta-client-cert" pod "9fbc563c-bde6-4c2b-8c32-570ec33285d3" (UID: "9fbc563c-bde6-4c2b-8c32-570ec33285d3"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:18:42.238853 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.238742 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbc563c-bde6-4c2b-8c32-570ec33285d3-kube-api-access-rwb8l" pod "9fbc563c-bde6-4c2b-8c32-570ec33285d3" (UID: "9fbc563c-bde6-4c2b-8c32-570ec33285d3"). InnerVolumeSpecName "kube-api-access-rwb8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:18:42.238853 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.238793 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbc563c-bde6-4c2b-8c32-570ec33285d3-monitoring-tls-certs" pod "9fbc563c-bde6-4c2b-8c32-570ec33285d3" (UID: "9fbc563c-bde6-4c2b-8c32-570ec33285d3"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:18:42.337165 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.337137 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9fbc563c-bde6-4c2b-8c32-570ec33285d3-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:18:42.337165 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.337163 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwb8l\" (UniqueName: \"kubernetes.io/projected/9fbc563c-bde6-4c2b-8c32-570ec33285d3-kube-api-access-rwb8l\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:18:42.337338 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.337173 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9fbc563c-bde6-4c2b-8c32-570ec33285d3-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:18:42.337338 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.337183 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9fbc563c-bde6-4c2b-8c32-570ec33285d3-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:18:42.965948 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.965913 4925 generic.go:356] "Generic (PLEG): container finished" podID="9fbc563c-bde6-4c2b-8c32-570ec33285d3" containerID="975cc315a46a9ac5f0daca6764280be0d01ff1439384cd39dec933aa6d23cfee" exitCode=0 Oct 30 03:18:42.966444 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.965968 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:18:42.966444 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.966007 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9fbc563c-bde6-4c2b-8c32-570ec33285d3","Type":"ContainerDied","Data":"975cc315a46a9ac5f0daca6764280be0d01ff1439384cd39dec933aa6d23cfee"} Oct 30 03:18:42.966444 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.966070 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9fbc563c-bde6-4c2b-8c32-570ec33285d3","Type":"ContainerDied","Data":"6c27933a18d2512f4e8a5211dc5f26542310161c12f59880f671b4d84f5a1bb1"} Oct 30 03:18:42.966444 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.966096 4925 scope.go:122] "RemoveContainer" containerID="975cc315a46a9ac5f0daca6764280be0d01ff1439384cd39dec933aa6d23cfee" Oct 30 03:18:42.975384 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.975369 4925 scope.go:122] "RemoveContainer" containerID="975cc315a46a9ac5f0daca6764280be0d01ff1439384cd39dec933aa6d23cfee" Oct 30 03:18:42.975620 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:18:42.975602 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"975cc315a46a9ac5f0daca6764280be0d01ff1439384cd39dec933aa6d23cfee\": container with ID starting with 975cc315a46a9ac5f0daca6764280be0d01ff1439384cd39dec933aa6d23cfee not found: ID does not exist" containerID="975cc315a46a9ac5f0daca6764280be0d01ff1439384cd39dec933aa6d23cfee" Oct 30 03:18:42.975670 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:42.975630 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"975cc315a46a9ac5f0daca6764280be0d01ff1439384cd39dec933aa6d23cfee"} err="failed to get container status \"975cc315a46a9ac5f0daca6764280be0d01ff1439384cd39dec933aa6d23cfee\": rpc error: code = NotFound desc = could not find container \"975cc315a46a9ac5f0daca6764280be0d01ff1439384cd39dec933aa6d23cfee\": container with ID starting with 975cc315a46a9ac5f0daca6764280be0d01ff1439384cd39dec933aa6d23cfee not found: ID does not exist" Oct 30 03:18:43.019725 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:43.019700 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:18:43.024075 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:43.024051 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:18:43.118694 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:43.118661 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dd04619-86fd-4bd2-ad4b-1e81fbbeb190" path="/var/lib/kubelet/pods/5dd04619-86fd-4bd2-ad4b-1e81fbbeb190/volumes" Oct 30 03:18:43.119073 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:43.119059 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbc563c-bde6-4c2b-8c32-570ec33285d3" path="/var/lib/kubelet/pods/9fbc563c-bde6-4c2b-8c32-570ec33285d3/volumes" Oct 30 03:18:48.163019 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:18:48.162982 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:18:48.434631 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.434540 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b"] Oct 30 03:18:48.434986 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.434966 4925 cpu_manager.go:414] "containerMap: removing container" podUID="9fbc563c-bde6-4c2b-8c32-570ec33285d3" containerName="otc-container" Oct 30 03:18:48.435081 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.434987 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbc563c-bde6-4c2b-8c32-570ec33285d3" containerName="otc-container" Oct 30 03:18:48.435081 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.435011 4925 cpu_manager.go:414] "containerMap: removing container" podUID="5dd04619-86fd-4bd2-ad4b-1e81fbbeb190" containerName="korrel8r" Oct 30 03:18:48.435081 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.435019 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd04619-86fd-4bd2-ad4b-1e81fbbeb190" containerName="korrel8r" Oct 30 03:18:48.435230 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.435113 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbc563c-bde6-4c2b-8c32-570ec33285d3" containerName="otc-container" Oct 30 03:18:48.435230 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.435128 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd04619-86fd-4bd2-ad4b-1e81fbbeb190" containerName="korrel8r" Oct 30 03:18:48.440363 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.440336 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" Oct 30 03:18:48.451013 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.450987 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn"] Oct 30 03:18:48.454177 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.454157 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" Oct 30 03:18:48.470508 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.470466 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 03:18:48.470666 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.470619 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 03:18:48.470738 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.470730 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 03:18:48.470859 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.470851 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-9wh74\"" Oct 30 03:18:48.470973 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.470958 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 03:18:48.473145 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.473115 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b"] Oct 30 03:18:48.475292 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.475271 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn"] Oct 30 03:18:48.489549 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.489520 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwmb4\" (UniqueName: \"kubernetes.io/projected/61f0ff87-430a-47cc-bc65-72491b15a41b-kube-api-access-hwmb4\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-c5c2b\" (UID: \"61f0ff87-430a-47cc-bc65-72491b15a41b\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" Oct 30 03:18:48.489633 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.489582 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/61f0ff87-430a-47cc-bc65-72491b15a41b-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-c5c2b\" (UID: \"61f0ff87-430a-47cc-bc65-72491b15a41b\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" Oct 30 03:18:48.489701 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.489678 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61f0ff87-430a-47cc-bc65-72491b15a41b-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-c5c2b\" (UID: \"61f0ff87-430a-47cc-bc65-72491b15a41b\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" Oct 30 03:18:48.561453 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.561425 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 03:18:48.561453 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.561429 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 03:18:48.561453 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.561451 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-qsdhb\"" Oct 30 03:18:48.590983 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.590956 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwmb4\" (UniqueName: \"kubernetes.io/projected/61f0ff87-430a-47cc-bc65-72491b15a41b-kube-api-access-hwmb4\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-c5c2b\" (UID: \"61f0ff87-430a-47cc-bc65-72491b15a41b\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" Oct 30 03:18:48.591106 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.591042 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/451685d8-55ce-4734-84b5-7faa42ca1702-tls\") pod \"korrel8r-5cbcc89b8f-hgsgn\" (UID: \"451685d8-55ce-4734-84b5-7faa42ca1702\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" Oct 30 03:18:48.591106 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.591070 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8mhj\" (UniqueName: \"kubernetes.io/projected/451685d8-55ce-4734-84b5-7faa42ca1702-kube-api-access-h8mhj\") pod \"korrel8r-5cbcc89b8f-hgsgn\" (UID: \"451685d8-55ce-4734-84b5-7faa42ca1702\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" Oct 30 03:18:48.591106 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.591098 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/61f0ff87-430a-47cc-bc65-72491b15a41b-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-c5c2b\" (UID: \"61f0ff87-430a-47cc-bc65-72491b15a41b\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" Oct 30 03:18:48.591272 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.591197 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61f0ff87-430a-47cc-bc65-72491b15a41b-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-c5c2b\" (UID: \"61f0ff87-430a-47cc-bc65-72491b15a41b\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" Oct 30 03:18:48.591272 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.591249 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/451685d8-55ce-4734-84b5-7faa42ca1702-config\") pod \"korrel8r-5cbcc89b8f-hgsgn\" (UID: \"451685d8-55ce-4734-84b5-7faa42ca1702\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" Oct 30 03:18:48.591861 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.591842 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61f0ff87-430a-47cc-bc65-72491b15a41b-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-c5c2b\" (UID: \"61f0ff87-430a-47cc-bc65-72491b15a41b\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" Oct 30 03:18:48.593540 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.593519 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/61f0ff87-430a-47cc-bc65-72491b15a41b-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-c5c2b\" (UID: \"61f0ff87-430a-47cc-bc65-72491b15a41b\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" Oct 30 03:18:48.614616 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.614590 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwmb4\" (UniqueName: \"kubernetes.io/projected/61f0ff87-430a-47cc-bc65-72491b15a41b-kube-api-access-hwmb4\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-c5c2b\" (UID: \"61f0ff87-430a-47cc-bc65-72491b15a41b\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" Oct 30 03:18:48.692169 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.692093 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/451685d8-55ce-4734-84b5-7faa42ca1702-tls\") pod \"korrel8r-5cbcc89b8f-hgsgn\" (UID: \"451685d8-55ce-4734-84b5-7faa42ca1702\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" Oct 30 03:18:48.692169 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.692125 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8mhj\" (UniqueName: \"kubernetes.io/projected/451685d8-55ce-4734-84b5-7faa42ca1702-kube-api-access-h8mhj\") pod \"korrel8r-5cbcc89b8f-hgsgn\" (UID: \"451685d8-55ce-4734-84b5-7faa42ca1702\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" Oct 30 03:18:48.692169 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.692155 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/451685d8-55ce-4734-84b5-7faa42ca1702-config\") pod \"korrel8r-5cbcc89b8f-hgsgn\" (UID: \"451685d8-55ce-4734-84b5-7faa42ca1702\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" Oct 30 03:18:48.692982 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.692962 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/451685d8-55ce-4734-84b5-7faa42ca1702-config\") pod \"korrel8r-5cbcc89b8f-hgsgn\" (UID: \"451685d8-55ce-4734-84b5-7faa42ca1702\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" Oct 30 03:18:48.694536 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.694512 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/451685d8-55ce-4734-84b5-7faa42ca1702-tls\") pod \"korrel8r-5cbcc89b8f-hgsgn\" (UID: \"451685d8-55ce-4734-84b5-7faa42ca1702\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" Oct 30 03:18:48.716165 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.716144 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8mhj\" (UniqueName: \"kubernetes.io/projected/451685d8-55ce-4734-84b5-7faa42ca1702-kube-api-access-h8mhj\") pod \"korrel8r-5cbcc89b8f-hgsgn\" (UID: \"451685d8-55ce-4734-84b5-7faa42ca1702\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" Oct 30 03:18:48.756072 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.756047 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" Oct 30 03:18:48.774169 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:48.774147 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" Oct 30 03:18:49.057913 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:49.057722 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b"] Oct 30 03:18:49.060107 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:18:49.060078 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f0ff87_430a_47cc_bc65_72491b15a41b.slice/crio-5f856ce46f20d45651692b2b045ffed853e52e07c87a78261aa344da89ee87c0 WatchSource:0}: Error finding container 5f856ce46f20d45651692b2b045ffed853e52e07c87a78261aa344da89ee87c0: Status 404 returned error can't find the container with id 5f856ce46f20d45651692b2b045ffed853e52e07c87a78261aa344da89ee87c0 Oct 30 03:18:49.062036 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:49.062017 4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:18:49.089996 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:49.089896 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn"] Oct 30 03:18:49.092146 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:18:49.092118 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451685d8_55ce_4734_84b5_7faa42ca1702.slice/crio-0f101f6e7d83de4924fb2da752a0a34d926762211f9ef5396724243558934312 WatchSource:0}: Error finding container 0f101f6e7d83de4924fb2da752a0a34d926762211f9ef5396724243558934312: Status 404 returned error can't find the container with id 0f101f6e7d83de4924fb2da752a0a34d926762211f9ef5396724243558934312 Oct 30 03:18:50.006203 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.005817 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" event={"ID":"451685d8-55ce-4734-84b5-7faa42ca1702","Type":"ContainerStarted","Data":"e2d0125e5fed51e5621f54126b04994da1dae6ef7dcc2abc43fff42a43529600"} Oct 30 03:18:50.006203 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.005883 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" event={"ID":"451685d8-55ce-4734-84b5-7faa42ca1702","Type":"ContainerStarted","Data":"0f101f6e7d83de4924fb2da752a0a34d926762211f9ef5396724243558934312"} Oct 30 03:18:50.006203 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.006002 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" Oct 30 03:18:50.009825 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.009790 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" event={"ID":"61f0ff87-430a-47cc-bc65-72491b15a41b","Type":"ContainerStarted","Data":"5f856ce46f20d45651692b2b045ffed853e52e07c87a78261aa344da89ee87c0"} Oct 30 03:18:50.019858 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.019824 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:18:50.023541 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.023513 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:18:50.042399 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.042373 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 03:18:50.043160 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.043135 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 03:18:50.055145 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.055111 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-txljs\"" Oct 30 03:18:50.055305 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.055233 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 03:18:50.055305 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.055286 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:18:50.058420 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.058392 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:18:50.059968 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.059917 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" podStartSLOduration=2.059899898 podStartE2EDuration="2.059899898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:50.058796159 +0000 UTC m=+2291.072074264" watchObservedRunningTime="2026-10-30 03:18:50.059899898 +0000 UTC m=+2291.073178004" Oct 30 03:18:50.106947 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.106417 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/df5e38c1-d61f-4e87-b854-6ca066cd9818-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"df5e38c1-d61f-4e87-b854-6ca066cd9818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:18:50.106947 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.106460 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"df5e38c1-d61f-4e87-b854-6ca066cd9818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:18:50.106947 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.106491 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gknq\" (UniqueName: \"kubernetes.io/projected/df5e38c1-d61f-4e87-b854-6ca066cd9818-kube-api-access-5gknq\") pod \"data-science-collector-collector-0\" (UID: \"df5e38c1-d61f-4e87-b854-6ca066cd9818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:18:50.106947 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.106573 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"df5e38c1-d61f-4e87-b854-6ca066cd9818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:18:50.106947 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.106646 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"df5e38c1-d61f-4e87-b854-6ca066cd9818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:18:50.207450 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.207407 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"df5e38c1-d61f-4e87-b854-6ca066cd9818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:18:50.207612 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.207459 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/df5e38c1-d61f-4e87-b854-6ca066cd9818-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"df5e38c1-d61f-4e87-b854-6ca066cd9818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:18:50.207612 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.207481 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"df5e38c1-d61f-4e87-b854-6ca066cd9818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:18:50.207612 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.207508 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gknq\" (UniqueName: \"kubernetes.io/projected/df5e38c1-d61f-4e87-b854-6ca066cd9818-kube-api-access-5gknq\") pod \"data-science-collector-collector-0\" (UID: \"df5e38c1-d61f-4e87-b854-6ca066cd9818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:18:50.207612 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.207546 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"df5e38c1-d61f-4e87-b854-6ca066cd9818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:18:50.208659 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.208628 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/df5e38c1-d61f-4e87-b854-6ca066cd9818-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"df5e38c1-d61f-4e87-b854-6ca066cd9818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:18:50.210636 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.210613 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"df5e38c1-d61f-4e87-b854-6ca066cd9818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:18:50.211080 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.211063 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"df5e38c1-d61f-4e87-b854-6ca066cd9818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:18:50.211298 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.211274 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"df5e38c1-d61f-4e87-b854-6ca066cd9818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:18:50.232163 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.232136 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gknq\" (UniqueName: \"kubernetes.io/projected/df5e38c1-d61f-4e87-b854-6ca066cd9818-kube-api-access-5gknq\") pod \"data-science-collector-collector-0\" (UID: \"df5e38c1-d61f-4e87-b854-6ca066cd9818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:18:50.344849 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.344816 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:18:50.660853 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.660820 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:18:50.663527 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:18:50.663494 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5e38c1_d61f_4e87_b854_6ca066cd9818.slice/crio-b3e4178732fadca6c23f9a91b4a1ba14bf12eb63f8b48a8e46f610f5d8848bf4 WatchSource:0}: Error finding container b3e4178732fadca6c23f9a91b4a1ba14bf12eb63f8b48a8e46f610f5d8848bf4: Status 404 returned error can't find the container with id b3e4178732fadca6c23f9a91b4a1ba14bf12eb63f8b48a8e46f610f5d8848bf4 Oct 30 03:18:50.669775 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.669717 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:18:50.669893 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:50.669821 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:18:51.017744 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:51.017664 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"df5e38c1-d61f-4e87-b854-6ca066cd9818","Type":"ContainerStarted","Data":"91a3ea3330644a0d424b90e82c2471e9f8f91d1fd2bae00f6ce480544aff534b"} Oct 30 03:18:51.017744 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:51.017726 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"df5e38c1-d61f-4e87-b854-6ca066cd9818","Type":"ContainerStarted","Data":"b3e4178732fadca6c23f9a91b4a1ba14bf12eb63f8b48a8e46f610f5d8848bf4"} Oct 30 03:18:51.063090 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:51.062987 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.062972719 podStartE2EDuration="2.062972719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:51.061450219 +0000 UTC m=+2292.074728326" watchObservedRunningTime="2026-10-30 03:18:51.062972719 +0000 UTC m=+2292.076250821" Oct 30 03:18:52.023706 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:52.023616 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" event={"ID":"61f0ff87-430a-47cc-bc65-72491b15a41b","Type":"ContainerStarted","Data":"33d2de93d1e9b82c8c2a7832f241c865f093ee767c761733c50ae00232ad3951"} Oct 30 03:18:53.030283 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.030202 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" event={"ID":"61f0ff87-430a-47cc-bc65-72491b15a41b","Type":"ContainerStarted","Data":"6a7a608125ca2e1f95a09e20a932f8a095db66bf34675112348bf31cae2c7ebf"} Oct 30 03:18:53.072991 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.072938 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" podStartSLOduration=1.572352838 podStartE2EDuration="5.072924551s" totalImagesPullingTime="3.500571713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:53.071434737 +0000 UTC m=+2294.084712840" watchObservedRunningTime="2026-10-30 03:18:53.072924551 +0000 UTC m=+2294.086202653" Oct 30 03:18:53.720522 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.720486 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 03:18:53.727137 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.727106 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.744670 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.744637 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 03:18:53.744815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.744713 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 03:18:53.744815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.744718 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 03:18:53.744928 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.744906 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-zdtts\"" Oct 30 03:18:53.757382 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.757357 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 03:18:53.759405 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.759384 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 03:18:53.841839 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.841807 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6d855555-576b-481b-8f4b-05318df0c932-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.841968 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.841845 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d855555-576b-481b-8f4b-05318df0c932-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.841968 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.841867 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.841968 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.841937 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d855555-576b-481b-8f4b-05318df0c932-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.842137 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.842088 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dt6l\" (UniqueName: \"kubernetes.io/projected/6d855555-576b-481b-8f4b-05318df0c932-kube-api-access-9dt6l\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.842199 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.842153 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.842254 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.842238 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.943066 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.943034 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.943204 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.943096 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.943204 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.943137 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6d855555-576b-481b-8f4b-05318df0c932-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.943204 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.943170 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d855555-576b-481b-8f4b-05318df0c932-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.943204 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.943192 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.943421 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.943219 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d855555-576b-481b-8f4b-05318df0c932-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.943421 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.943260 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dt6l\" (UniqueName: \"kubernetes.io/projected/6d855555-576b-481b-8f4b-05318df0c932-kube-api-access-9dt6l\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.943616 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.943579 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6d855555-576b-481b-8f4b-05318df0c932-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.945595 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.945566 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d855555-576b-481b-8f4b-05318df0c932-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.945717 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.945697 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.945809 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.945715 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.946570 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.946552 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d855555-576b-481b-8f4b-05318df0c932-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.946652 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.946569 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:53.966711 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:53.966687 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dt6l\" (UniqueName: \"kubernetes.io/projected/6d855555-576b-481b-8f4b-05318df0c932-kube-api-access-9dt6l\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:54.053258 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:54.053195 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:18:54.388222 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:54.388189 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 03:18:54.390115 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:18:54.390092 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d855555_576b_481b_8f4b_05318df0c932.slice/crio-a7a98458cf3e8aec97e09fba2670f48ccc6e2850f39746198a32fa070871f3bf WatchSource:0}: Error finding container a7a98458cf3e8aec97e09fba2670f48ccc6e2850f39746198a32fa070871f3bf: Status 404 returned error can't find the container with id a7a98458cf3e8aec97e09fba2670f48ccc6e2850f39746198a32fa070871f3bf Oct 30 03:18:55.041665 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:55.041618 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6d855555-576b-481b-8f4b-05318df0c932","Type":"ContainerStarted","Data":"a7a98458cf3e8aec97e09fba2670f48ccc6e2850f39746198a32fa070871f3bf"} Oct 30 03:18:56.047924 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:56.047885 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6d855555-576b-481b-8f4b-05318df0c932","Type":"ContainerStarted","Data":"3060e95a8efb9c7898188af81c9cf2e3bed2f7681490184b6e18871e109e0f92"} Oct 30 03:18:56.901367 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:56.901331 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 03:18:56.905246 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:56.905228 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:56.921221 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:56.921195 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 03:18:56.973346 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:56.973319 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 03:18:56.973595 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:56.973581 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 03:18:56.977210 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:56.977158 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 03:18:56.977210 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:56.977163 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 03:18:56.977210 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:56.977176 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 03:18:56.977210 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:56.977202 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 03:18:56.977474 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:56.977215 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 03:18:56.977474 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:56.977259 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 03:18:56.977474 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:56.977304 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-mgknz\"" Oct 30 03:18:56.977474 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:56.977344 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 03:18:57.071078 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.071051 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.071476 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.071085 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.071476 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.071110 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.071476 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.071164 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.071476 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.071277 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3032256-3634-450c-b706-bf9d6e2df813-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.071476 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.071328 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.071476 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.071366 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.071476 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.071399 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.071476 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.071439 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.071476 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.071473 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d274r\" (UniqueName: \"kubernetes.io/projected/a3032256-3634-450c-b706-bf9d6e2df813-kube-api-access-d274r\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.071817 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.071559 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3032256-3634-450c-b706-bf9d6e2df813-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.071817 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.071578 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.172261 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.172190 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.172261 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.172234 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d274r\" (UniqueName: \"kubernetes.io/projected/a3032256-3634-450c-b706-bf9d6e2df813-kube-api-access-d274r\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.172423 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.172288 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3032256-3634-450c-b706-bf9d6e2df813-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.172423 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.172314 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.172423 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.172392 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.172581 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.172450 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.172581 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.172496 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.172581 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.172519 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.172740 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.172594 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3032256-3634-450c-b706-bf9d6e2df813-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.172740 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.172625 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.172740 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.172670 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.172947 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.172746 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.173464 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.173435 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.173637 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.173511 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.174292 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.174268 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.175429 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.175397 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.175525 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.175421 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.175592 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.175535 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.175721 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.175703 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.175886 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.175869 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3032256-3634-450c-b706-bf9d6e2df813-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.175951 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.175918 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.176333 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.176315 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3032256-3634-450c-b706-bf9d6e2df813-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.176956 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.176935 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:18:57.195840 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:18:57.195818 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d274r\" (UniqueName: \"kubernetes.io/projected/a3032256-3634-450c-b706-bf9d6e2df813-kube-api-access-d274r\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:19:00.072807 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:00.072769 4925 generic.go:356] "Generic (PLEG): container finished" podID="6d855555-576b-481b-8f4b-05318df0c932" containerID="3060e95a8efb9c7898188af81c9cf2e3bed2f7681490184b6e18871e109e0f92" exitCode=0 Oct 30 03:19:00.073344 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:00.072833 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6d855555-576b-481b-8f4b-05318df0c932","Type":"ContainerDied","Data":"3060e95a8efb9c7898188af81c9cf2e3bed2f7681490184b6e18871e109e0f92"} Oct 30 03:19:00.604821 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:00.604785 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:19:00.634379 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:00.634345 4925 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:19:00.705343 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:00.705302 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:19:00.705495 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:00.705440 4925 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:19:00.719944 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:00.719915 4925 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") DevicePath \"csi-ec2ccb9dcc2629e5a59ea91641dbb6b15d80e87697977b7c41508ff0003b2f36\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:19:00.914548 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:00.914519 4925 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a535375d22e374ba333968465471c7604a06f9d95a513910fe4bb083b629e986/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:19:00.941314 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:00.941284 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:19:01.020320 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:01.020291 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" Oct 30 03:19:01.125773 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:01.125722 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:19:01.475151 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:01.475123 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 03:19:01.477182 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:19:01.477156 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3032256_3634_450c_b706_bf9d6e2df813.slice/crio-caa78a2c9a43ccf26a2e264396ab21b90a4e619cb4a89b498aba9cb3dc4f76f5 WatchSource:0}: Error finding container caa78a2c9a43ccf26a2e264396ab21b90a4e619cb4a89b498aba9cb3dc4f76f5: Status 404 returned error can't find the container with id caa78a2c9a43ccf26a2e264396ab21b90a4e619cb4a89b498aba9cb3dc4f76f5 Oct 30 03:19:02.088861 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:02.088785 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6d855555-576b-481b-8f4b-05318df0c932","Type":"ContainerStarted","Data":"b7cc879bb16bfca0b4e95c290d181690b6387c2251b89bae2ddbada9421d42b9"} Oct 30 03:19:02.088861 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:02.088855 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6d855555-576b-481b-8f4b-05318df0c932","Type":"ContainerStarted","Data":"506a8df06b70f2d87c86a1a03533bc36fec25983dbbc869877d6a076efe5e4cd"} Oct 30 03:19:02.089101 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:02.089073 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:19:02.090444 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:02.090418 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a3032256-3634-450c-b706-bf9d6e2df813","Type":"ContainerStarted","Data":"d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531"} Oct 30 03:19:02.090549 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:02.090469 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a3032256-3634-450c-b706-bf9d6e2df813","Type":"ContainerStarted","Data":"caa78a2c9a43ccf26a2e264396ab21b90a4e619cb4a89b498aba9cb3dc4f76f5"} Oct 30 03:19:02.152475 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:02.152419 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.425921615 podStartE2EDuration="9.152406716s" totalImagesPullingTime="2.538964133s" totalInitContainerRuntime="3.187520968s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:02.138441083 +0000 UTC m=+2303.151719187" watchObservedRunningTime="2026-10-30 03:19:02.152406716 +0000 UTC m=+2303.165684818" Oct 30 03:19:06.110470 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:06.110434 4925 generic.go:356] "Generic (PLEG): container finished" podID="a3032256-3634-450c-b706-bf9d6e2df813" containerID="d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531" exitCode=0 Oct 30 03:19:06.110900 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:06.110504 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a3032256-3634-450c-b706-bf9d6e2df813","Type":"ContainerDied","Data":"d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531"} Oct 30 03:19:08.095746 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:08.095717 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:19:10.132697 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:10.132655 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a3032256-3634-450c-b706-bf9d6e2df813","Type":"ContainerStarted","Data":"d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08"} Oct 30 03:19:10.133059 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:10.132721 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a3032256-3634-450c-b706-bf9d6e2df813","Type":"ContainerStarted","Data":"24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2"} Oct 30 03:19:12.144795 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:12.144740 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a3032256-3634-450c-b706-bf9d6e2df813","Type":"ContainerStarted","Data":"489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a"} Oct 30 03:19:12.238864 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:12.238815 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=10.875068201 podStartE2EDuration="19.238800298s" totalImagesPullingTime="5.078911178s" totalInitContainerRuntime="3.284820919s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:18:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:12.225440907 +0000 UTC m=+2313.238719011" watchObservedRunningTime="2026-10-30 03:19:12.238800298 +0000 UTC m=+2313.252078401" Oct 30 03:19:16.126524 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:16.126482 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:19:16.127013 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:16.126550 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:19:16.132035 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:16.132014 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:19:16.174774 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:19:16.174723 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:19:18.161503 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:19:18.161468 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:19:48.161271 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:19:48.161216 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:20:18.160627 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:20:18.160590 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:20:39.335218 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:20:39.335193 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:20:39.344418 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:20:39.344395 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:20:39.349701 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:20:39.349683 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:20:39.358180 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:20:39.358164 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:20:48.162533 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:20:48.162493 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:21:18.161302 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:21:18.161257 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:21:48.161264 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:21:48.161224 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:22:18.161078 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:22:18.161039 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:22:48.161581 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:22:48.161543 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:23:18.161376 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:23:18.161338 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:23:48.161654 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:23:48.161572 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:24:18.161054 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:24:18.161019 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:24:48.161945 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:24:48.161910 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:25:18.160801 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:25:18.160726 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:25:39.375287 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:25:39.375168 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:25:39.383710 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:25:39.383689 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:25:39.384545 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:25:39.384526 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:25:39.392656 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:25:39.392638 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:25:48.161417 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:25:48.161377 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:26:18.161645 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:26:18.161608 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:26:48.162204 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:26:48.162118 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:27:18.161601 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:27:18.161566 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:27:48.161182 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:27:48.161148 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:28:18.161655 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:28:18.161620 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:28:48.160206 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:28:48.160168 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:29:18.160700 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:29:18.160666 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:29:20.944108 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:20.944070 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b"] Oct 30 03:29:20.944518 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:20.944364 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" podUID="61f0ff87-430a-47cc-bc65-72491b15a41b" containerName="kube-rbac-proxy" containerID="cri-o://33d2de93d1e9b82c8c2a7832f241c865f093ee767c761733c50ae00232ad3951" gracePeriod=30 Oct 30 03:29:20.944518 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:20.944426 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" podUID="61f0ff87-430a-47cc-bc65-72491b15a41b" containerName="prom-label-proxy" containerID="cri-o://6a7a608125ca2e1f95a09e20a932f8a095db66bf34675112348bf31cae2c7ebf" gracePeriod=30 Oct 30 03:29:20.966968 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:20.959386 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn"] Oct 30 03:29:20.966968 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:20.959587 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" podUID="451685d8-55ce-4734-84b5-7faa42ca1702" containerName="korrel8r" containerID="cri-o://e2d0125e5fed51e5621f54126b04994da1dae6ef7dcc2abc43fff42a43529600" gracePeriod=30 Oct 30 03:29:21.018550 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.018512 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" podUID="451685d8-55ce-4734-84b5-7faa42ca1702" containerName="korrel8r" probeResult="failure" output="dial tcp 10.132.0.47:8443: connect: connection refused" Oct 30 03:29:21.090493 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.090453 4925 generic.go:356] "Generic (PLEG): container finished" podID="451685d8-55ce-4734-84b5-7faa42ca1702" containerID="e2d0125e5fed51e5621f54126b04994da1dae6ef7dcc2abc43fff42a43529600" exitCode=0 Oct 30 03:29:21.090631 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.090548 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" event={"ID":"451685d8-55ce-4734-84b5-7faa42ca1702","Type":"ContainerDied","Data":"e2d0125e5fed51e5621f54126b04994da1dae6ef7dcc2abc43fff42a43529600"} Oct 30 03:29:21.092963 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.092938 4925 generic.go:356] "Generic (PLEG): container finished" podID="61f0ff87-430a-47cc-bc65-72491b15a41b" containerID="6a7a608125ca2e1f95a09e20a932f8a095db66bf34675112348bf31cae2c7ebf" exitCode=0 Oct 30 03:29:21.093011 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.092965 4925 generic.go:356] "Generic (PLEG): container finished" podID="61f0ff87-430a-47cc-bc65-72491b15a41b" containerID="33d2de93d1e9b82c8c2a7832f241c865f093ee767c761733c50ae00232ad3951" exitCode=0 Oct 30 03:29:21.093052 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.093007 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" event={"ID":"61f0ff87-430a-47cc-bc65-72491b15a41b","Type":"ContainerDied","Data":"6a7a608125ca2e1f95a09e20a932f8a095db66bf34675112348bf31cae2c7ebf"} Oct 30 03:29:21.093142 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.093056 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" event={"ID":"61f0ff87-430a-47cc-bc65-72491b15a41b","Type":"ContainerDied","Data":"33d2de93d1e9b82c8c2a7832f241c865f093ee767c761733c50ae00232ad3951"} Oct 30 03:29:21.221602 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.221577 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" Oct 30 03:29:21.224267 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.224243 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" Oct 30 03:29:21.359280 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.359247 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61f0ff87-430a-47cc-bc65-72491b15a41b-config\" (UniqueName: \"kubernetes.io/configmap/61f0ff87-430a-47cc-bc65-72491b15a41b-config\") pod \"61f0ff87-430a-47cc-bc65-72491b15a41b\" (UID: \"61f0ff87-430a-47cc-bc65-72491b15a41b\") " Oct 30 03:29:21.359475 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.359329 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/451685d8-55ce-4734-84b5-7faa42ca1702-config\" (UniqueName: \"kubernetes.io/configmap/451685d8-55ce-4734-84b5-7faa42ca1702-config\") pod \"451685d8-55ce-4734-84b5-7faa42ca1702\" (UID: \"451685d8-55ce-4734-84b5-7faa42ca1702\") " Oct 30 03:29:21.359475 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.359370 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/451685d8-55ce-4734-84b5-7faa42ca1702-kube-api-access-h8mhj\" (UniqueName: \"kubernetes.io/projected/451685d8-55ce-4734-84b5-7faa42ca1702-kube-api-access-h8mhj\") pod \"451685d8-55ce-4734-84b5-7faa42ca1702\" (UID: \"451685d8-55ce-4734-84b5-7faa42ca1702\") " Oct 30 03:29:21.359475 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.359403 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61f0ff87-430a-47cc-bc65-72491b15a41b-tls\" (UniqueName: \"kubernetes.io/secret/61f0ff87-430a-47cc-bc65-72491b15a41b-tls\") pod \"61f0ff87-430a-47cc-bc65-72491b15a41b\" (UID: \"61f0ff87-430a-47cc-bc65-72491b15a41b\") " Oct 30 03:29:21.359621 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.359475 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f0ff87-430a-47cc-bc65-72491b15a41b-kube-api-access-hwmb4\" (UniqueName: \"kubernetes.io/projected/61f0ff87-430a-47cc-bc65-72491b15a41b-kube-api-access-hwmb4\") pod \"61f0ff87-430a-47cc-bc65-72491b15a41b\" (UID: \"61f0ff87-430a-47cc-bc65-72491b15a41b\") " Oct 30 03:29:21.359621 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.359507 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/451685d8-55ce-4734-84b5-7faa42ca1702-tls\" (UniqueName: \"kubernetes.io/secret/451685d8-55ce-4734-84b5-7faa42ca1702-tls\") pod \"451685d8-55ce-4734-84b5-7faa42ca1702\" (UID: \"451685d8-55ce-4734-84b5-7faa42ca1702\") " Oct 30 03:29:21.360203 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.360109 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f0ff87-430a-47cc-bc65-72491b15a41b-config" pod "61f0ff87-430a-47cc-bc65-72491b15a41b" (UID: "61f0ff87-430a-47cc-bc65-72491b15a41b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:29:21.360203 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.360154 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/451685d8-55ce-4734-84b5-7faa42ca1702-config" pod "451685d8-55ce-4734-84b5-7faa42ca1702" (UID: "451685d8-55ce-4734-84b5-7faa42ca1702"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:29:21.364065 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.364040 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f0ff87-430a-47cc-bc65-72491b15a41b-kube-api-access-hwmb4" pod "61f0ff87-430a-47cc-bc65-72491b15a41b" (UID: "61f0ff87-430a-47cc-bc65-72491b15a41b"). InnerVolumeSpecName "kube-api-access-hwmb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:29:21.364166 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.364041 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/451685d8-55ce-4734-84b5-7faa42ca1702-tls" pod "451685d8-55ce-4734-84b5-7faa42ca1702" (UID: "451685d8-55ce-4734-84b5-7faa42ca1702"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:21.364166 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.364070 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61f0ff87-430a-47cc-bc65-72491b15a41b-tls" pod "61f0ff87-430a-47cc-bc65-72491b15a41b" (UID: "61f0ff87-430a-47cc-bc65-72491b15a41b"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:21.364166 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.364133 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451685d8-55ce-4734-84b5-7faa42ca1702-kube-api-access-h8mhj" pod "451685d8-55ce-4734-84b5-7faa42ca1702" (UID: "451685d8-55ce-4734-84b5-7faa42ca1702"). InnerVolumeSpecName "kube-api-access-h8mhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:29:21.425422 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.425392 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 03:29:21.425683 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.425657 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="a3032256-3634-450c-b706-bf9d6e2df813" containerName="prometheus" containerID="cri-o://24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2" gracePeriod=600 Oct 30 03:29:21.425762 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.425695 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="a3032256-3634-450c-b706-bf9d6e2df813" containerName="config-reloader" containerID="cri-o://d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08" gracePeriod=600 Oct 30 03:29:21.425818 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.425701 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="a3032256-3634-450c-b706-bf9d6e2df813" containerName="thanos-sidecar" containerID="cri-o://489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a" gracePeriod=600 Oct 30 03:29:21.431054 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.431022 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 03:29:21.431352 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.431313 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6d855555-576b-481b-8f4b-05318df0c932" containerName="alertmanager" containerID="cri-o://506a8df06b70f2d87c86a1a03533bc36fec25983dbbc869877d6a076efe5e4cd" gracePeriod=120 Oct 30 03:29:21.431549 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.431444 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6d855555-576b-481b-8f4b-05318df0c932" containerName="config-reloader" containerID="cri-o://b7cc879bb16bfca0b4e95c290d181690b6387c2251b89bae2ddbada9421d42b9" gracePeriod=120 Oct 30 03:29:21.460398 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.460346 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/451685d8-55ce-4734-84b5-7faa42ca1702-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:21.460398 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.460367 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8mhj\" (UniqueName: \"kubernetes.io/projected/451685d8-55ce-4734-84b5-7faa42ca1702-kube-api-access-h8mhj\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:21.460398 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.460377 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/61f0ff87-430a-47cc-bc65-72491b15a41b-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:21.460398 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.460385 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwmb4\" (UniqueName: \"kubernetes.io/projected/61f0ff87-430a-47cc-bc65-72491b15a41b-kube-api-access-hwmb4\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:21.460398 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.460393 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/451685d8-55ce-4734-84b5-7faa42ca1702-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:21.460398 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.460402 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61f0ff87-430a-47cc-bc65-72491b15a41b-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:21.857495 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.857470 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:29:21.965223 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.965192 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3032256-3634-450c-b706-bf9d6e2df813-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3032256-3634-450c-b706-bf9d6e2df813-config-out\") pod \"a3032256-3634-450c-b706-bf9d6e2df813\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " Oct 30 03:29:21.965619 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.965267 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"a3032256-3634-450c-b706-bf9d6e2df813\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " Oct 30 03:29:21.965619 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.965307 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"a3032256-3634-450c-b706-bf9d6e2df813\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " Oct 30 03:29:21.965619 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.965363 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3032256-3634-450c-b706-bf9d6e2df813-kube-api-access-d274r\" (UniqueName: \"kubernetes.io/projected/a3032256-3634-450c-b706-bf9d6e2df813-kube-api-access-d274r\") pod \"a3032256-3634-450c-b706-bf9d6e2df813\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " Oct 30 03:29:21.965619 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.965394 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"a3032256-3634-450c-b706-bf9d6e2df813\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " Oct 30 03:29:21.965619 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.965445 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3032256-3634-450c-b706-bf9d6e2df813-tls-assets\" (UniqueName: \"kubernetes.io/projected/a3032256-3634-450c-b706-bf9d6e2df813-tls-assets\") pod \"a3032256-3634-450c-b706-bf9d6e2df813\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " Oct 30 03:29:21.965921 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.965674 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-2" pod "a3032256-3634-450c-b706-bf9d6e2df813" (UID: "a3032256-3634-450c-b706-bf9d6e2df813"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:29:21.965921 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.965735 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"a3032256-3634-450c-b706-bf9d6e2df813\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " Oct 30 03:29:21.965921 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.965814 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-thanos-prometheus-http-client-file\") pod \"a3032256-3634-450c-b706-bf9d6e2df813\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " Oct 30 03:29:21.965921 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.965851 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-secret-prometheus-web-tls-ca\") pod \"a3032256-3634-450c-b706-bf9d6e2df813\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " Oct 30 03:29:21.965921 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.965855 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-0" pod "a3032256-3634-450c-b706-bf9d6e2df813" (UID: "a3032256-3634-450c-b706-bf9d6e2df813"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:29:21.966167 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.965940 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"a3032256-3634-450c-b706-bf9d6e2df813\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " Oct 30 03:29:21.966167 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.965973 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-config\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-config\") pod \"a3032256-3634-450c-b706-bf9d6e2df813\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " Oct 30 03:29:21.966167 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.966004 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config\") pod \"a3032256-3634-450c-b706-bf9d6e2df813\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " Oct 30 03:29:21.966167 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.966041 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"a3032256-3634-450c-b706-bf9d6e2df813\" (UID: \"a3032256-3634-450c-b706-bf9d6e2df813\") " Oct 30 03:29:21.966800 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.966369 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:21.966800 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.966407 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:21.967416 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.967387 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-1" pod "a3032256-3634-450c-b706-bf9d6e2df813" (UID: "a3032256-3634-450c-b706-bf9d6e2df813"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:29:21.968289 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.968263 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3032256-3634-450c-b706-bf9d6e2df813-tls-assets" pod "a3032256-3634-450c-b706-bf9d6e2df813" (UID: "a3032256-3634-450c-b706-bf9d6e2df813"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:29:21.968402 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.968353 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "a3032256-3634-450c-b706-bf9d6e2df813" (UID: "a3032256-3634-450c-b706-bf9d6e2df813"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:21.968507 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.968408 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3032256-3634-450c-b706-bf9d6e2df813-config-out" pod "a3032256-3634-450c-b706-bf9d6e2df813" (UID: "a3032256-3634-450c-b706-bf9d6e2df813"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:29:21.968987 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.968946 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "a3032256-3634-450c-b706-bf9d6e2df813" (UID: "a3032256-3634-450c-b706-bf9d6e2df813"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:21.969194 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.969169 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3032256-3634-450c-b706-bf9d6e2df813-kube-api-access-d274r" pod "a3032256-3634-450c-b706-bf9d6e2df813" (UID: "a3032256-3634-450c-b706-bf9d6e2df813"). InnerVolumeSpecName "kube-api-access-d274r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:29:21.969194 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.969176 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-thanos-prometheus-http-client-file" pod "a3032256-3634-450c-b706-bf9d6e2df813" (UID: "a3032256-3634-450c-b706-bf9d6e2df813"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:21.969670 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.969647 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-secret-prometheus-web-tls-ca" pod "a3032256-3634-450c-b706-bf9d6e2df813" (UID: "a3032256-3634-450c-b706-bf9d6e2df813"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:21.970275 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.970244 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-config" pod "a3032256-3634-450c-b706-bf9d6e2df813" (UID: "a3032256-3634-450c-b706-bf9d6e2df813"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:21.973386 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.973367 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4" pod "a3032256-3634-450c-b706-bf9d6e2df813" (UID: "a3032256-3634-450c-b706-bf9d6e2df813"). InnerVolumeSpecName "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 03:29:21.983840 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:21.983786 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config" pod "a3032256-3634-450c-b706-bf9d6e2df813" (UID: "a3032256-3634-450c-b706-bf9d6e2df813"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:22.067714 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.067679 4925 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:22.067714 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.067716 4925 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-secret-prometheus-web-tls-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:22.067941 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.067772 4925 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-139-19.ec2.internal\" " Oct 30 03:29:22.067941 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.067787 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:22.067941 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.067801 4925 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:22.067941 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.067816 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3032256-3634-450c-b706-bf9d6e2df813-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:22.067941 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.067831 4925 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3032256-3634-450c-b706-bf9d6e2df813-config-out\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:22.067941 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.067845 4925 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:22.067941 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.067860 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d274r\" (UniqueName: \"kubernetes.io/projected/a3032256-3634-450c-b706-bf9d6e2df813-kube-api-access-d274r\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:22.067941 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.067872 4925 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3032256-3634-450c-b706-bf9d6e2df813-tls-assets\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:22.067941 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.067887 4925 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a3032256-3634-450c-b706-bf9d6e2df813-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:22.084426 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.084402 4925 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4") on node "ip-10-0-139-19.ec2.internal" Oct 30 03:29:22.094772 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.094719 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:29:22.095018 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.094989 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="df5e38c1-d61f-4e87-b854-6ca066cd9818" containerName="otc-container" containerID="cri-o://91a3ea3330644a0d424b90e82c2471e9f8f91d1fd2bae00f6ce480544aff534b" gracePeriod=30 Oct 30 03:29:22.102087 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.102063 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" Oct 30 03:29:22.105230 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.103848 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn" event={"ID":"451685d8-55ce-4734-84b5-7faa42ca1702","Type":"ContainerDied","Data":"0f101f6e7d83de4924fb2da752a0a34d926762211f9ef5396724243558934312"} Oct 30 03:29:22.105230 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.103909 4925 scope.go:122] "RemoveContainer" containerID="e2d0125e5fed51e5621f54126b04994da1dae6ef7dcc2abc43fff42a43529600" Oct 30 03:29:22.108515 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.108493 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" Oct 30 03:29:22.108606 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.108525 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b" event={"ID":"61f0ff87-430a-47cc-bc65-72491b15a41b","Type":"ContainerDied","Data":"5f856ce46f20d45651692b2b045ffed853e52e07c87a78261aa344da89ee87c0"} Oct 30 03:29:22.113179 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.113128 4925 generic.go:356] "Generic (PLEG): container finished" podID="6d855555-576b-481b-8f4b-05318df0c932" containerID="b7cc879bb16bfca0b4e95c290d181690b6387c2251b89bae2ddbada9421d42b9" exitCode=0 Oct 30 03:29:22.113292 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.113195 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6d855555-576b-481b-8f4b-05318df0c932","Type":"ContainerDied","Data":"b7cc879bb16bfca0b4e95c290d181690b6387c2251b89bae2ddbada9421d42b9"} Oct 30 03:29:22.116501 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.116476 4925 generic.go:356] "Generic (PLEG): container finished" podID="a3032256-3634-450c-b706-bf9d6e2df813" containerID="489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a" exitCode=0 Oct 30 03:29:22.116501 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.116500 4925 generic.go:356] "Generic (PLEG): container finished" podID="a3032256-3634-450c-b706-bf9d6e2df813" containerID="d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08" exitCode=0 Oct 30 03:29:22.116670 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.116510 4925 generic.go:356] "Generic (PLEG): container finished" podID="a3032256-3634-450c-b706-bf9d6e2df813" containerID="24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2" exitCode=0 Oct 30 03:29:22.116670 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.116556 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:29:22.116670 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.116581 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a3032256-3634-450c-b706-bf9d6e2df813","Type":"ContainerDied","Data":"489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a"} Oct 30 03:29:22.116670 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.116642 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a3032256-3634-450c-b706-bf9d6e2df813","Type":"ContainerDied","Data":"d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08"} Oct 30 03:29:22.116915 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.116699 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a3032256-3634-450c-b706-bf9d6e2df813","Type":"ContainerDied","Data":"24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2"} Oct 30 03:29:22.116915 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.116737 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a3032256-3634-450c-b706-bf9d6e2df813","Type":"ContainerDied","Data":"caa78a2c9a43ccf26a2e264396ab21b90a4e619cb4a89b498aba9cb3dc4f76f5"} Oct 30 03:29:22.169127 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.169091 4925 reconciler_common.go:299] "Volume detached for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"csi-ec2ccb9dcc2629e5a59ea91641dbb6b15d80e87697977b7c41508ff0003b2f36\"" Oct 30 03:29:22.183520 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.183489 4925 scope.go:122] "RemoveContainer" containerID="6a7a608125ca2e1f95a09e20a932f8a095db66bf34675112348bf31cae2c7ebf" Oct 30 03:29:22.206233 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.206210 4925 scope.go:122] "RemoveContainer" containerID="33d2de93d1e9b82c8c2a7832f241c865f093ee767c761733c50ae00232ad3951" Oct 30 03:29:22.216866 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.216845 4925 scope.go:122] "RemoveContainer" containerID="489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a" Oct 30 03:29:22.227689 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.227620 4925 scope.go:122] "RemoveContainer" containerID="d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08" Oct 30 03:29:22.240652 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.240631 4925 scope.go:122] "RemoveContainer" containerID="24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2" Oct 30 03:29:22.241724 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.241674 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b"] Oct 30 03:29:22.244032 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.244010 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-c5c2b"] Oct 30 03:29:22.250495 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.250471 4925 scope.go:122] "RemoveContainer" containerID="d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531" Oct 30 03:29:22.258458 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.258441 4925 scope.go:122] "RemoveContainer" containerID="489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a" Oct 30 03:29:22.258770 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:29:22.258731 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a\": container with ID starting with 489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a not found: ID does not exist" containerID="489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a" Oct 30 03:29:22.258863 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.258778 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a"} err="failed to get container status \"489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a\": rpc error: code = NotFound desc = could not find container \"489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a\": container with ID starting with 489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a not found: ID does not exist" Oct 30 03:29:22.258863 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.258796 4925 scope.go:122] "RemoveContainer" containerID="d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08" Oct 30 03:29:22.259073 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:29:22.259017 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08\": container with ID starting with d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08 not found: ID does not exist" containerID="d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08" Oct 30 03:29:22.259073 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.259039 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08"} err="failed to get container status \"d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08\": rpc error: code = NotFound desc = could not find container \"d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08\": container with ID starting with d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08 not found: ID does not exist" Oct 30 03:29:22.259073 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.259053 4925 scope.go:122] "RemoveContainer" containerID="24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2" Oct 30 03:29:22.259311 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:29:22.259281 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2\": container with ID starting with 24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2 not found: ID does not exist" containerID="24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2" Oct 30 03:29:22.259379 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.259318 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2"} err="failed to get container status \"24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2\": rpc error: code = NotFound desc = could not find container \"24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2\": container with ID starting with 24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2 not found: ID does not exist" Oct 30 03:29:22.259379 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.259338 4925 scope.go:122] "RemoveContainer" containerID="d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531" Oct 30 03:29:22.259590 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:29:22.259568 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531\": container with ID starting with d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531 not found: ID does not exist" containerID="d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531" Oct 30 03:29:22.259656 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.259595 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531"} err="failed to get container status \"d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531\": rpc error: code = NotFound desc = could not find container \"d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531\": container with ID starting with d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531 not found: ID does not exist" Oct 30 03:29:22.259656 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.259609 4925 scope.go:122] "RemoveContainer" containerID="489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a" Oct 30 03:29:22.259867 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.259845 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a"} err="failed to get container status \"489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a\": rpc error: code = NotFound desc = could not find container \"489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a\": container with ID starting with 489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a not found: ID does not exist" Oct 30 03:29:22.259939 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.259870 4925 scope.go:122] "RemoveContainer" containerID="d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08" Oct 30 03:29:22.260069 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.260051 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08"} err="failed to get container status \"d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08\": rpc error: code = NotFound desc = could not find container \"d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08\": container with ID starting with d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08 not found: ID does not exist" Oct 30 03:29:22.260125 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.260069 4925 scope.go:122] "RemoveContainer" containerID="24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2" Oct 30 03:29:22.260277 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.260251 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2"} err="failed to get container status \"24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2\": rpc error: code = NotFound desc = could not find container \"24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2\": container with ID starting with 24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2 not found: ID does not exist" Oct 30 03:29:22.260337 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.260277 4925 scope.go:122] "RemoveContainer" containerID="d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531" Oct 30 03:29:22.260472 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.260455 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531"} err="failed to get container status \"d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531\": rpc error: code = NotFound desc = could not find container \"d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531\": container with ID starting with d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531 not found: ID does not exist" Oct 30 03:29:22.260532 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.260471 4925 scope.go:122] "RemoveContainer" containerID="489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a" Oct 30 03:29:22.260655 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.260633 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a"} err="failed to get container status \"489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a\": rpc error: code = NotFound desc = could not find container \"489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a\": container with ID starting with 489205e969825f945c403e26120e53892721d8eac679b20fb65c1c2a43ba876a not found: ID does not exist" Oct 30 03:29:22.260812 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.260657 4925 scope.go:122] "RemoveContainer" containerID="d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08" Oct 30 03:29:22.260896 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.260879 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08"} err="failed to get container status \"d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08\": rpc error: code = NotFound desc = could not find container \"d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08\": container with ID starting with d8cf91792f4e5271e180750bd63730f5b614ea76dae69619d91deac541337c08 not found: ID does not exist" Oct 30 03:29:22.260946 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.260898 4925 scope.go:122] "RemoveContainer" containerID="24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2" Oct 30 03:29:22.261194 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.261160 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2"} err="failed to get container status \"24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2\": rpc error: code = NotFound desc = could not find container \"24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2\": container with ID starting with 24ba0500e091733a5282d049fc77948ff1488b122119da4f353f821ec26855a2 not found: ID does not exist" Oct 30 03:29:22.261257 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.261195 4925 scope.go:122] "RemoveContainer" containerID="d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531" Oct 30 03:29:22.261421 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.261405 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531"} err="failed to get container status \"d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531\": rpc error: code = NotFound desc = could not find container \"d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531\": container with ID starting with d64f7bee881000b1f137fd6294941a3b34a280b8082bae3181234f62c1f58531 not found: ID does not exist" Oct 30 03:29:22.275309 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.275283 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn"] Oct 30 03:29:22.280410 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.280387 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-hgsgn"] Oct 30 03:29:22.316366 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.316339 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 03:29:22.325162 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.325071 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 03:29:22.340280 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.340256 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:29:22.471847 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.471803 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-data-science-collector-ta-client-cert\") pod \"df5e38c1-d61f-4e87-b854-6ca066cd9818\" (UID: \"df5e38c1-d61f-4e87-b854-6ca066cd9818\") " Oct 30 03:29:22.471999 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.471907 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-tls-certs\" (UniqueName: \"kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-tls-certs\") pod \"df5e38c1-d61f-4e87-b854-6ca066cd9818\" (UID: \"df5e38c1-d61f-4e87-b854-6ca066cd9818\") " Oct 30 03:29:22.471999 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.471940 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df5e38c1-d61f-4e87-b854-6ca066cd9818-kube-api-access-5gknq\" (UniqueName: \"kubernetes.io/projected/df5e38c1-d61f-4e87-b854-6ca066cd9818-kube-api-access-5gknq\") pod \"df5e38c1-d61f-4e87-b854-6ca066cd9818\" (UID: \"df5e38c1-d61f-4e87-b854-6ca066cd9818\") " Oct 30 03:29:22.471999 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.471955 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-monitoring-tls-certs\") pod \"df5e38c1-d61f-4e87-b854-6ca066cd9818\" (UID: \"df5e38c1-d61f-4e87-b854-6ca066cd9818\") " Oct 30 03:29:22.472148 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.472107 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df5e38c1-d61f-4e87-b854-6ca066cd9818-otc-internal\" (UniqueName: \"kubernetes.io/configmap/df5e38c1-d61f-4e87-b854-6ca066cd9818-otc-internal\") pod \"df5e38c1-d61f-4e87-b854-6ca066cd9818\" (UID: \"df5e38c1-d61f-4e87-b854-6ca066cd9818\") " Oct 30 03:29:22.472463 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.472436 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df5e38c1-d61f-4e87-b854-6ca066cd9818-otc-internal" pod "df5e38c1-d61f-4e87-b854-6ca066cd9818" (UID: "df5e38c1-d61f-4e87-b854-6ca066cd9818"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:29:22.473865 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.473844 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-data-science-collector-ta-client-cert" pod "df5e38c1-d61f-4e87-b854-6ca066cd9818" (UID: "df5e38c1-d61f-4e87-b854-6ca066cd9818"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:22.473959 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.473894 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5e38c1-d61f-4e87-b854-6ca066cd9818-kube-api-access-5gknq" pod "df5e38c1-d61f-4e87-b854-6ca066cd9818" (UID: "df5e38c1-d61f-4e87-b854-6ca066cd9818"). InnerVolumeSpecName "kube-api-access-5gknq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:29:22.474222 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.474204 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-tls-certs" pod "df5e38c1-d61f-4e87-b854-6ca066cd9818" (UID: "df5e38c1-d61f-4e87-b854-6ca066cd9818"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:22.474261 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.474223 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-monitoring-tls-certs" pod "df5e38c1-d61f-4e87-b854-6ca066cd9818" (UID: "df5e38c1-d61f-4e87-b854-6ca066cd9818"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:22.573605 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.573545 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gknq\" (UniqueName: \"kubernetes.io/projected/df5e38c1-d61f-4e87-b854-6ca066cd9818-kube-api-access-5gknq\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:22.573605 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.573571 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:22.573605 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.573580 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/df5e38c1-d61f-4e87-b854-6ca066cd9818-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:22.573605 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.573590 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:22.573605 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:22.573599 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5e38c1-d61f-4e87-b854-6ca066cd9818-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:23.095620 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:23.095578 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6d855555-576b-481b-8f4b-05318df0c932" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.49:9093/-/ready\": dial tcp 10.132.0.49:9093: connect: connection refused" Oct 30 03:29:23.122596 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:23.122561 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="451685d8-55ce-4734-84b5-7faa42ca1702" path="/var/lib/kubelet/pods/451685d8-55ce-4734-84b5-7faa42ca1702/volumes" Oct 30 03:29:23.123399 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:23.123374 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f0ff87-430a-47cc-bc65-72491b15a41b" path="/var/lib/kubelet/pods/61f0ff87-430a-47cc-bc65-72491b15a41b/volumes" Oct 30 03:29:23.123947 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:23.123930 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3032256-3634-450c-b706-bf9d6e2df813" path="/var/lib/kubelet/pods/a3032256-3634-450c-b706-bf9d6e2df813/volumes" Oct 30 03:29:23.127242 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:23.127216 4925 generic.go:356] "Generic (PLEG): container finished" podID="df5e38c1-d61f-4e87-b854-6ca066cd9818" containerID="91a3ea3330644a0d424b90e82c2471e9f8f91d1fd2bae00f6ce480544aff534b" exitCode=0 Oct 30 03:29:23.127344 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:23.127278 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:29:23.127344 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:23.127292 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"df5e38c1-d61f-4e87-b854-6ca066cd9818","Type":"ContainerDied","Data":"91a3ea3330644a0d424b90e82c2471e9f8f91d1fd2bae00f6ce480544aff534b"} Oct 30 03:29:23.127460 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:23.127354 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"df5e38c1-d61f-4e87-b854-6ca066cd9818","Type":"ContainerDied","Data":"b3e4178732fadca6c23f9a91b4a1ba14bf12eb63f8b48a8e46f610f5d8848bf4"} Oct 30 03:29:23.127460 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:23.127371 4925 scope.go:122] "RemoveContainer" containerID="91a3ea3330644a0d424b90e82c2471e9f8f91d1fd2bae00f6ce480544aff534b" Oct 30 03:29:23.139723 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:23.139704 4925 scope.go:122] "RemoveContainer" containerID="91a3ea3330644a0d424b90e82c2471e9f8f91d1fd2bae00f6ce480544aff534b" Oct 30 03:29:23.140010 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:29:23.139992 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a3ea3330644a0d424b90e82c2471e9f8f91d1fd2bae00f6ce480544aff534b\": container with ID starting with 91a3ea3330644a0d424b90e82c2471e9f8f91d1fd2bae00f6ce480544aff534b not found: ID does not exist" containerID="91a3ea3330644a0d424b90e82c2471e9f8f91d1fd2bae00f6ce480544aff534b" Oct 30 03:29:23.140085 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:23.140015 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a3ea3330644a0d424b90e82c2471e9f8f91d1fd2bae00f6ce480544aff534b"} err="failed to get container status \"91a3ea3330644a0d424b90e82c2471e9f8f91d1fd2bae00f6ce480544aff534b\": rpc error: code = NotFound desc = could not find container \"91a3ea3330644a0d424b90e82c2471e9f8f91d1fd2bae00f6ce480544aff534b\": container with ID starting with 91a3ea3330644a0d424b90e82c2471e9f8f91d1fd2bae00f6ce480544aff534b not found: ID does not exist" Oct 30 03:29:23.183260 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:23.183232 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:29:23.189328 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:23.189303 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:29:25.118997 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:25.118965 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df5e38c1-d61f-4e87-b854-6ca066cd9818" path="/var/lib/kubelet/pods/df5e38c1-d61f-4e87-b854-6ca066cd9818/volumes" Oct 30 03:29:26.557245 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557212 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-4x4ds"] Oct 30 03:29:26.557600 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557544 4925 cpu_manager.go:414] "containerMap: removing container" podUID="61f0ff87-430a-47cc-bc65-72491b15a41b" containerName="prom-label-proxy" Oct 30 03:29:26.557600 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557559 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f0ff87-430a-47cc-bc65-72491b15a41b" containerName="prom-label-proxy" Oct 30 03:29:26.557600 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557570 4925 cpu_manager.go:414] "containerMap: removing container" podUID="a3032256-3634-450c-b706-bf9d6e2df813" containerName="init-config-reloader" Oct 30 03:29:26.557600 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557576 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3032256-3634-450c-b706-bf9d6e2df813" containerName="init-config-reloader" Oct 30 03:29:26.557600 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557585 4925 cpu_manager.go:414] "containerMap: removing container" podUID="a3032256-3634-450c-b706-bf9d6e2df813" containerName="config-reloader" Oct 30 03:29:26.557600 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557590 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3032256-3634-450c-b706-bf9d6e2df813" containerName="config-reloader" Oct 30 03:29:26.557815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557607 4925 cpu_manager.go:414] "containerMap: removing container" podUID="61f0ff87-430a-47cc-bc65-72491b15a41b" containerName="kube-rbac-proxy" Oct 30 03:29:26.557815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557612 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f0ff87-430a-47cc-bc65-72491b15a41b" containerName="kube-rbac-proxy" Oct 30 03:29:26.557815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557621 4925 cpu_manager.go:414] "containerMap: removing container" podUID="a3032256-3634-450c-b706-bf9d6e2df813" containerName="thanos-sidecar" Oct 30 03:29:26.557815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557627 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3032256-3634-450c-b706-bf9d6e2df813" containerName="thanos-sidecar" Oct 30 03:29:26.557815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557639 4925 cpu_manager.go:414] "containerMap: removing container" podUID="df5e38c1-d61f-4e87-b854-6ca066cd9818" containerName="otc-container" Oct 30 03:29:26.557815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557644 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df5e38c1-d61f-4e87-b854-6ca066cd9818" containerName="otc-container" Oct 30 03:29:26.557815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557652 4925 cpu_manager.go:414] "containerMap: removing container" podUID="451685d8-55ce-4734-84b5-7faa42ca1702" containerName="korrel8r" Oct 30 03:29:26.557815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557656 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="451685d8-55ce-4734-84b5-7faa42ca1702" containerName="korrel8r" Oct 30 03:29:26.557815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557664 4925 cpu_manager.go:414] "containerMap: removing container" podUID="a3032256-3634-450c-b706-bf9d6e2df813" containerName="prometheus" Oct 30 03:29:26.557815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557669 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3032256-3634-450c-b706-bf9d6e2df813" containerName="prometheus" Oct 30 03:29:26.557815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557731 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3032256-3634-450c-b706-bf9d6e2df813" containerName="thanos-sidecar" Oct 30 03:29:26.557815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557740 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="df5e38c1-d61f-4e87-b854-6ca066cd9818" containerName="otc-container" Oct 30 03:29:26.557815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557746 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="451685d8-55ce-4734-84b5-7faa42ca1702" containerName="korrel8r" Oct 30 03:29:26.557815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557763 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3032256-3634-450c-b706-bf9d6e2df813" containerName="prometheus" Oct 30 03:29:26.557815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557769 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f0ff87-430a-47cc-bc65-72491b15a41b" containerName="prom-label-proxy" Oct 30 03:29:26.557815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557778 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3032256-3634-450c-b706-bf9d6e2df813" containerName="config-reloader" Oct 30 03:29:26.557815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.557789 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f0ff87-430a-47cc-bc65-72491b15a41b" containerName="kube-rbac-proxy" Oct 30 03:29:26.562294 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.562277 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" Oct 30 03:29:26.592096 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.592070 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 03:29:26.592303 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.592286 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-9f5ks\"" Oct 30 03:29:26.592377 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.592290 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 03:29:26.594010 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.593984 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-4x4ds"] Oct 30 03:29:26.704043 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.704015 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff186267-a39c-42f5-b817-1731ce550268-config\") pod \"korrel8r-6644769df5-4x4ds\" (UID: \"ff186267-a39c-42f5-b817-1731ce550268\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" Oct 30 03:29:26.704185 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.704052 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ff186267-a39c-42f5-b817-1731ce550268-tls\") pod \"korrel8r-6644769df5-4x4ds\" (UID: \"ff186267-a39c-42f5-b817-1731ce550268\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" Oct 30 03:29:26.704228 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.704187 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvvn7\" (UniqueName: \"kubernetes.io/projected/ff186267-a39c-42f5-b817-1731ce550268-kube-api-access-rvvn7\") pod \"korrel8r-6644769df5-4x4ds\" (UID: \"ff186267-a39c-42f5-b817-1731ce550268\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" Oct 30 03:29:26.805437 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.805408 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvvn7\" (UniqueName: \"kubernetes.io/projected/ff186267-a39c-42f5-b817-1731ce550268-kube-api-access-rvvn7\") pod \"korrel8r-6644769df5-4x4ds\" (UID: \"ff186267-a39c-42f5-b817-1731ce550268\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" Oct 30 03:29:26.805593 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.805451 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff186267-a39c-42f5-b817-1731ce550268-config\") pod \"korrel8r-6644769df5-4x4ds\" (UID: \"ff186267-a39c-42f5-b817-1731ce550268\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" Oct 30 03:29:26.805593 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.805470 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ff186267-a39c-42f5-b817-1731ce550268-tls\") pod \"korrel8r-6644769df5-4x4ds\" (UID: \"ff186267-a39c-42f5-b817-1731ce550268\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" Oct 30 03:29:26.806191 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.806165 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff186267-a39c-42f5-b817-1731ce550268-config\") pod \"korrel8r-6644769df5-4x4ds\" (UID: \"ff186267-a39c-42f5-b817-1731ce550268\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" Oct 30 03:29:26.807880 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.807836 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ff186267-a39c-42f5-b817-1731ce550268-tls\") pod \"korrel8r-6644769df5-4x4ds\" (UID: \"ff186267-a39c-42f5-b817-1731ce550268\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" Oct 30 03:29:26.829525 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.829501 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvvn7\" (UniqueName: \"kubernetes.io/projected/ff186267-a39c-42f5-b817-1731ce550268-kube-api-access-rvvn7\") pod \"korrel8r-6644769df5-4x4ds\" (UID: \"ff186267-a39c-42f5-b817-1731ce550268\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" Oct 30 03:29:26.873875 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:26.873849 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" Oct 30 03:29:27.170495 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:27.170470 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-4x4ds"] Oct 30 03:29:27.173354 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:29:27.173327 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff186267_a39c_42f5_b817_1731ce550268.slice/crio-adb7cbbad6e18f987d2bd136aa31e15d478721e74ec6da806a56c4e2fb910d48 WatchSource:0}: Error finding container adb7cbbad6e18f987d2bd136aa31e15d478721e74ec6da806a56c4e2fb910d48: Status 404 returned error can't find the container with id adb7cbbad6e18f987d2bd136aa31e15d478721e74ec6da806a56c4e2fb910d48 Oct 30 03:29:27.306351 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:27.306319 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 03:29:27.310409 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:27.310384 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 03:29:27.333307 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:27.333268 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 03:29:27.368461 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:27.368433 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-kxk4q\"" Oct 30 03:29:27.370063 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:27.370040 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:29:27.399671 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:27.399642 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-645168e1\"" Oct 30 03:29:27.411061 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:27.411038 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/620c802b-45a3-496a-a53f-53d153d51877-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"620c802b-45a3-496a-a53f-53d153d51877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 03:29:27.411190 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:27.411079 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/620c802b-45a3-496a-a53f-53d153d51877-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"620c802b-45a3-496a-a53f-53d153d51877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 03:29:27.411272 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:27.411241 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhgqx\" (UniqueName: \"kubernetes.io/projected/620c802b-45a3-496a-a53f-53d153d51877-kube-api-access-fhgqx\") pod \"data-science-collector-collector-1\" (UID: \"620c802b-45a3-496a-a53f-53d153d51877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 03:29:27.512588 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:27.512508 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/620c802b-45a3-496a-a53f-53d153d51877-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"620c802b-45a3-496a-a53f-53d153d51877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 03:29:27.512588 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:27.512550 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/620c802b-45a3-496a-a53f-53d153d51877-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"620c802b-45a3-496a-a53f-53d153d51877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 03:29:27.512807 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:27.512593 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhgqx\" (UniqueName: \"kubernetes.io/projected/620c802b-45a3-496a-a53f-53d153d51877-kube-api-access-fhgqx\") pod \"data-science-collector-collector-1\" (UID: \"620c802b-45a3-496a-a53f-53d153d51877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 03:29:27.513284 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:27.513257 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/620c802b-45a3-496a-a53f-53d153d51877-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"620c802b-45a3-496a-a53f-53d153d51877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 03:29:27.514899 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:27.514882 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/620c802b-45a3-496a-a53f-53d153d51877-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"620c802b-45a3-496a-a53f-53d153d51877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 03:29:27.536860 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:27.536840 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhgqx\" (UniqueName: \"kubernetes.io/projected/620c802b-45a3-496a-a53f-53d153d51877-kube-api-access-fhgqx\") pod \"data-science-collector-collector-1\" (UID: \"620c802b-45a3-496a-a53f-53d153d51877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 03:29:27.625224 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:27.625195 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 03:29:27.934275 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:27.933887 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 03:29:27.937011 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:29:27.936981 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620c802b_45a3_496a_a53f_53d153d51877.slice/crio-20eb0cf62c5ed1abc5a9497e0870451e2f98a6bb07045a9dd1d6730407024b01 WatchSource:0}: Error finding container 20eb0cf62c5ed1abc5a9497e0870451e2f98a6bb07045a9dd1d6730407024b01: Status 404 returned error can't find the container with id 20eb0cf62c5ed1abc5a9497e0870451e2f98a6bb07045a9dd1d6730407024b01 Oct 30 03:29:27.939856 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:27.939817 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:29:27.939932 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:27.939894 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:29:28.095544 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:28.095503 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6d855555-576b-481b-8f4b-05318df0c932" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.49:9093/-/ready\": dial tcp 10.132.0.49:9093: connect: connection refused" Oct 30 03:29:28.157723 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:28.157684 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"620c802b-45a3-496a-a53f-53d153d51877","Type":"ContainerStarted","Data":"f269714f7bee6472227a4b0864beb7b2e98cd62c52a0fb453b3fd7ff2ef951d6"} Oct 30 03:29:28.157914 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:28.157740 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"620c802b-45a3-496a-a53f-53d153d51877","Type":"ContainerStarted","Data":"20eb0cf62c5ed1abc5a9497e0870451e2f98a6bb07045a9dd1d6730407024b01"} Oct 30 03:29:28.159236 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:28.159211 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" event={"ID":"ff186267-a39c-42f5-b817-1731ce550268","Type":"ContainerStarted","Data":"fbca9e42eb958a97bd339f531f172a075113064f2033f42515e483fde0126a98"} Oct 30 03:29:28.159358 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:28.159252 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" event={"ID":"ff186267-a39c-42f5-b817-1731ce550268","Type":"ContainerStarted","Data":"adb7cbbad6e18f987d2bd136aa31e15d478721e74ec6da806a56c4e2fb910d48"} Oct 30 03:29:28.159358 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:28.159313 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" Oct 30 03:29:28.200211 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:28.200128 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.200115612 podStartE2EDuration="1.200115612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:28.199192672 +0000 UTC m=+2929.212470775" watchObservedRunningTime="2026-10-30 03:29:28.200115612 +0000 UTC m=+2929.213393712" Oct 30 03:29:28.247709 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:28.247654 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" podStartSLOduration=2.247640217 podStartE2EDuration="2.247640217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:28.245986223 +0000 UTC m=+2929.259264326" watchObservedRunningTime="2026-10-30 03:29:28.247640217 +0000 UTC m=+2929.260918318" Oct 30 03:29:31.686100 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.686075 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:29:31.845865 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.845841 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d855555-576b-481b-8f4b-05318df0c932-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d855555-576b-481b-8f4b-05318df0c932-config-out\") pod \"6d855555-576b-481b-8f4b-05318df0c932\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " Oct 30 03:29:31.846008 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.845904 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-config-volume\" (UniqueName: \"kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-config-volume\") pod \"6d855555-576b-481b-8f4b-05318df0c932\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " Oct 30 03:29:31.846008 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.845941 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d855555-576b-481b-8f4b-05318df0c932-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6d855555-576b-481b-8f4b-05318df0c932-alertmanager-data-science-monitoringstack-db\") pod \"6d855555-576b-481b-8f4b-05318df0c932\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " Oct 30 03:29:31.846008 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.845967 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-cluster-tls-config\") pod \"6d855555-576b-481b-8f4b-05318df0c932\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " Oct 30 03:29:31.846161 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.846035 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-web-config\" (UniqueName: \"kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-web-config\") pod \"6d855555-576b-481b-8f4b-05318df0c932\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " Oct 30 03:29:31.846161 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.846064 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d855555-576b-481b-8f4b-05318df0c932-kube-api-access-9dt6l\" (UniqueName: \"kubernetes.io/projected/6d855555-576b-481b-8f4b-05318df0c932-kube-api-access-9dt6l\") pod \"6d855555-576b-481b-8f4b-05318df0c932\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " Oct 30 03:29:31.846161 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.846090 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d855555-576b-481b-8f4b-05318df0c932-tls-assets\" (UniqueName: \"kubernetes.io/projected/6d855555-576b-481b-8f4b-05318df0c932-tls-assets\") pod \"6d855555-576b-481b-8f4b-05318df0c932\" (UID: \"6d855555-576b-481b-8f4b-05318df0c932\") " Oct 30 03:29:31.846501 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.846313 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d855555-576b-481b-8f4b-05318df0c932-alertmanager-data-science-monitoringstack-db" pod "6d855555-576b-481b-8f4b-05318df0c932" (UID: "6d855555-576b-481b-8f4b-05318df0c932"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:29:31.846501 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.846472 4925 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6d855555-576b-481b-8f4b-05318df0c932-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:31.848422 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.848392 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d855555-576b-481b-8f4b-05318df0c932-config-out" pod "6d855555-576b-481b-8f4b-05318df0c932" (UID: "6d855555-576b-481b-8f4b-05318df0c932"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:29:31.848517 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.848414 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d855555-576b-481b-8f4b-05318df0c932-tls-assets" pod "6d855555-576b-481b-8f4b-05318df0c932" (UID: "6d855555-576b-481b-8f4b-05318df0c932"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:29:31.848568 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.848535 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-config-volume" pod "6d855555-576b-481b-8f4b-05318df0c932" (UID: "6d855555-576b-481b-8f4b-05318df0c932"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:31.848701 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.848676 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d855555-576b-481b-8f4b-05318df0c932-kube-api-access-9dt6l" pod "6d855555-576b-481b-8f4b-05318df0c932" (UID: "6d855555-576b-481b-8f4b-05318df0c932"). InnerVolumeSpecName "kube-api-access-9dt6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:29:31.852438 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.852415 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-cluster-tls-config" pod "6d855555-576b-481b-8f4b-05318df0c932" (UID: "6d855555-576b-481b-8f4b-05318df0c932"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:31.859565 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.859537 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-web-config" pod "6d855555-576b-481b-8f4b-05318df0c932" (UID: "6d855555-576b-481b-8f4b-05318df0c932"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:29:31.947695 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.947664 4925 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-web-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:31.947695 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.947691 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dt6l\" (UniqueName: \"kubernetes.io/projected/6d855555-576b-481b-8f4b-05318df0c932-kube-api-access-9dt6l\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:31.947872 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.947706 4925 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d855555-576b-481b-8f4b-05318df0c932-tls-assets\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:31.947872 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.947717 4925 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d855555-576b-481b-8f4b-05318df0c932-config-out\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:31.947872 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.947728 4925 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-config-volume\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:31.947872 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:31.947738 4925 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6d855555-576b-481b-8f4b-05318df0c932-cluster-tls-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:29:32.179011 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:32.178917 4925 generic.go:356] "Generic (PLEG): container finished" podID="6d855555-576b-481b-8f4b-05318df0c932" containerID="506a8df06b70f2d87c86a1a03533bc36fec25983dbbc869877d6a076efe5e4cd" exitCode=0 Oct 30 03:29:32.179011 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:32.178980 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:29:32.179011 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:32.178997 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6d855555-576b-481b-8f4b-05318df0c932","Type":"ContainerDied","Data":"506a8df06b70f2d87c86a1a03533bc36fec25983dbbc869877d6a076efe5e4cd"} Oct 30 03:29:32.179272 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:32.179070 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6d855555-576b-481b-8f4b-05318df0c932","Type":"ContainerDied","Data":"a7a98458cf3e8aec97e09fba2670f48ccc6e2850f39746198a32fa070871f3bf"} Oct 30 03:29:32.179272 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:32.179106 4925 scope.go:122] "RemoveContainer" containerID="b7cc879bb16bfca0b4e95c290d181690b6387c2251b89bae2ddbada9421d42b9" Oct 30 03:29:32.191376 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:32.191357 4925 scope.go:122] "RemoveContainer" containerID="506a8df06b70f2d87c86a1a03533bc36fec25983dbbc869877d6a076efe5e4cd" Oct 30 03:29:32.199355 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:32.199339 4925 scope.go:122] "RemoveContainer" containerID="3060e95a8efb9c7898188af81c9cf2e3bed2f7681490184b6e18871e109e0f92" Oct 30 03:29:32.206827 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:32.206808 4925 scope.go:122] "RemoveContainer" containerID="b7cc879bb16bfca0b4e95c290d181690b6387c2251b89bae2ddbada9421d42b9" Oct 30 03:29:32.207068 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:29:32.207047 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7cc879bb16bfca0b4e95c290d181690b6387c2251b89bae2ddbada9421d42b9\": container with ID starting with b7cc879bb16bfca0b4e95c290d181690b6387c2251b89bae2ddbada9421d42b9 not found: ID does not exist" containerID="b7cc879bb16bfca0b4e95c290d181690b6387c2251b89bae2ddbada9421d42b9" Oct 30 03:29:32.207138 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:32.207084 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7cc879bb16bfca0b4e95c290d181690b6387c2251b89bae2ddbada9421d42b9"} err="failed to get container status \"b7cc879bb16bfca0b4e95c290d181690b6387c2251b89bae2ddbada9421d42b9\": rpc error: code = NotFound desc = could not find container \"b7cc879bb16bfca0b4e95c290d181690b6387c2251b89bae2ddbada9421d42b9\": container with ID starting with b7cc879bb16bfca0b4e95c290d181690b6387c2251b89bae2ddbada9421d42b9 not found: ID does not exist" Oct 30 03:29:32.207138 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:32.207102 4925 scope.go:122] "RemoveContainer" containerID="506a8df06b70f2d87c86a1a03533bc36fec25983dbbc869877d6a076efe5e4cd" Oct 30 03:29:32.207304 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:29:32.207289 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"506a8df06b70f2d87c86a1a03533bc36fec25983dbbc869877d6a076efe5e4cd\": container with ID starting with 506a8df06b70f2d87c86a1a03533bc36fec25983dbbc869877d6a076efe5e4cd not found: ID does not exist" containerID="506a8df06b70f2d87c86a1a03533bc36fec25983dbbc869877d6a076efe5e4cd" Oct 30 03:29:32.207341 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:32.207309 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506a8df06b70f2d87c86a1a03533bc36fec25983dbbc869877d6a076efe5e4cd"} err="failed to get container status \"506a8df06b70f2d87c86a1a03533bc36fec25983dbbc869877d6a076efe5e4cd\": rpc error: code = NotFound desc = could not find container \"506a8df06b70f2d87c86a1a03533bc36fec25983dbbc869877d6a076efe5e4cd\": container with ID starting with 506a8df06b70f2d87c86a1a03533bc36fec25983dbbc869877d6a076efe5e4cd not found: ID does not exist" Oct 30 03:29:32.207341 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:32.207321 4925 scope.go:122] "RemoveContainer" containerID="3060e95a8efb9c7898188af81c9cf2e3bed2f7681490184b6e18871e109e0f92" Oct 30 03:29:32.207535 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:29:32.207515 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3060e95a8efb9c7898188af81c9cf2e3bed2f7681490184b6e18871e109e0f92\": container with ID starting with 3060e95a8efb9c7898188af81c9cf2e3bed2f7681490184b6e18871e109e0f92 not found: ID does not exist" containerID="3060e95a8efb9c7898188af81c9cf2e3bed2f7681490184b6e18871e109e0f92" Oct 30 03:29:32.207586 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:32.207539 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3060e95a8efb9c7898188af81c9cf2e3bed2f7681490184b6e18871e109e0f92"} err="failed to get container status \"3060e95a8efb9c7898188af81c9cf2e3bed2f7681490184b6e18871e109e0f92\": rpc error: code = NotFound desc = could not find container \"3060e95a8efb9c7898188af81c9cf2e3bed2f7681490184b6e18871e109e0f92\": container with ID starting with 3060e95a8efb9c7898188af81c9cf2e3bed2f7681490184b6e18871e109e0f92 not found: ID does not exist" Oct 30 03:29:32.237710 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:32.237687 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 03:29:32.250558 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:32.250536 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 03:29:33.124968 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:33.124938 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d855555-576b-481b-8f4b-05318df0c932" path="/var/lib/kubelet/pods/6d855555-576b-481b-8f4b-05318df0c932/volumes" Oct 30 03:29:39.165630 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:29:39.165602 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" Oct 30 03:29:48.160500 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:29:48.160465 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:30:18.161186 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:30:18.161148 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:30:20.155076 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.155040 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-4x4ds"] Oct 30 03:30:20.155479 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.155272 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" podUID="ff186267-a39c-42f5-b817-1731ce550268" containerName="korrel8r" containerID="cri-o://fbca9e42eb958a97bd339f531f172a075113064f2033f42515e483fde0126a98" gracePeriod=30 Oct 30 03:30:20.397396 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.397371 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" Oct 30 03:30:20.415400 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.415340 4925 generic.go:356] "Generic (PLEG): container finished" podID="ff186267-a39c-42f5-b817-1731ce550268" containerID="fbca9e42eb958a97bd339f531f172a075113064f2033f42515e483fde0126a98" exitCode=0 Oct 30 03:30:20.415400 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.415383 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" Oct 30 03:30:20.415541 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.415394 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" event={"ID":"ff186267-a39c-42f5-b817-1731ce550268","Type":"ContainerDied","Data":"fbca9e42eb958a97bd339f531f172a075113064f2033f42515e483fde0126a98"} Oct 30 03:30:20.415541 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.415464 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-4x4ds" event={"ID":"ff186267-a39c-42f5-b817-1731ce550268","Type":"ContainerDied","Data":"adb7cbbad6e18f987d2bd136aa31e15d478721e74ec6da806a56c4e2fb910d48"} Oct 30 03:30:20.415541 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.415487 4925 scope.go:122] "RemoveContainer" containerID="fbca9e42eb958a97bd339f531f172a075113064f2033f42515e483fde0126a98" Oct 30 03:30:20.423520 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.423507 4925 scope.go:122] "RemoveContainer" containerID="fbca9e42eb958a97bd339f531f172a075113064f2033f42515e483fde0126a98" Oct 30 03:30:20.423785 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:30:20.423763 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbca9e42eb958a97bd339f531f172a075113064f2033f42515e483fde0126a98\": container with ID starting with fbca9e42eb958a97bd339f531f172a075113064f2033f42515e483fde0126a98 not found: ID does not exist" containerID="fbca9e42eb958a97bd339f531f172a075113064f2033f42515e483fde0126a98" Oct 30 03:30:20.423855 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.423794 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbca9e42eb958a97bd339f531f172a075113064f2033f42515e483fde0126a98"} err="failed to get container status \"fbca9e42eb958a97bd339f531f172a075113064f2033f42515e483fde0126a98\": rpc error: code = NotFound desc = could not find container \"fbca9e42eb958a97bd339f531f172a075113064f2033f42515e483fde0126a98\": container with ID starting with fbca9e42eb958a97bd339f531f172a075113064f2033f42515e483fde0126a98 not found: ID does not exist" Oct 30 03:30:20.499810 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.499780 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff186267-a39c-42f5-b817-1731ce550268-kube-api-access-rvvn7\" (UniqueName: \"kubernetes.io/projected/ff186267-a39c-42f5-b817-1731ce550268-kube-api-access-rvvn7\") pod \"ff186267-a39c-42f5-b817-1731ce550268\" (UID: \"ff186267-a39c-42f5-b817-1731ce550268\") " Oct 30 03:30:20.499972 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.499844 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff186267-a39c-42f5-b817-1731ce550268-config\" (UniqueName: \"kubernetes.io/configmap/ff186267-a39c-42f5-b817-1731ce550268-config\") pod \"ff186267-a39c-42f5-b817-1731ce550268\" (UID: \"ff186267-a39c-42f5-b817-1731ce550268\") " Oct 30 03:30:20.499972 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.499874 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff186267-a39c-42f5-b817-1731ce550268-tls\" (UniqueName: \"kubernetes.io/secret/ff186267-a39c-42f5-b817-1731ce550268-tls\") pod \"ff186267-a39c-42f5-b817-1731ce550268\" (UID: \"ff186267-a39c-42f5-b817-1731ce550268\") " Oct 30 03:30:20.500285 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.500262 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff186267-a39c-42f5-b817-1731ce550268-config" pod "ff186267-a39c-42f5-b817-1731ce550268" (UID: "ff186267-a39c-42f5-b817-1731ce550268"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:30:20.501727 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.501707 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff186267-a39c-42f5-b817-1731ce550268-kube-api-access-rvvn7" pod "ff186267-a39c-42f5-b817-1731ce550268" (UID: "ff186267-a39c-42f5-b817-1731ce550268"). InnerVolumeSpecName "kube-api-access-rvvn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:30:20.501829 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.501808 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff186267-a39c-42f5-b817-1731ce550268-tls" pod "ff186267-a39c-42f5-b817-1731ce550268" (UID: "ff186267-a39c-42f5-b817-1731ce550268"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:30:20.578973 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.578939 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 03:30:20.579162 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.579124 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="620c802b-45a3-496a-a53f-53d153d51877" containerName="otc-container" containerID="cri-o://f269714f7bee6472227a4b0864beb7b2e98cd62c52a0fb453b3fd7ff2ef951d6" gracePeriod=30 Oct 30 03:30:20.600546 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.600514 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvvn7\" (UniqueName: \"kubernetes.io/projected/ff186267-a39c-42f5-b817-1731ce550268-kube-api-access-rvvn7\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:30:20.600645 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.600551 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff186267-a39c-42f5-b817-1731ce550268-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:30:20.600645 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.600565 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ff186267-a39c-42f5-b817-1731ce550268-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:30:20.764141 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.764109 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-4x4ds"] Oct 30 03:30:20.767282 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.767259 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-4x4ds"] Oct 30 03:30:20.817148 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.817128 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 03:30:20.902877 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.902851 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/620c802b-45a3-496a-a53f-53d153d51877-otc-internal\" (UniqueName: \"kubernetes.io/configmap/620c802b-45a3-496a-a53f-53d153d51877-otc-internal\") pod \"620c802b-45a3-496a-a53f-53d153d51877\" (UID: \"620c802b-45a3-496a-a53f-53d153d51877\") " Oct 30 03:30:20.903034 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.902888 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620c802b-45a3-496a-a53f-53d153d51877-kube-api-access-fhgqx\" (UniqueName: \"kubernetes.io/projected/620c802b-45a3-496a-a53f-53d153d51877-kube-api-access-fhgqx\") pod \"620c802b-45a3-496a-a53f-53d153d51877\" (UID: \"620c802b-45a3-496a-a53f-53d153d51877\") " Oct 30 03:30:20.903034 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.902943 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/620c802b-45a3-496a-a53f-53d153d51877-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/620c802b-45a3-496a-a53f-53d153d51877-monitoring-tls-certs\") pod \"620c802b-45a3-496a-a53f-53d153d51877\" (UID: \"620c802b-45a3-496a-a53f-53d153d51877\") " Oct 30 03:30:20.903218 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.903191 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620c802b-45a3-496a-a53f-53d153d51877-otc-internal" pod "620c802b-45a3-496a-a53f-53d153d51877" (UID: "620c802b-45a3-496a-a53f-53d153d51877"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:30:20.904954 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.904925 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/620c802b-45a3-496a-a53f-53d153d51877-monitoring-tls-certs" pod "620c802b-45a3-496a-a53f-53d153d51877" (UID: "620c802b-45a3-496a-a53f-53d153d51877"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:30:20.905048 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:20.905030 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620c802b-45a3-496a-a53f-53d153d51877-kube-api-access-fhgqx" pod "620c802b-45a3-496a-a53f-53d153d51877" (UID: "620c802b-45a3-496a-a53f-53d153d51877"). InnerVolumeSpecName "kube-api-access-fhgqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:30:21.004556 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:21.004489 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/620c802b-45a3-496a-a53f-53d153d51877-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:30:21.004556 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:21.004513 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/620c802b-45a3-496a-a53f-53d153d51877-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:30:21.004556 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:21.004523 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhgqx\" (UniqueName: \"kubernetes.io/projected/620c802b-45a3-496a-a53f-53d153d51877-kube-api-access-fhgqx\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:30:21.120287 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:21.120253 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff186267-a39c-42f5-b817-1731ce550268" path="/var/lib/kubelet/pods/ff186267-a39c-42f5-b817-1731ce550268/volumes" Oct 30 03:30:21.421809 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:21.421745 4925 generic.go:356] "Generic (PLEG): container finished" podID="620c802b-45a3-496a-a53f-53d153d51877" containerID="f269714f7bee6472227a4b0864beb7b2e98cd62c52a0fb453b3fd7ff2ef951d6" exitCode=0 Oct 30 03:30:21.422232 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:21.421819 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 03:30:21.422232 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:21.421832 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"620c802b-45a3-496a-a53f-53d153d51877","Type":"ContainerDied","Data":"f269714f7bee6472227a4b0864beb7b2e98cd62c52a0fb453b3fd7ff2ef951d6"} Oct 30 03:30:21.422232 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:21.421882 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"620c802b-45a3-496a-a53f-53d153d51877","Type":"ContainerDied","Data":"20eb0cf62c5ed1abc5a9497e0870451e2f98a6bb07045a9dd1d6730407024b01"} Oct 30 03:30:21.422232 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:21.421911 4925 scope.go:122] "RemoveContainer" containerID="f269714f7bee6472227a4b0864beb7b2e98cd62c52a0fb453b3fd7ff2ef951d6" Oct 30 03:30:21.430313 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:21.430290 4925 scope.go:122] "RemoveContainer" containerID="f269714f7bee6472227a4b0864beb7b2e98cd62c52a0fb453b3fd7ff2ef951d6" Oct 30 03:30:21.430579 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:30:21.430558 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f269714f7bee6472227a4b0864beb7b2e98cd62c52a0fb453b3fd7ff2ef951d6\": container with ID starting with f269714f7bee6472227a4b0864beb7b2e98cd62c52a0fb453b3fd7ff2ef951d6 not found: ID does not exist" containerID="f269714f7bee6472227a4b0864beb7b2e98cd62c52a0fb453b3fd7ff2ef951d6" Oct 30 03:30:21.430627 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:21.430590 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f269714f7bee6472227a4b0864beb7b2e98cd62c52a0fb453b3fd7ff2ef951d6"} err="failed to get container status \"f269714f7bee6472227a4b0864beb7b2e98cd62c52a0fb453b3fd7ff2ef951d6\": rpc error: code = NotFound desc = could not find container \"f269714f7bee6472227a4b0864beb7b2e98cd62c52a0fb453b3fd7ff2ef951d6\": container with ID starting with f269714f7bee6472227a4b0864beb7b2e98cd62c52a0fb453b3fd7ff2ef951d6 not found: ID does not exist" Oct 30 03:30:21.476408 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:21.476378 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 03:30:21.480320 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:21.480299 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 03:30:23.119187 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:23.119153 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="620c802b-45a3-496a-a53f-53d153d51877" path="/var/lib/kubelet/pods/620c802b-45a3-496a-a53f-53d153d51877/volumes" Oct 30 03:30:25.404353 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.404324 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59"] Oct 30 03:30:25.404816 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.404663 4925 cpu_manager.go:414] "containerMap: removing container" podUID="620c802b-45a3-496a-a53f-53d153d51877" containerName="otc-container" Oct 30 03:30:25.404816 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.404674 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620c802b-45a3-496a-a53f-53d153d51877" containerName="otc-container" Oct 30 03:30:25.404816 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.404693 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6d855555-576b-481b-8f4b-05318df0c932" containerName="config-reloader" Oct 30 03:30:25.404816 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.404698 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d855555-576b-481b-8f4b-05318df0c932" containerName="config-reloader" Oct 30 03:30:25.404816 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.404716 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6d855555-576b-481b-8f4b-05318df0c932" containerName="init-config-reloader" Oct 30 03:30:25.404816 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.404721 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d855555-576b-481b-8f4b-05318df0c932" containerName="init-config-reloader" Oct 30 03:30:25.404816 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.404727 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6d855555-576b-481b-8f4b-05318df0c932" containerName="alertmanager" Oct 30 03:30:25.404816 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.404732 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d855555-576b-481b-8f4b-05318df0c932" containerName="alertmanager" Oct 30 03:30:25.404816 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.404739 4925 cpu_manager.go:414] "containerMap: removing container" podUID="ff186267-a39c-42f5-b817-1731ce550268" containerName="korrel8r" Oct 30 03:30:25.404816 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.404743 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff186267-a39c-42f5-b817-1731ce550268" containerName="korrel8r" Oct 30 03:30:25.405118 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.404845 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d855555-576b-481b-8f4b-05318df0c932" containerName="alertmanager" Oct 30 03:30:25.405118 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.404853 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff186267-a39c-42f5-b817-1731ce550268" containerName="korrel8r" Oct 30 03:30:25.405118 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.404863 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="620c802b-45a3-496a-a53f-53d153d51877" containerName="otc-container" Oct 30 03:30:25.405118 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.404873 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d855555-576b-481b-8f4b-05318df0c932" containerName="config-reloader" Oct 30 03:30:25.409197 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.409177 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:30:25.426961 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.426942 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 03:30:25.428518 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.428495 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59"] Oct 30 03:30:25.439556 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.439533 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 03:30:25.439662 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.439573 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 03:30:25.439722 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.439678 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 03:30:25.526686 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.526661 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 03:30:25.526846 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.526700 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 03:30:25.526919 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.526899 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-ffqjr\"" Oct 30 03:30:25.541710 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.541680 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-25b59\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:30:25.541836 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.541737 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/5975a38d-4a75-4604-9db8-f130c3d0ec0a-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-25b59\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:30:25.541836 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.541829 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-25b59\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:30:25.541930 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.541862 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-25b59\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:30:25.541930 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.541890 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zl9c\" (UniqueName: \"kubernetes.io/projected/5975a38d-4a75-4604-9db8-f130c3d0ec0a-kube-api-access-9zl9c\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-25b59\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:30:25.543089 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.543067 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2"] Oct 30 03:30:25.546690 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.546672 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" Oct 30 03:30:25.574072 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.574047 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2"] Oct 30 03:30:25.627376 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.627341 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 03:30:25.627505 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.627381 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-9chrn\"" Oct 30 03:30:25.632259 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.632240 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 03:30:25.642558 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.642536 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgdz6\" (UniqueName: \"kubernetes.io/projected/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-kube-api-access-fgdz6\") pod \"korrel8r-66d6f7f696-9m2s2\" (UID: \"b3a97a3f-60bb-43b3-942f-1ff03e25f94e\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" Oct 30 03:30:25.642647 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.642565 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-25b59\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:30:25.642647 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.642589 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zl9c\" (UniqueName: \"kubernetes.io/projected/5975a38d-4a75-4604-9db8-f130c3d0ec0a-kube-api-access-9zl9c\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-25b59\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:30:25.642647 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.642605 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-tls\") pod \"korrel8r-66d6f7f696-9m2s2\" (UID: \"b3a97a3f-60bb-43b3-942f-1ff03e25f94e\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" Oct 30 03:30:25.642845 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.642812 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-25b59\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:30:25.642902 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.642860 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-config\") pod \"korrel8r-66d6f7f696-9m2s2\" (UID: \"b3a97a3f-60bb-43b3-942f-1ff03e25f94e\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" Oct 30 03:30:25.642902 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.642880 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/5975a38d-4a75-4604-9db8-f130c3d0ec0a-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-25b59\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:30:25.643011 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.642915 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-25b59\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:30:25.643109 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:30:25.643091 4925 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 03:30:25.643170 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:30:25.643160 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-prometheus-client-cert podName:5975a38d-4a75-4604-9db8-f130c3d0ec0a nodeName:}" failed. No retries permitted until 2026-10-30 03:30:26.143139988 +0000 UTC m=+2987.156418088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-25b59" (UID: "5975a38d-4a75-4604-9db8-f130c3d0ec0a") : secret "prometheus-operated-tls" not found Oct 30 03:30:25.643521 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.643500 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/5975a38d-4a75-4604-9db8-f130c3d0ec0a-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-25b59\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:30:25.645018 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.644994 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-25b59\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:30:25.645144 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.645127 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-25b59\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:30:25.666895 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.666839 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zl9c\" (UniqueName: \"kubernetes.io/projected/5975a38d-4a75-4604-9db8-f130c3d0ec0a-kube-api-access-9zl9c\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-25b59\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:30:25.743958 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.743933 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-tls\") pod \"korrel8r-66d6f7f696-9m2s2\" (UID: \"b3a97a3f-60bb-43b3-942f-1ff03e25f94e\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" Oct 30 03:30:25.744053 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.743979 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-config\") pod \"korrel8r-66d6f7f696-9m2s2\" (UID: \"b3a97a3f-60bb-43b3-942f-1ff03e25f94e\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" Oct 30 03:30:25.744140 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.744118 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgdz6\" (UniqueName: \"kubernetes.io/projected/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-kube-api-access-fgdz6\") pod \"korrel8r-66d6f7f696-9m2s2\" (UID: \"b3a97a3f-60bb-43b3-942f-1ff03e25f94e\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" Oct 30 03:30:25.744645 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.744626 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-config\") pod \"korrel8r-66d6f7f696-9m2s2\" (UID: \"b3a97a3f-60bb-43b3-942f-1ff03e25f94e\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" Oct 30 03:30:25.746171 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.746149 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-tls\") pod \"korrel8r-66d6f7f696-9m2s2\" (UID: \"b3a97a3f-60bb-43b3-942f-1ff03e25f94e\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" Oct 30 03:30:25.767548 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.767524 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgdz6\" (UniqueName: \"kubernetes.io/projected/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-kube-api-access-fgdz6\") pod \"korrel8r-66d6f7f696-9m2s2\" (UID: \"b3a97a3f-60bb-43b3-942f-1ff03e25f94e\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" Oct 30 03:30:25.858537 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:25.858511 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" Oct 30 03:30:26.146652 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:26.146620 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-25b59\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:30:26.148896 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:26.148870 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-25b59\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:30:26.154791 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:26.154745 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2"] Oct 30 03:30:26.157170 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:30:26.157145 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a97a3f_60bb_43b3_942f_1ff03e25f94e.slice/crio-94c632d4f9364a9cef87ab580622f21880af6b16ab69963f20c8e22f860c7cd8 WatchSource:0}: Error finding container 94c632d4f9364a9cef87ab580622f21880af6b16ab69963f20c8e22f860c7cd8: Status 404 returned error can't find the container with id 94c632d4f9364a9cef87ab580622f21880af6b16ab69963f20c8e22f860c7cd8 Oct 30 03:30:26.321795 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:26.321764 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:30:26.449068 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:26.448970 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" event={"ID":"b3a97a3f-60bb-43b3-942f-1ff03e25f94e","Type":"ContainerStarted","Data":"2cbdef1ff8817a9fe0ddbca18a8764adabc0acee8cebf80aa28232d8d8e48689"} Oct 30 03:30:26.449068 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:26.449034 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" event={"ID":"b3a97a3f-60bb-43b3-942f-1ff03e25f94e","Type":"ContainerStarted","Data":"94c632d4f9364a9cef87ab580622f21880af6b16ab69963f20c8e22f860c7cd8"} Oct 30 03:30:26.449568 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:26.449088 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" Oct 30 03:30:26.490475 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:26.490418 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" podStartSLOduration=1.490402996 podStartE2EDuration="1.490402996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:26.490333589 +0000 UTC m=+2987.503611691" watchObservedRunningTime="2026-10-30 03:30:26.490402996 +0000 UTC m=+2987.503681096" Oct 30 03:30:26.620169 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:26.620140 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59"] Oct 30 03:30:26.623172 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:30:26.623146 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5975a38d_4a75_4604_9db8_f130c3d0ec0a.slice/crio-370d92b858d9f31b31d317d15d7d438addf87ecd7b4733ceafee6e977958b666 WatchSource:0}: Error finding container 370d92b858d9f31b31d317d15d7d438addf87ecd7b4733ceafee6e977958b666: Status 404 returned error can't find the container with id 370d92b858d9f31b31d317d15d7d438addf87ecd7b4733ceafee6e977958b666 Oct 30 03:30:27.111695 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.111661 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:30:27.116815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.116790 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:30:27.137382 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.137358 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:30:27.148039 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.148016 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 03:30:27.148356 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.148336 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:30:27.148438 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.148422 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-72c79cac\"" Oct 30 03:30:27.148528 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.148511 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 03:30:27.253235 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.253152 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:30:27.253235 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.253217 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:30:27.253439 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.253334 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvdmr\" (UniqueName: \"kubernetes.io/projected/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-kube-api-access-zvdmr\") pod \"data-science-collector-collector-0\" (UID: \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:30:27.253439 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.253395 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:30:27.253439 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.253419 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:30:27.354190 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.354150 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:30:27.354190 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.354193 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:30:27.354433 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.354222 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvdmr\" (UniqueName: \"kubernetes.io/projected/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-kube-api-access-zvdmr\") pod \"data-science-collector-collector-0\" (UID: \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:30:27.354433 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.354254 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:30:27.354433 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.354268 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:30:27.355012 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.354981 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:30:27.356769 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.356734 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:30:27.356877 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.356845 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:30:27.356877 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.356848 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:30:27.378059 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.378021 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvdmr\" (UniqueName: \"kubernetes.io/projected/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-kube-api-access-zvdmr\") pod \"data-science-collector-collector-0\" (UID: \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:30:27.430661 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.430625 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:30:27.457611 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.457577 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" event={"ID":"5975a38d-4a75-4604-9db8-f130c3d0ec0a","Type":"ContainerStarted","Data":"341c6926b1c602f39aa5c694dfebfe2402e039926f0e2274fabdd98e5c099192"} Oct 30 03:30:27.457997 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.457640 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" event={"ID":"5975a38d-4a75-4604-9db8-f130c3d0ec0a","Type":"ContainerStarted","Data":"370d92b858d9f31b31d317d15d7d438addf87ecd7b4733ceafee6e977958b666"} Oct 30 03:30:27.457997 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.457808 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:30:27.502319 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.502212 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" podStartSLOduration=2.502194205 podStartE2EDuration="2.502194205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:27.499226783 +0000 UTC m=+2988.512504885" watchObservedRunningTime="2026-10-30 03:30:27.502194205 +0000 UTC m=+2988.515472307" Oct 30 03:30:27.729910 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.729871 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:30:27.732337 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:30:27.732307 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32bd2a5c_0666_4ab4_a163_9f3b3783e6c7.slice/crio-ec2b3882e26cd404571e1b2e616838666588b9c84b0576052471fd152c4cdf23 WatchSource:0}: Error finding container ec2b3882e26cd404571e1b2e616838666588b9c84b0576052471fd152c4cdf23: Status 404 returned error can't find the container with id ec2b3882e26cd404571e1b2e616838666588b9c84b0576052471fd152c4cdf23 Oct 30 03:30:27.734854 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.734820 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:30:27.734933 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:27.734899 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:30:28.464662 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:28.464623 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7","Type":"ContainerStarted","Data":"c6ed74c79fdf423fb101edc6440ff4263768a633acc9be0fe0d0713e5672437b"} Oct 30 03:30:28.465067 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:28.464687 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7","Type":"ContainerStarted","Data":"ec2b3882e26cd404571e1b2e616838666588b9c84b0576052471fd152c4cdf23"} Oct 30 03:30:28.507938 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:28.507895 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.507883762 podStartE2EDuration="1.507883762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:28.50749182 +0000 UTC m=+2989.520769922" watchObservedRunningTime="2026-10-30 03:30:28.507883762 +0000 UTC m=+2989.521161863" Oct 30 03:30:31.404188 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.404153 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 03:30:31.408112 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.408088 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.424001 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.423977 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 03:30:31.424130 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.424030 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 03:30:31.424130 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.424055 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 03:30:31.424130 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.424059 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 03:30:31.436657 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.436636 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-zdtxd\"" Oct 30 03:30:31.439315 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.439289 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 03:30:31.487377 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.487347 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.487377 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.487378 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.487556 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.487406 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/7e184cfe-90df-4725-bae5-3ddede59c6bf-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.487556 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.487535 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.487617 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.487574 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e184cfe-90df-4725-bae5-3ddede59c6bf-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.487713 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.487691 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8msx\" (UniqueName: \"kubernetes.io/projected/7e184cfe-90df-4725-bae5-3ddede59c6bf-kube-api-access-c8msx\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.487774 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.487734 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e184cfe-90df-4725-bae5-3ddede59c6bf-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.530828 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.530798 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 03:30:31.534540 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.534525 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.555736 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.555713 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 03:30:31.555909 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.555885 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 03:30:31.555972 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.555923 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 03:30:31.563643 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.563621 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 03:30:31.588446 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.588411 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8msx\" (UniqueName: \"kubernetes.io/projected/7e184cfe-90df-4725-bae5-3ddede59c6bf-kube-api-access-c8msx\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.588557 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.588455 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e184cfe-90df-4725-bae5-3ddede59c6bf-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.588557 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.588487 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q97sl\" (UniqueName: \"kubernetes.io/projected/b0db7e1c-3383-430b-8b0f-64507e87db6c-kube-api-access-q97sl\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.588557 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.588518 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.588557 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.588556 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b0db7e1c-3383-430b-8b0f-64507e87db6c-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.588870 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.588691 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.588870 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.588722 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.588870 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.588766 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/7e184cfe-90df-4725-bae5-3ddede59c6bf-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.588870 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.588793 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b0db7e1c-3383-430b-8b0f-64507e87db6c-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.588870 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.588854 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.589177 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.588888 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.589177 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.588922 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e184cfe-90df-4725-bae5-3ddede59c6bf-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.589177 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.588950 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.589419 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.589399 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.589491 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.589430 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.589491 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.589483 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.589601 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.589510 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.589601 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.589568 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.589701 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.589597 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.590435 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.590415 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/7e184cfe-90df-4725-bae5-3ddede59c6bf-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.591670 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.591461 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e184cfe-90df-4725-bae5-3ddede59c6bf-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.591881 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.591858 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.592252 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.592225 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e184cfe-90df-4725-bae5-3ddede59c6bf-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.592326 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.592254 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.593567 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.593547 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.612847 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.612823 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8msx\" (UniqueName: \"kubernetes.io/projected/7e184cfe-90df-4725-bae5-3ddede59c6bf-kube-api-access-c8msx\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:31.624267 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.624241 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 03:30:31.624455 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.624429 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 03:30:31.624553 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.624470 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 03:30:31.624553 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.624500 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 03:30:31.624553 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.624502 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 03:30:31.624649 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.624590 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-4s68n\"" Oct 30 03:30:31.690341 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.690277 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.690341 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.690310 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.690341 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.690330 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.690563 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.690364 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.690563 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.690488 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.690563 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.690555 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.690720 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.690585 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.690720 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.690632 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q97sl\" (UniqueName: \"kubernetes.io/projected/b0db7e1c-3383-430b-8b0f-64507e87db6c-kube-api-access-q97sl\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.690720 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.690662 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.690720 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.690704 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b0db7e1c-3383-430b-8b0f-64507e87db6c-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.690936 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.690777 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b0db7e1c-3383-430b-8b0f-64507e87db6c-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.690936 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.690876 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.691421 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.691222 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.691544 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.691415 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.691544 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.691439 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.693299 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.693274 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.694043 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.694021 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b0db7e1c-3383-430b-8b0f-64507e87db6c-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.694330 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.694300 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.694413 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.694324 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.694413 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.694390 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.694664 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.694648 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b0db7e1c-3383-430b-8b0f-64507e87db6c-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.694726 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.694704 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.694986 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.694967 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.714357 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.714339 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q97sl\" (UniqueName: \"kubernetes.io/projected/b0db7e1c-3383-430b-8b0f-64507e87db6c-kube-api-access-q97sl\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:31.725154 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:31.725137 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:32.043331 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:32.043304 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 03:30:32.045921 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:30:32.045884 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e184cfe_90df_4725_bae5_3ddede59c6bf.slice/crio-d0ed4c040a3aae6bce294ee307d9f37d0b5762de93f9fc2c66607b5c801cb419 WatchSource:0}: Error finding container d0ed4c040a3aae6bce294ee307d9f37d0b5762de93f9fc2c66607b5c801cb419: Status 404 returned error can't find the container with id d0ed4c040a3aae6bce294ee307d9f37d0b5762de93f9fc2c66607b5c801cb419 Oct 30 03:30:32.485608 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:32.485575 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"7e184cfe-90df-4725-bae5-3ddede59c6bf","Type":"ContainerStarted","Data":"29a3cd2f2953db55ef49f3643116bb76cb46d794790360b0628f17c3bbe4f21e"} Oct 30 03:30:32.485977 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:32.485643 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"7e184cfe-90df-4725-bae5-3ddede59c6bf","Type":"ContainerStarted","Data":"d0ed4c040a3aae6bce294ee307d9f37d0b5762de93f9fc2c66607b5c801cb419"} Oct 30 03:30:33.467510 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:33.467482 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:30:35.727501 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:35.727464 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:35.757442 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:35.757413 4925 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:35.828069 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:35.828032 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:35.828229 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:35.828204 4925 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:35.842983 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:35.842964 4925 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") DevicePath \"csi-77aa142b6cc95d8f79f4aef80349e4d929aed6d462958597286d83ed1cc8fc2c\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:35.944596 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:35.944570 4925 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7bd02a6ced0e86eb5067dae735620268cfbf25efbbfc3866c7163efbdf26dd99/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:35.970924 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:35.970903 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:36.053709 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:36.053656 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:36.387557 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:36.387529 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 03:30:36.389921 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:30:36.389894 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0db7e1c_3383_430b_8b0f_64507e87db6c.slice/crio-51a1aa10c35e7e6b952b89943d6d3f7a8bd7cf9139839b269fa47c1f0d0b21b1 WatchSource:0}: Error finding container 51a1aa10c35e7e6b952b89943d6d3f7a8bd7cf9139839b269fa47c1f0d0b21b1: Status 404 returned error can't find the container with id 51a1aa10c35e7e6b952b89943d6d3f7a8bd7cf9139839b269fa47c1f0d0b21b1 Oct 30 03:30:36.506218 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:36.506188 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"b0db7e1c-3383-430b-8b0f-64507e87db6c","Type":"ContainerStarted","Data":"51a1aa10c35e7e6b952b89943d6d3f7a8bd7cf9139839b269fa47c1f0d0b21b1"} Oct 30 03:30:36.509504 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:36.508436 4925 generic.go:356] "Generic (PLEG): container finished" podID="7e184cfe-90df-4725-bae5-3ddede59c6bf" containerID="29a3cd2f2953db55ef49f3643116bb76cb46d794790360b0628f17c3bbe4f21e" exitCode=0 Oct 30 03:30:36.509504 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:36.508526 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"7e184cfe-90df-4725-bae5-3ddede59c6bf","Type":"ContainerDied","Data":"29a3cd2f2953db55ef49f3643116bb76cb46d794790360b0628f17c3bbe4f21e"} Oct 30 03:30:37.460085 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:37.460057 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" Oct 30 03:30:37.517623 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:37.517579 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"7e184cfe-90df-4725-bae5-3ddede59c6bf","Type":"ContainerStarted","Data":"4cfb77077bb0171a11e48cee12f8cf77d8c3207e6a5427471a442786eb8c6312"} Oct 30 03:30:37.517801 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:37.517662 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"7e184cfe-90df-4725-bae5-3ddede59c6bf","Type":"ContainerStarted","Data":"394d950675de19b9fb2c6af7aa0fdb6451b2b56068faea62428a80cbf2b94288"} Oct 30 03:30:37.518119 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:37.518072 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:37.519283 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:37.519257 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"b0db7e1c-3383-430b-8b0f-64507e87db6c","Type":"ContainerStarted","Data":"45099e76f59a007d1598fd840a404d70b924c4805af6cafc58c467a22a26ddf8"} Oct 30 03:30:37.566669 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:37.566623 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.180380571 podStartE2EDuration="6.566608655s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.386228084s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:37.565609463 +0000 UTC m=+2998.578887578" watchObservedRunningTime="2026-10-30 03:30:37.566608655 +0000 UTC m=+2998.579886761" Oct 30 03:30:39.407552 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:39.407440 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:30:39.438518 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:39.416154 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:30:39.438518 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:39.417737 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:30:39.438518 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:39.426328 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:30:40.535436 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:40.535409 4925 generic.go:356] "Generic (PLEG): container finished" podID="b0db7e1c-3383-430b-8b0f-64507e87db6c" containerID="45099e76f59a007d1598fd840a404d70b924c4805af6cafc58c467a22a26ddf8" exitCode=0 Oct 30 03:30:40.535765 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:40.535474 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"b0db7e1c-3383-430b-8b0f-64507e87db6c","Type":"ContainerDied","Data":"45099e76f59a007d1598fd840a404d70b924c4805af6cafc58c467a22a26ddf8"} Oct 30 03:30:41.542841 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:41.542807 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"b0db7e1c-3383-430b-8b0f-64507e87db6c","Type":"ContainerStarted","Data":"727fc5c57caaecfb30f592d983ef54da71682e326dc53f642a87facbe46a1f59"} Oct 30 03:30:41.543219 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:41.542876 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"b0db7e1c-3383-430b-8b0f-64507e87db6c","Type":"ContainerStarted","Data":"f43a319e2d26aef1419e890f952e3e192acfef64e1b5e277e10fb2fe254fd018"} Oct 30 03:30:41.543219 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:41.542899 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"b0db7e1c-3383-430b-8b0f-64507e87db6c","Type":"ContainerStarted","Data":"914a92b7f570547e2bc138c83c57dea930f0b56777754ee7fa94705333118437"} Oct 30 03:30:41.620467 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:41.620416 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=7.234196492 podStartE2EDuration="10.620401682s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.38620519s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:41.606988323 +0000 UTC m=+3002.620266425" watchObservedRunningTime="2026-10-30 03:30:41.620401682 +0000 UTC m=+3002.633679783" Oct 30 03:30:43.525100 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:43.525072 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:30:46.054686 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:46.054654 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:48.160660 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:30:48.160625 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:30:51.054327 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:51.054224 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:51.059595 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:51.059572 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:30:51.593526 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:30:51.593495 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:31:18.163976 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:31:18.163941 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:31:48.161826 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:31:48.161790 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:32:18.162390 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:32:18.162352 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:32:48.162729 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:32:48.162654 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:33:18.161326 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:33:18.161289 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:33:48.162097 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:33:48.162062 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:34:18.160487 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:34:18.160448 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:34:48.161617 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:34:48.161579 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:35:18.161588 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:35:18.161548 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:35:39.446044 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:35:39.445948 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:35:39.453980 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:35:39.453958 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:35:39.454628 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:35:39.454611 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:35:39.462029 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:35:39.462006 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:35:48.161193 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:35:48.161152 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:36:18.161737 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:36:18.161692 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:36:48.162099 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:36:48.162060 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:37:18.165267 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:37:18.165221 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:37:48.161458 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:37:48.161416 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:38:18.162183 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:38:18.162145 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:38:48.161929 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:38:48.161847 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:39:18.161797 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:39:18.161740 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:39:48.161955 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:39:48.161915 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:40:18.161097 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:40:18.161056 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:40:39.478576 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:40:39.478477 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:40:39.486903 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:40:39.486883 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:40:39.487037 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:40:39.486961 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:40:39.495270 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:40:39.495255 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:40:48.161625 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:40:48.161589 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:41:18.161145 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:41:18.161101 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:41:21.397799 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.397699 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59"] Oct 30 03:41:21.398286 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.397926 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" podUID="5975a38d-4a75-4604-9db8-f130c3d0ec0a" containerName="kube-rbac-proxy" containerID="cri-o://341c6926b1c602f39aa5c694dfebfe2402e039926f0e2274fabdd98e5c099192" gracePeriod=30 Oct 30 03:41:21.444913 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.444880 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2"] Oct 30 03:41:21.445101 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.445081 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" podUID="b3a97a3f-60bb-43b3-942f-1ff03e25f94e" containerName="korrel8r" containerID="cri-o://2cbdef1ff8817a9fe0ddbca18a8764adabc0acee8cebf80aa28232d8d8e48689" gracePeriod=30 Oct 30 03:41:21.650963 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.650888 4925 generic.go:356] "Generic (PLEG): container finished" podID="5975a38d-4a75-4604-9db8-f130c3d0ec0a" containerID="341c6926b1c602f39aa5c694dfebfe2402e039926f0e2274fabdd98e5c099192" exitCode=0 Oct 30 03:41:21.650963 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.650914 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" event={"ID":"5975a38d-4a75-4604-9db8-f130c3d0ec0a","Type":"ContainerDied","Data":"341c6926b1c602f39aa5c694dfebfe2402e039926f0e2274fabdd98e5c099192"} Oct 30 03:41:21.652959 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.652857 4925 generic.go:356] "Generic (PLEG): container finished" podID="b3a97a3f-60bb-43b3-942f-1ff03e25f94e" containerID="2cbdef1ff8817a9fe0ddbca18a8764adabc0acee8cebf80aa28232d8d8e48689" exitCode=0 Oct 30 03:41:21.652959 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.652922 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" event={"ID":"b3a97a3f-60bb-43b3-942f-1ff03e25f94e","Type":"ContainerDied","Data":"2cbdef1ff8817a9fe0ddbca18a8764adabc0acee8cebf80aa28232d8d8e48689"} Oct 30 03:41:21.705352 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.705319 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:41:21.708267 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.708246 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" Oct 30 03:41:21.750185 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.750150 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-tls\" (UniqueName: \"kubernetes.io/secret/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-tls\") pod \"b3a97a3f-60bb-43b3-942f-1ff03e25f94e\" (UID: \"b3a97a3f-60bb-43b3-942f-1ff03e25f94e\") " Oct 30 03:41:21.750384 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.750214 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-prometheus-client-cert\") pod \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " Oct 30 03:41:21.750384 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.750246 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-kube-rbac-proxy-config\") pod \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " Oct 30 03:41:21.750384 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.750272 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-tls\" (UniqueName: \"kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-tls\") pod \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " Oct 30 03:41:21.750384 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.750337 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-config\" (UniqueName: \"kubernetes.io/configmap/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-config\") pod \"b3a97a3f-60bb-43b3-942f-1ff03e25f94e\" (UID: \"b3a97a3f-60bb-43b3-942f-1ff03e25f94e\") " Oct 30 03:41:21.750384 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.750380 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5975a38d-4a75-4604-9db8-f130c3d0ec0a-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/5975a38d-4a75-4604-9db8-f130c3d0ec0a-prometheus-ca\") pod \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " Oct 30 03:41:21.755179 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.751311 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-config" pod "b3a97a3f-60bb-43b3-942f-1ff03e25f94e" (UID: "b3a97a3f-60bb-43b3-942f-1ff03e25f94e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:41:21.755179 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.752147 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5975a38d-4a75-4604-9db8-f130c3d0ec0a-prometheus-ca" pod "5975a38d-4a75-4604-9db8-f130c3d0ec0a" (UID: "5975a38d-4a75-4604-9db8-f130c3d0ec0a"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:41:21.755179 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.753591 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5975a38d-4a75-4604-9db8-f130c3d0ec0a-kube-api-access-9zl9c\" (UniqueName: \"kubernetes.io/projected/5975a38d-4a75-4604-9db8-f130c3d0ec0a-kube-api-access-9zl9c\") pod \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\" (UID: \"5975a38d-4a75-4604-9db8-f130c3d0ec0a\") " Oct 30 03:41:21.755179 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.753680 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-kube-api-access-fgdz6\" (UniqueName: \"kubernetes.io/projected/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-kube-api-access-fgdz6\") pod \"b3a97a3f-60bb-43b3-942f-1ff03e25f94e\" (UID: \"b3a97a3f-60bb-43b3-942f-1ff03e25f94e\") " Oct 30 03:41:21.755179 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.754011 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/5975a38d-4a75-4604-9db8-f130c3d0ec0a-prometheus-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:21.755179 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.754032 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:21.756492 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.756351 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-tls" pod "5975a38d-4a75-4604-9db8-f130c3d0ec0a" (UID: "5975a38d-4a75-4604-9db8-f130c3d0ec0a"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:41:21.756806 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.756762 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-tls" pod "b3a97a3f-60bb-43b3-942f-1ff03e25f94e" (UID: "b3a97a3f-60bb-43b3-942f-1ff03e25f94e"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:41:21.756806 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.756775 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-kube-rbac-proxy-config" pod "5975a38d-4a75-4604-9db8-f130c3d0ec0a" (UID: "5975a38d-4a75-4604-9db8-f130c3d0ec0a"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:41:21.759227 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.759201 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-kube-api-access-fgdz6" pod "b3a97a3f-60bb-43b3-942f-1ff03e25f94e" (UID: "b3a97a3f-60bb-43b3-942f-1ff03e25f94e"). InnerVolumeSpecName "kube-api-access-fgdz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:41:21.759437 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.759404 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-prometheus-client-cert" pod "5975a38d-4a75-4604-9db8-f130c3d0ec0a" (UID: "5975a38d-4a75-4604-9db8-f130c3d0ec0a"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:41:21.760654 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.760618 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5975a38d-4a75-4604-9db8-f130c3d0ec0a-kube-api-access-9zl9c" pod "5975a38d-4a75-4604-9db8-f130c3d0ec0a" (UID: "5975a38d-4a75-4604-9db8-f130c3d0ec0a"). InnerVolumeSpecName "kube-api-access-9zl9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:41:21.855451 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.855417 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:21.855451 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.855445 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-prometheus-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:21.855451 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.855457 4925 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-kube-rbac-proxy-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:21.855659 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.855465 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5975a38d-4a75-4604-9db8-f130c3d0ec0a-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:21.855659 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.855474 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zl9c\" (UniqueName: \"kubernetes.io/projected/5975a38d-4a75-4604-9db8-f130c3d0ec0a-kube-api-access-9zl9c\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:21.855659 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.855482 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgdz6\" (UniqueName: \"kubernetes.io/projected/b3a97a3f-60bb-43b3-942f-1ff03e25f94e-kube-api-access-fgdz6\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:21.895009 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.894968 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 03:41:21.895314 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.895270 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="b0db7e1c-3383-430b-8b0f-64507e87db6c" containerName="prometheus" containerID="cri-o://914a92b7f570547e2bc138c83c57dea930f0b56777754ee7fa94705333118437" gracePeriod=600 Oct 30 03:41:21.895448 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.895310 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="b0db7e1c-3383-430b-8b0f-64507e87db6c" containerName="thanos-sidecar" containerID="cri-o://727fc5c57caaecfb30f592d983ef54da71682e326dc53f642a87facbe46a1f59" gracePeriod=600 Oct 30 03:41:21.895448 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.895374 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="b0db7e1c-3383-430b-8b0f-64507e87db6c" containerName="config-reloader" containerID="cri-o://f43a319e2d26aef1419e890f952e3e192acfef64e1b5e277e10fb2fe254fd018" gracePeriod=600 Oct 30 03:41:21.909776 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.908550 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 03:41:21.909776 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.908855 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="7e184cfe-90df-4725-bae5-3ddede59c6bf" containerName="alertmanager" containerID="cri-o://394d950675de19b9fb2c6af7aa0fdb6451b2b56068faea62428a80cbf2b94288" gracePeriod=120 Oct 30 03:41:21.909776 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:21.908967 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="7e184cfe-90df-4725-bae5-3ddede59c6bf" containerName="config-reloader" containerID="cri-o://4cfb77077bb0171a11e48cee12f8cf77d8c3207e6a5427471a442786eb8c6312" gracePeriod=120 Oct 30 03:41:22.658866 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.658835 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" event={"ID":"5975a38d-4a75-4604-9db8-f130c3d0ec0a","Type":"ContainerDied","Data":"370d92b858d9f31b31d317d15d7d438addf87ecd7b4733ceafee6e977958b666"} Oct 30 03:41:22.658866 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.658860 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59" Oct 30 03:41:22.659239 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.658957 4925 scope.go:122] "RemoveContainer" containerID="341c6926b1c602f39aa5c694dfebfe2402e039926f0e2274fabdd98e5c099192" Oct 30 03:41:22.661212 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.661168 4925 generic.go:356] "Generic (PLEG): container finished" podID="7e184cfe-90df-4725-bae5-3ddede59c6bf" containerID="4cfb77077bb0171a11e48cee12f8cf77d8c3207e6a5427471a442786eb8c6312" exitCode=0 Oct 30 03:41:22.661306 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.661218 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"7e184cfe-90df-4725-bae5-3ddede59c6bf","Type":"ContainerDied","Data":"4cfb77077bb0171a11e48cee12f8cf77d8c3207e6a5427471a442786eb8c6312"} Oct 30 03:41:22.662683 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.662667 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" Oct 30 03:41:22.662783 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.662684 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2" event={"ID":"b3a97a3f-60bb-43b3-942f-1ff03e25f94e","Type":"ContainerDied","Data":"94c632d4f9364a9cef87ab580622f21880af6b16ab69963f20c8e22f860c7cd8"} Oct 30 03:41:22.665291 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.665270 4925 generic.go:356] "Generic (PLEG): container finished" podID="b0db7e1c-3383-430b-8b0f-64507e87db6c" containerID="727fc5c57caaecfb30f592d983ef54da71682e326dc53f642a87facbe46a1f59" exitCode=0 Oct 30 03:41:22.665373 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.665294 4925 generic.go:356] "Generic (PLEG): container finished" podID="b0db7e1c-3383-430b-8b0f-64507e87db6c" containerID="f43a319e2d26aef1419e890f952e3e192acfef64e1b5e277e10fb2fe254fd018" exitCode=0 Oct 30 03:41:22.665373 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.665303 4925 generic.go:356] "Generic (PLEG): container finished" podID="b0db7e1c-3383-430b-8b0f-64507e87db6c" containerID="914a92b7f570547e2bc138c83c57dea930f0b56777754ee7fa94705333118437" exitCode=0 Oct 30 03:41:22.665373 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.665347 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"b0db7e1c-3383-430b-8b0f-64507e87db6c","Type":"ContainerDied","Data":"727fc5c57caaecfb30f592d983ef54da71682e326dc53f642a87facbe46a1f59"} Oct 30 03:41:22.665480 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.665410 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"b0db7e1c-3383-430b-8b0f-64507e87db6c","Type":"ContainerDied","Data":"f43a319e2d26aef1419e890f952e3e192acfef64e1b5e277e10fb2fe254fd018"} Oct 30 03:41:22.665480 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.665448 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"b0db7e1c-3383-430b-8b0f-64507e87db6c","Type":"ContainerDied","Data":"914a92b7f570547e2bc138c83c57dea930f0b56777754ee7fa94705333118437"} Oct 30 03:41:22.665538 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.665486 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"b0db7e1c-3383-430b-8b0f-64507e87db6c","Type":"ContainerDied","Data":"51a1aa10c35e7e6b952b89943d6d3f7a8bd7cf9139839b269fa47c1f0d0b21b1"} Oct 30 03:41:22.665538 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.665521 4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51a1aa10c35e7e6b952b89943d6d3f7a8bd7cf9139839b269fa47c1f0d0b21b1" Oct 30 03:41:22.700953 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.700936 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:41:22.724925 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.724904 4925 scope.go:122] "RemoveContainer" containerID="2cbdef1ff8817a9fe0ddbca18a8764adabc0acee8cebf80aa28232d8d8e48689" Oct 30 03:41:22.740895 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.740470 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59"] Oct 30 03:41:22.743010 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.742987 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-25b59"] Oct 30 03:41:22.762354 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.762317 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:41:22.762599 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.762570 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="32bd2a5c-0666-4ab4-a163-9f3b3783e6c7" containerName="otc-container" containerID="cri-o://c6ed74c79fdf423fb101edc6440ff4263768a633acc9be0fe0d0713e5672437b" gracePeriod=30 Oct 30 03:41:22.764516 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.764053 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-secret-prometheus-web-tls-ca\") pod \"b0db7e1c-3383-430b-8b0f-64507e87db6c\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " Oct 30 03:41:22.764516 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.764091 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"b0db7e1c-3383-430b-8b0f-64507e87db6c\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " Oct 30 03:41:22.764516 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.764112 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"b0db7e1c-3383-430b-8b0f-64507e87db6c\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " Oct 30 03:41:22.764516 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.764133 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"b0db7e1c-3383-430b-8b0f-64507e87db6c\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " Oct 30 03:41:22.764516 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.764151 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-thanos-prometheus-http-client-file\") pod \"b0db7e1c-3383-430b-8b0f-64507e87db6c\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " Oct 30 03:41:22.764516 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.764223 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"b0db7e1c-3383-430b-8b0f-64507e87db6c\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " Oct 30 03:41:22.764516 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.764264 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0db7e1c-3383-430b-8b0f-64507e87db6c-kube-api-access-q97sl\" (UniqueName: \"kubernetes.io/projected/b0db7e1c-3383-430b-8b0f-64507e87db6c-kube-api-access-q97sl\") pod \"b0db7e1c-3383-430b-8b0f-64507e87db6c\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " Oct 30 03:41:22.764516 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.764281 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config\") pod \"b0db7e1c-3383-430b-8b0f-64507e87db6c\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " Oct 30 03:41:22.764516 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.764315 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0db7e1c-3383-430b-8b0f-64507e87db6c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b0db7e1c-3383-430b-8b0f-64507e87db6c-config-out\") pod \"b0db7e1c-3383-430b-8b0f-64507e87db6c\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " Oct 30 03:41:22.764516 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.764337 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"b0db7e1c-3383-430b-8b0f-64507e87db6c\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " Oct 30 03:41:22.764516 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.764387 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-config\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-config\") pod \"b0db7e1c-3383-430b-8b0f-64507e87db6c\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " Oct 30 03:41:22.764516 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.764411 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0db7e1c-3383-430b-8b0f-64507e87db6c-tls-assets\" (UniqueName: \"kubernetes.io/projected/b0db7e1c-3383-430b-8b0f-64507e87db6c-tls-assets\") pod \"b0db7e1c-3383-430b-8b0f-64507e87db6c\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " Oct 30 03:41:22.764516 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.764443 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"b0db7e1c-3383-430b-8b0f-64507e87db6c\" (UID: \"b0db7e1c-3383-430b-8b0f-64507e87db6c\") " Oct 30 03:41:22.765629 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.765409 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-1" pod "b0db7e1c-3383-430b-8b0f-64507e87db6c" (UID: "b0db7e1c-3383-430b-8b0f-64507e87db6c"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:41:22.769162 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.768995 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-0" pod "b0db7e1c-3383-430b-8b0f-64507e87db6c" (UID: "b0db7e1c-3383-430b-8b0f-64507e87db6c"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:41:22.769782 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.769503 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-2" pod "b0db7e1c-3383-430b-8b0f-64507e87db6c" (UID: "b0db7e1c-3383-430b-8b0f-64507e87db6c"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:41:22.787352 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.786977 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0db7e1c-3383-430b-8b0f-64507e87db6c-kube-api-access-q97sl" pod "b0db7e1c-3383-430b-8b0f-64507e87db6c" (UID: "b0db7e1c-3383-430b-8b0f-64507e87db6c"). InnerVolumeSpecName "kube-api-access-q97sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:41:22.787352 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.787026 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2"] Oct 30 03:41:22.787352 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.787204 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "b0db7e1c-3383-430b-8b0f-64507e87db6c" (UID: "b0db7e1c-3383-430b-8b0f-64507e87db6c"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:41:22.787352 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.787321 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-secret-prometheus-web-tls-ca" pod "b0db7e1c-3383-430b-8b0f-64507e87db6c" (UID: "b0db7e1c-3383-430b-8b0f-64507e87db6c"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:41:22.788057 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.788027 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "b0db7e1c-3383-430b-8b0f-64507e87db6c" (UID: "b0db7e1c-3383-430b-8b0f-64507e87db6c"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:41:22.788323 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.788204 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-thanos-prometheus-http-client-file" pod "b0db7e1c-3383-430b-8b0f-64507e87db6c" (UID: "b0db7e1c-3383-430b-8b0f-64507e87db6c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:41:22.788492 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.788456 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-config" pod "b0db7e1c-3383-430b-8b0f-64507e87db6c" (UID: "b0db7e1c-3383-430b-8b0f-64507e87db6c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:41:22.791421 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.791404 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-9m2s2"] Oct 30 03:41:22.792135 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.792114 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0db7e1c-3383-430b-8b0f-64507e87db6c-config-out" pod "b0db7e1c-3383-430b-8b0f-64507e87db6c" (UID: "b0db7e1c-3383-430b-8b0f-64507e87db6c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:41:22.792675 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.792581 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0db7e1c-3383-430b-8b0f-64507e87db6c-tls-assets" pod "b0db7e1c-3383-430b-8b0f-64507e87db6c" (UID: "b0db7e1c-3383-430b-8b0f-64507e87db6c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:41:22.798521 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.798120 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085" pod "b0db7e1c-3383-430b-8b0f-64507e87db6c" (UID: "b0db7e1c-3383-430b-8b0f-64507e87db6c"). InnerVolumeSpecName "pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 03:41:22.805655 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.805613 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config" pod "b0db7e1c-3383-430b-8b0f-64507e87db6c" (UID: "b0db7e1c-3383-430b-8b0f-64507e87db6c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:41:22.866385 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.866337 4925 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b0db7e1c-3383-430b-8b0f-64507e87db6c-config-out\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:22.866385 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.866377 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:22.866385 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.866392 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:22.866649 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.866402 4925 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b0db7e1c-3383-430b-8b0f-64507e87db6c-tls-assets\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:22.866649 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.866411 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:22.866649 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.866420 4925 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-secret-prometheus-web-tls-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:22.866649 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.866429 4925 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:22.866649 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.866439 4925 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:22.866649 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.866449 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b0db7e1c-3383-430b-8b0f-64507e87db6c-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:22.866649 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.866462 4925 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:22.866649 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.866488 4925 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") on node \"ip-10-0-139-19.ec2.internal\" " Oct 30 03:41:22.866649 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.866497 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q97sl\" (UniqueName: \"kubernetes.io/projected/b0db7e1c-3383-430b-8b0f-64507e87db6c-kube-api-access-q97sl\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:22.866649 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.866505 4925 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b0db7e1c-3383-430b-8b0f-64507e87db6c-web-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:22.879152 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.879131 4925 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085") on node "ip-10-0-139-19.ec2.internal" Oct 30 03:41:22.921527 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.921464 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:22.967451 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.967419 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e184cfe-90df-4725-bae5-3ddede59c6bf-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/7e184cfe-90df-4725-bae5-3ddede59c6bf-alertmanager-data-science-monitoringstack-db\") pod \"7e184cfe-90df-4725-bae5-3ddede59c6bf\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " Oct 30 03:41:22.967596 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.967481 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e184cfe-90df-4725-bae5-3ddede59c6bf-tls-assets\" (UniqueName: \"kubernetes.io/projected/7e184cfe-90df-4725-bae5-3ddede59c6bf-tls-assets\") pod \"7e184cfe-90df-4725-bae5-3ddede59c6bf\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " Oct 30 03:41:22.967596 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.967520 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-web-config\" (UniqueName: \"kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-web-config\") pod \"7e184cfe-90df-4725-bae5-3ddede59c6bf\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " Oct 30 03:41:22.967596 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.967547 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-config-volume\" (UniqueName: \"kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-config-volume\") pod \"7e184cfe-90df-4725-bae5-3ddede59c6bf\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " Oct 30 03:41:22.967841 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.967608 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e184cfe-90df-4725-bae5-3ddede59c6bf-config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e184cfe-90df-4725-bae5-3ddede59c6bf-config-out\") pod \"7e184cfe-90df-4725-bae5-3ddede59c6bf\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " Oct 30 03:41:22.967841 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.967666 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e184cfe-90df-4725-bae5-3ddede59c6bf-kube-api-access-c8msx\" (UniqueName: \"kubernetes.io/projected/7e184cfe-90df-4725-bae5-3ddede59c6bf-kube-api-access-c8msx\") pod \"7e184cfe-90df-4725-bae5-3ddede59c6bf\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " Oct 30 03:41:22.967841 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.967728 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-cluster-tls-config\") pod \"7e184cfe-90df-4725-bae5-3ddede59c6bf\" (UID: \"7e184cfe-90df-4725-bae5-3ddede59c6bf\") " Oct 30 03:41:22.968006 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.967986 4925 reconciler_common.go:299] "Volume detached for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"csi-77aa142b6cc95d8f79f4aef80349e4d929aed6d462958597286d83ed1cc8fc2c\"" Oct 30 03:41:22.968077 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.968049 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e184cfe-90df-4725-bae5-3ddede59c6bf-alertmanager-data-science-monitoringstack-db" pod "7e184cfe-90df-4725-bae5-3ddede59c6bf" (UID: "7e184cfe-90df-4725-bae5-3ddede59c6bf"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:41:22.974725 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.974694 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e184cfe-90df-4725-bae5-3ddede59c6bf-kube-api-access-c8msx" pod "7e184cfe-90df-4725-bae5-3ddede59c6bf" (UID: "7e184cfe-90df-4725-bae5-3ddede59c6bf"). InnerVolumeSpecName "kube-api-access-c8msx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:41:22.976305 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.976277 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-config-volume" pod "7e184cfe-90df-4725-bae5-3ddede59c6bf" (UID: "7e184cfe-90df-4725-bae5-3ddede59c6bf"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:41:22.976419 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.976347 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e184cfe-90df-4725-bae5-3ddede59c6bf-tls-assets" pod "7e184cfe-90df-4725-bae5-3ddede59c6bf" (UID: "7e184cfe-90df-4725-bae5-3ddede59c6bf"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:41:22.978283 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.978260 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-cluster-tls-config" pod "7e184cfe-90df-4725-bae5-3ddede59c6bf" (UID: "7e184cfe-90df-4725-bae5-3ddede59c6bf"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:41:22.978404 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.978332 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e184cfe-90df-4725-bae5-3ddede59c6bf-config-out" pod "7e184cfe-90df-4725-bae5-3ddede59c6bf" (UID: "7e184cfe-90df-4725-bae5-3ddede59c6bf"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:41:22.986942 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:22.986922 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-web-config" pod "7e184cfe-90df-4725-bae5-3ddede59c6bf" (UID: "7e184cfe-90df-4725-bae5-3ddede59c6bf"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:41:23.000990 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.000969 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:41:23.068338 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.068305 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-monitoring-tls-certs\") pod \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\" (UID: \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\") " Oct 30 03:41:23.068481 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.068406 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-otc-internal\" (UniqueName: \"kubernetes.io/configmap/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-otc-internal\") pod \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\" (UID: \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\") " Oct 30 03:41:23.068481 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.068446 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-tls-certs\") pod \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\" (UID: \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\") " Oct 30 03:41:23.068567 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.068522 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-data-science-collector-ta-client-cert\") pod \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\" (UID: \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\") " Oct 30 03:41:23.068567 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.068556 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-kube-api-access-zvdmr\" (UniqueName: \"kubernetes.io/projected/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-kube-api-access-zvdmr\") pod \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\" (UID: \"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7\") " Oct 30 03:41:23.068804 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.068783 4925 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e184cfe-90df-4725-bae5-3ddede59c6bf-config-out\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:23.068912 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.068805 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8msx\" (UniqueName: \"kubernetes.io/projected/7e184cfe-90df-4725-bae5-3ddede59c6bf-kube-api-access-c8msx\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:23.068912 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.068819 4925 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-cluster-tls-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:23.068912 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.068826 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-otc-internal" pod "32bd2a5c-0666-4ab4-a163-9f3b3783e6c7" (UID: "32bd2a5c-0666-4ab4-a163-9f3b3783e6c7"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:41:23.068912 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.068833 4925 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/7e184cfe-90df-4725-bae5-3ddede59c6bf-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:23.068912 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.068859 4925 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e184cfe-90df-4725-bae5-3ddede59c6bf-tls-assets\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:23.068912 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.068867 4925 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-web-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:23.068912 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.068879 4925 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7e184cfe-90df-4725-bae5-3ddede59c6bf-config-volume\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:23.070727 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.070699 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-kube-api-access-zvdmr" pod "32bd2a5c-0666-4ab4-a163-9f3b3783e6c7" (UID: "32bd2a5c-0666-4ab4-a163-9f3b3783e6c7"). InnerVolumeSpecName "kube-api-access-zvdmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:41:23.071093 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.071076 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-tls-certs" pod "32bd2a5c-0666-4ab4-a163-9f3b3783e6c7" (UID: "32bd2a5c-0666-4ab4-a163-9f3b3783e6c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:41:23.071247 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.071222 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-monitoring-tls-certs" pod "32bd2a5c-0666-4ab4-a163-9f3b3783e6c7" (UID: "32bd2a5c-0666-4ab4-a163-9f3b3783e6c7"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:41:23.072857 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.072820 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-data-science-collector-ta-client-cert" pod "32bd2a5c-0666-4ab4-a163-9f3b3783e6c7" (UID: "32bd2a5c-0666-4ab4-a163-9f3b3783e6c7"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:41:23.120820 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.120791 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5975a38d-4a75-4604-9db8-f130c3d0ec0a" path="/var/lib/kubelet/pods/5975a38d-4a75-4604-9db8-f130c3d0ec0a/volumes" Oct 30 03:41:23.121227 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.121214 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3a97a3f-60bb-43b3-942f-1ff03e25f94e" path="/var/lib/kubelet/pods/b3a97a3f-60bb-43b3-942f-1ff03e25f94e/volumes" Oct 30 03:41:23.170223 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.170198 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:23.170223 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.170224 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:23.170366 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.170233 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:23.170366 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.170242 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:23.170366 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.170252 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvdmr\" (UniqueName: \"kubernetes.io/projected/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7-kube-api-access-zvdmr\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:41:23.675013 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.674927 4925 generic.go:356] "Generic (PLEG): container finished" podID="7e184cfe-90df-4725-bae5-3ddede59c6bf" containerID="394d950675de19b9fb2c6af7aa0fdb6451b2b56068faea62428a80cbf2b94288" exitCode=0 Oct 30 03:41:23.675013 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.674976 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:23.675551 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.675020 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"7e184cfe-90df-4725-bae5-3ddede59c6bf","Type":"ContainerDied","Data":"394d950675de19b9fb2c6af7aa0fdb6451b2b56068faea62428a80cbf2b94288"} Oct 30 03:41:23.675551 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.675092 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"7e184cfe-90df-4725-bae5-3ddede59c6bf","Type":"ContainerDied","Data":"d0ed4c040a3aae6bce294ee307d9f37d0b5762de93f9fc2c66607b5c801cb419"} Oct 30 03:41:23.675551 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.675135 4925 scope.go:122] "RemoveContainer" containerID="4cfb77077bb0171a11e48cee12f8cf77d8c3207e6a5427471a442786eb8c6312" Oct 30 03:41:23.677631 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.677608 4925 generic.go:356] "Generic (PLEG): container finished" podID="32bd2a5c-0666-4ab4-a163-9f3b3783e6c7" containerID="c6ed74c79fdf423fb101edc6440ff4263768a633acc9be0fe0d0713e5672437b" exitCode=0 Oct 30 03:41:23.677840 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.677690 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:41:23.677840 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.677685 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7","Type":"ContainerDied","Data":"c6ed74c79fdf423fb101edc6440ff4263768a633acc9be0fe0d0713e5672437b"} Oct 30 03:41:23.677840 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.677816 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"32bd2a5c-0666-4ab4-a163-9f3b3783e6c7","Type":"ContainerDied","Data":"ec2b3882e26cd404571e1b2e616838666588b9c84b0576052471fd152c4cdf23"} Oct 30 03:41:23.677983 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.677967 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 03:41:23.685607 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.685590 4925 scope.go:122] "RemoveContainer" containerID="394d950675de19b9fb2c6af7aa0fdb6451b2b56068faea62428a80cbf2b94288" Oct 30 03:41:23.693765 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.693733 4925 scope.go:122] "RemoveContainer" containerID="29a3cd2f2953db55ef49f3643116bb76cb46d794790360b0628f17c3bbe4f21e" Oct 30 03:41:23.701543 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.701525 4925 scope.go:122] "RemoveContainer" containerID="4cfb77077bb0171a11e48cee12f8cf77d8c3207e6a5427471a442786eb8c6312" Oct 30 03:41:23.701822 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:41:23.701803 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cfb77077bb0171a11e48cee12f8cf77d8c3207e6a5427471a442786eb8c6312\": container with ID starting with 4cfb77077bb0171a11e48cee12f8cf77d8c3207e6a5427471a442786eb8c6312 not found: ID does not exist" containerID="4cfb77077bb0171a11e48cee12f8cf77d8c3207e6a5427471a442786eb8c6312" Oct 30 03:41:23.701910 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.701832 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cfb77077bb0171a11e48cee12f8cf77d8c3207e6a5427471a442786eb8c6312"} err="failed to get container status \"4cfb77077bb0171a11e48cee12f8cf77d8c3207e6a5427471a442786eb8c6312\": rpc error: code = NotFound desc = could not find container \"4cfb77077bb0171a11e48cee12f8cf77d8c3207e6a5427471a442786eb8c6312\": container with ID starting with 4cfb77077bb0171a11e48cee12f8cf77d8c3207e6a5427471a442786eb8c6312 not found: ID does not exist" Oct 30 03:41:23.701910 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.701853 4925 scope.go:122] "RemoveContainer" containerID="394d950675de19b9fb2c6af7aa0fdb6451b2b56068faea62428a80cbf2b94288" Oct 30 03:41:23.702131 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:41:23.702113 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"394d950675de19b9fb2c6af7aa0fdb6451b2b56068faea62428a80cbf2b94288\": container with ID starting with 394d950675de19b9fb2c6af7aa0fdb6451b2b56068faea62428a80cbf2b94288 not found: ID does not exist" containerID="394d950675de19b9fb2c6af7aa0fdb6451b2b56068faea62428a80cbf2b94288" Oct 30 03:41:23.702170 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.702136 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"394d950675de19b9fb2c6af7aa0fdb6451b2b56068faea62428a80cbf2b94288"} err="failed to get container status \"394d950675de19b9fb2c6af7aa0fdb6451b2b56068faea62428a80cbf2b94288\": rpc error: code = NotFound desc = could not find container \"394d950675de19b9fb2c6af7aa0fdb6451b2b56068faea62428a80cbf2b94288\": container with ID starting with 394d950675de19b9fb2c6af7aa0fdb6451b2b56068faea62428a80cbf2b94288 not found: ID does not exist" Oct 30 03:41:23.702170 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.702150 4925 scope.go:122] "RemoveContainer" containerID="29a3cd2f2953db55ef49f3643116bb76cb46d794790360b0628f17c3bbe4f21e" Oct 30 03:41:23.702403 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:41:23.702378 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a3cd2f2953db55ef49f3643116bb76cb46d794790360b0628f17c3bbe4f21e\": container with ID starting with 29a3cd2f2953db55ef49f3643116bb76cb46d794790360b0628f17c3bbe4f21e not found: ID does not exist" containerID="29a3cd2f2953db55ef49f3643116bb76cb46d794790360b0628f17c3bbe4f21e" Oct 30 03:41:23.702473 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.702402 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a3cd2f2953db55ef49f3643116bb76cb46d794790360b0628f17c3bbe4f21e"} err="failed to get container status \"29a3cd2f2953db55ef49f3643116bb76cb46d794790360b0628f17c3bbe4f21e\": rpc error: code = NotFound desc = could not find container \"29a3cd2f2953db55ef49f3643116bb76cb46d794790360b0628f17c3bbe4f21e\": container with ID starting with 29a3cd2f2953db55ef49f3643116bb76cb46d794790360b0628f17c3bbe4f21e not found: ID does not exist" Oct 30 03:41:23.702473 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.702424 4925 scope.go:122] "RemoveContainer" containerID="c6ed74c79fdf423fb101edc6440ff4263768a633acc9be0fe0d0713e5672437b" Oct 30 03:41:23.710249 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.710232 4925 scope.go:122] "RemoveContainer" containerID="c6ed74c79fdf423fb101edc6440ff4263768a633acc9be0fe0d0713e5672437b" Oct 30 03:41:23.710499 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:41:23.710481 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ed74c79fdf423fb101edc6440ff4263768a633acc9be0fe0d0713e5672437b\": container with ID starting with c6ed74c79fdf423fb101edc6440ff4263768a633acc9be0fe0d0713e5672437b not found: ID does not exist" containerID="c6ed74c79fdf423fb101edc6440ff4263768a633acc9be0fe0d0713e5672437b" Oct 30 03:41:23.710562 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.710506 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ed74c79fdf423fb101edc6440ff4263768a633acc9be0fe0d0713e5672437b"} err="failed to get container status \"c6ed74c79fdf423fb101edc6440ff4263768a633acc9be0fe0d0713e5672437b\": rpc error: code = NotFound desc = could not find container \"c6ed74c79fdf423fb101edc6440ff4263768a633acc9be0fe0d0713e5672437b\": container with ID starting with c6ed74c79fdf423fb101edc6440ff4263768a633acc9be0fe0d0713e5672437b not found: ID does not exist" Oct 30 03:41:23.729101 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.729082 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:41:23.733410 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.733389 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:41:23.772648 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.772628 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 03:41:23.776799 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.776780 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 03:41:23.815566 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.815544 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 03:41:23.823112 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:23.823093 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 03:41:25.125132 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:25.125090 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32bd2a5c-0666-4ab4-a163-9f3b3783e6c7" path="/var/lib/kubelet/pods/32bd2a5c-0666-4ab4-a163-9f3b3783e6c7/volumes" Oct 30 03:41:25.125524 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:25.125469 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e184cfe-90df-4725-bae5-3ddede59c6bf" path="/var/lib/kubelet/pods/7e184cfe-90df-4725-bae5-3ddede59c6bf/volumes" Oct 30 03:41:25.125955 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:25.125942 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0db7e1c-3383-430b-8b0f-64507e87db6c" path="/var/lib/kubelet/pods/b0db7e1c-3383-430b-8b0f-64507e87db6c/volumes" Oct 30 03:41:28.972920 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.972886 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7"] Oct 30 03:41:28.973316 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973230 4925 cpu_manager.go:414] "containerMap: removing container" podUID="b0db7e1c-3383-430b-8b0f-64507e87db6c" containerName="init-config-reloader" Oct 30 03:41:28.973316 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973242 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0db7e1c-3383-430b-8b0f-64507e87db6c" containerName="init-config-reloader" Oct 30 03:41:28.973316 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973255 4925 cpu_manager.go:414] "containerMap: removing container" podUID="b0db7e1c-3383-430b-8b0f-64507e87db6c" containerName="thanos-sidecar" Oct 30 03:41:28.973316 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973260 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0db7e1c-3383-430b-8b0f-64507e87db6c" containerName="thanos-sidecar" Oct 30 03:41:28.973316 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973271 4925 cpu_manager.go:414] "containerMap: removing container" podUID="5975a38d-4a75-4604-9db8-f130c3d0ec0a" containerName="kube-rbac-proxy" Oct 30 03:41:28.973316 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973277 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5975a38d-4a75-4604-9db8-f130c3d0ec0a" containerName="kube-rbac-proxy" Oct 30 03:41:28.973316 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973285 4925 cpu_manager.go:414] "containerMap: removing container" podUID="b0db7e1c-3383-430b-8b0f-64507e87db6c" containerName="prometheus" Oct 30 03:41:28.973316 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973289 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0db7e1c-3383-430b-8b0f-64507e87db6c" containerName="prometheus" Oct 30 03:41:28.973316 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973297 4925 cpu_manager.go:414] "containerMap: removing container" podUID="b0db7e1c-3383-430b-8b0f-64507e87db6c" containerName="config-reloader" Oct 30 03:41:28.973316 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973301 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0db7e1c-3383-430b-8b0f-64507e87db6c" containerName="config-reloader" Oct 30 03:41:28.973316 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973308 4925 cpu_manager.go:414] "containerMap: removing container" podUID="32bd2a5c-0666-4ab4-a163-9f3b3783e6c7" containerName="otc-container" Oct 30 03:41:28.973316 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973313 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32bd2a5c-0666-4ab4-a163-9f3b3783e6c7" containerName="otc-container" Oct 30 03:41:28.973316 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973319 4925 cpu_manager.go:414] "containerMap: removing container" podUID="7e184cfe-90df-4725-bae5-3ddede59c6bf" containerName="init-config-reloader" Oct 30 03:41:28.973696 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973324 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e184cfe-90df-4725-bae5-3ddede59c6bf" containerName="init-config-reloader" Oct 30 03:41:28.973696 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973332 4925 cpu_manager.go:414] "containerMap: removing container" podUID="7e184cfe-90df-4725-bae5-3ddede59c6bf" containerName="config-reloader" Oct 30 03:41:28.973696 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973336 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e184cfe-90df-4725-bae5-3ddede59c6bf" containerName="config-reloader" Oct 30 03:41:28.973696 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973343 4925 cpu_manager.go:414] "containerMap: removing container" podUID="b3a97a3f-60bb-43b3-942f-1ff03e25f94e" containerName="korrel8r" Oct 30 03:41:28.973696 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973348 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3a97a3f-60bb-43b3-942f-1ff03e25f94e" containerName="korrel8r" Oct 30 03:41:28.973696 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973362 4925 cpu_manager.go:414] "containerMap: removing container" podUID="7e184cfe-90df-4725-bae5-3ddede59c6bf" containerName="alertmanager" Oct 30 03:41:28.973696 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973366 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e184cfe-90df-4725-bae5-3ddede59c6bf" containerName="alertmanager" Oct 30 03:41:28.973696 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973413 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e184cfe-90df-4725-bae5-3ddede59c6bf" containerName="config-reloader" Oct 30 03:41:28.973696 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973420 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3a97a3f-60bb-43b3-942f-1ff03e25f94e" containerName="korrel8r" Oct 30 03:41:28.973696 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973428 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e184cfe-90df-4725-bae5-3ddede59c6bf" containerName="alertmanager" Oct 30 03:41:28.973696 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973436 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0db7e1c-3383-430b-8b0f-64507e87db6c" containerName="thanos-sidecar" Oct 30 03:41:28.973696 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973442 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5975a38d-4a75-4604-9db8-f130c3d0ec0a" containerName="kube-rbac-proxy" Oct 30 03:41:28.973696 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973446 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0db7e1c-3383-430b-8b0f-64507e87db6c" containerName="prometheus" Oct 30 03:41:28.973696 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973451 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0db7e1c-3383-430b-8b0f-64507e87db6c" containerName="config-reloader" Oct 30 03:41:28.973696 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.973457 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="32bd2a5c-0666-4ab4-a163-9f3b3783e6c7" containerName="otc-container" Oct 30 03:41:28.978285 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.978262 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:41:28.996517 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.996488 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 03:41:28.996678 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.996559 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 03:41:28.996766 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.996691 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 03:41:28.997092 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.997052 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 03:41:28.997092 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.997083 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 03:41:28.997246 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.997124 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 03:41:28.997365 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.997342 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-74pt9\"" Oct 30 03:41:28.998310 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:28.998291 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7"] Oct 30 03:41:29.014551 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.014511 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-clrg7\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:41:29.014551 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.014554 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9mlc\" (UniqueName: \"kubernetes.io/projected/3b89f1c9-b82d-4bf9-9923-844990919dde-kube-api-access-g9mlc\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-clrg7\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:41:29.014729 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.014579 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/3b89f1c9-b82d-4bf9-9923-844990919dde-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-clrg7\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:41:29.014729 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.014657 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-clrg7\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:41:29.014729 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.014688 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-clrg7\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:41:29.050397 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.050366 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw"] Oct 30 03:41:29.053804 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.053785 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" Oct 30 03:41:29.077568 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.077537 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw"] Oct 30 03:41:29.096464 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.096439 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-k7pwh\"" Oct 30 03:41:29.096976 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.096953 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 03:41:29.097071 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.096980 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 03:41:29.115514 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.115489 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-clrg7\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:41:29.115595 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.115529 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9mlc\" (UniqueName: \"kubernetes.io/projected/3b89f1c9-b82d-4bf9-9923-844990919dde-kube-api-access-g9mlc\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-clrg7\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:41:29.115595 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.115557 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4609d443-3e1a-4d52-92e5-939f0f90fce7-tls\") pod \"korrel8r-5cbcc89b8f-rl5pw\" (UID: \"4609d443-3e1a-4d52-92e5-939f0f90fce7\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" Oct 30 03:41:29.115595 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.115590 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb2bg\" (UniqueName: \"kubernetes.io/projected/4609d443-3e1a-4d52-92e5-939f0f90fce7-kube-api-access-fb2bg\") pod \"korrel8r-5cbcc89b8f-rl5pw\" (UID: \"4609d443-3e1a-4d52-92e5-939f0f90fce7\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" Oct 30 03:41:29.115775 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.115616 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/3b89f1c9-b82d-4bf9-9923-844990919dde-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-clrg7\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:41:29.115775 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.115690 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-clrg7\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:41:29.115775 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.115719 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-clrg7\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:41:29.115934 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.115842 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4609d443-3e1a-4d52-92e5-939f0f90fce7-config\") pod \"korrel8r-5cbcc89b8f-rl5pw\" (UID: \"4609d443-3e1a-4d52-92e5-939f0f90fce7\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" Oct 30 03:41:29.115990 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:41:29.115945 4925 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 03:41:29.116043 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:41:29.116019 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-prometheus-client-cert podName:3b89f1c9-b82d-4bf9-9923-844990919dde nodeName:}" failed. No retries permitted until 2026-10-30 03:41:29.615998467 +0000 UTC m=+3650.629276566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-clrg7" (UID: "3b89f1c9-b82d-4bf9-9923-844990919dde") : secret "prometheus-operated-tls" not found Oct 30 03:41:29.116332 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.116310 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/3b89f1c9-b82d-4bf9-9923-844990919dde-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-clrg7\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:41:29.118087 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.118062 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-clrg7\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:41:29.118221 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.118139 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-clrg7\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:41:29.141515 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.141497 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9mlc\" (UniqueName: \"kubernetes.io/projected/3b89f1c9-b82d-4bf9-9923-844990919dde-kube-api-access-g9mlc\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-clrg7\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:41:29.216683 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.216657 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4609d443-3e1a-4d52-92e5-939f0f90fce7-config\") pod \"korrel8r-5cbcc89b8f-rl5pw\" (UID: \"4609d443-3e1a-4d52-92e5-939f0f90fce7\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" Oct 30 03:41:29.216806 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.216692 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4609d443-3e1a-4d52-92e5-939f0f90fce7-tls\") pod \"korrel8r-5cbcc89b8f-rl5pw\" (UID: \"4609d443-3e1a-4d52-92e5-939f0f90fce7\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" Oct 30 03:41:29.216806 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.216712 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb2bg\" (UniqueName: \"kubernetes.io/projected/4609d443-3e1a-4d52-92e5-939f0f90fce7-kube-api-access-fb2bg\") pod \"korrel8r-5cbcc89b8f-rl5pw\" (UID: \"4609d443-3e1a-4d52-92e5-939f0f90fce7\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" Oct 30 03:41:29.216892 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:41:29.216824 4925 secret.go:189] Couldn't get secret redhat-ods-monitoring/korrel8r-tls: secret "korrel8r-tls" not found Oct 30 03:41:29.216892 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:41:29.216887 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4609d443-3e1a-4d52-92e5-939f0f90fce7-tls podName:4609d443-3e1a-4d52-92e5-939f0f90fce7 nodeName:}" failed. No retries permitted until 2026-10-30 03:41:29.716870405 +0000 UTC m=+3650.730148489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/4609d443-3e1a-4d52-92e5-939f0f90fce7-tls") pod "korrel8r-5cbcc89b8f-rl5pw" (UID: "4609d443-3e1a-4d52-92e5-939f0f90fce7") : secret "korrel8r-tls" not found Oct 30 03:41:29.217477 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.217460 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4609d443-3e1a-4d52-92e5-939f0f90fce7-config\") pod \"korrel8r-5cbcc89b8f-rl5pw\" (UID: \"4609d443-3e1a-4d52-92e5-939f0f90fce7\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" Oct 30 03:41:29.240382 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.240333 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb2bg\" (UniqueName: \"kubernetes.io/projected/4609d443-3e1a-4d52-92e5-939f0f90fce7-kube-api-access-fb2bg\") pod \"korrel8r-5cbcc89b8f-rl5pw\" (UID: \"4609d443-3e1a-4d52-92e5-939f0f90fce7\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" Oct 30 03:41:29.619745 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.619708 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-clrg7\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:41:29.621991 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.621965 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-clrg7\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:41:29.721183 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.721129 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4609d443-3e1a-4d52-92e5-939f0f90fce7-tls\") pod \"korrel8r-5cbcc89b8f-rl5pw\" (UID: \"4609d443-3e1a-4d52-92e5-939f0f90fce7\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" Oct 30 03:41:29.723546 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.723528 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4609d443-3e1a-4d52-92e5-939f0f90fce7-tls\") pod \"korrel8r-5cbcc89b8f-rl5pw\" (UID: \"4609d443-3e1a-4d52-92e5-939f0f90fce7\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" Oct 30 03:41:29.891644 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.891552 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:41:29.967298 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:29.967269 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" Oct 30 03:41:30.191455 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.191420 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7"] Oct 30 03:41:30.192531 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:41:30.192504 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b89f1c9_b82d_4bf9_9923_844990919dde.slice/crio-7337a10cf238290c1af071dbdcd5313db181836d75e01a6b54ee9cf6d877e201 WatchSource:0}: Error finding container 7337a10cf238290c1af071dbdcd5313db181836d75e01a6b54ee9cf6d877e201: Status 404 returned error can't find the container with id 7337a10cf238290c1af071dbdcd5313db181836d75e01a6b54ee9cf6d877e201 Oct 30 03:41:30.266015 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.265994 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw"] Oct 30 03:41:30.267302 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:41:30.267279 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4609d443_3e1a_4d52_92e5_939f0f90fce7.slice/crio-56db9d1e5d8296030c0813797551e8913d72be259464eb6aeffc068743a5d72a WatchSource:0}: Error finding container 56db9d1e5d8296030c0813797551e8913d72be259464eb6aeffc068743a5d72a: Status 404 returned error can't find the container with id 56db9d1e5d8296030c0813797551e8913d72be259464eb6aeffc068743a5d72a Oct 30 03:41:30.580675 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.580641 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:41:30.584670 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.584649 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:41:30.600000 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.599978 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 03:41:30.600141 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.599997 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:41:30.600141 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.600045 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 03:41:30.600141 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.600113 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 03:41:30.609840 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.609806 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:41:30.627701 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.627671 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9dccc70e-aa27-484d-924c-97335acd7b36-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9dccc70e-aa27-484d-924c-97335acd7b36\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:41:30.627863 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.627713 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9dccc70e-aa27-484d-924c-97335acd7b36\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:41:30.627935 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.627854 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"9dccc70e-aa27-484d-924c-97335acd7b36\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:41:30.627935 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.627901 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54869\" (UniqueName: \"kubernetes.io/projected/9dccc70e-aa27-484d-924c-97335acd7b36-kube-api-access-54869\") pod \"data-science-collector-collector-0\" (UID: \"9dccc70e-aa27-484d-924c-97335acd7b36\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:41:30.628043 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.627932 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9dccc70e-aa27-484d-924c-97335acd7b36\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:41:30.714716 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.714672 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" event={"ID":"3b89f1c9-b82d-4bf9-9923-844990919dde","Type":"ContainerStarted","Data":"7337a10cf238290c1af071dbdcd5313db181836d75e01a6b54ee9cf6d877e201"} Oct 30 03:41:30.716553 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.716521 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" event={"ID":"4609d443-3e1a-4d52-92e5-939f0f90fce7","Type":"ContainerStarted","Data":"d7981097195b342a37b9e857cfb7711ca60f33b99debe24e59b4703d51b830fa"} Oct 30 03:41:30.716663 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.716575 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" event={"ID":"4609d443-3e1a-4d52-92e5-939f0f90fce7","Type":"ContainerStarted","Data":"56db9d1e5d8296030c0813797551e8913d72be259464eb6aeffc068743a5d72a"} Oct 30 03:41:30.716739 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.716674 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" Oct 30 03:41:30.729037 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.729012 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9dccc70e-aa27-484d-924c-97335acd7b36-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9dccc70e-aa27-484d-924c-97335acd7b36\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:41:30.729187 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.729042 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9dccc70e-aa27-484d-924c-97335acd7b36\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:41:30.729187 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.729079 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"9dccc70e-aa27-484d-924c-97335acd7b36\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:41:30.729187 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.729104 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54869\" (UniqueName: \"kubernetes.io/projected/9dccc70e-aa27-484d-924c-97335acd7b36-kube-api-access-54869\") pod \"data-science-collector-collector-0\" (UID: \"9dccc70e-aa27-484d-924c-97335acd7b36\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:41:30.729187 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.729133 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9dccc70e-aa27-484d-924c-97335acd7b36\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:41:30.729916 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.729888 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9dccc70e-aa27-484d-924c-97335acd7b36-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9dccc70e-aa27-484d-924c-97335acd7b36\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:41:30.731988 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.731965 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9dccc70e-aa27-484d-924c-97335acd7b36\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:41:30.732215 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.732198 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"9dccc70e-aa27-484d-924c-97335acd7b36\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:41:30.733015 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.732995 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9dccc70e-aa27-484d-924c-97335acd7b36\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:41:30.756174 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.756153 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54869\" (UniqueName: \"kubernetes.io/projected/9dccc70e-aa27-484d-924c-97335acd7b36-kube-api-access-54869\") pod \"data-science-collector-collector-0\" (UID: \"9dccc70e-aa27-484d-924c-97335acd7b36\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:41:30.759796 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.759725 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" podStartSLOduration=1.759709762 podStartE2EDuration="1.759709762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:30.75763155 +0000 UTC m=+3651.770909654" watchObservedRunningTime="2026-10-30 03:41:30.759709762 +0000 UTC m=+3651.772987865" Oct 30 03:41:30.899449 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:30.899424 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:41:31.213458 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:31.213432 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:41:31.215291 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:41:31.215264 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dccc70e_aa27_484d_924c_97335acd7b36.slice/crio-b5919bf88a86c7b16920074be7d9e0fde22ad254ffdb193421dac5e0b367677b WatchSource:0}: Error finding container b5919bf88a86c7b16920074be7d9e0fde22ad254ffdb193421dac5e0b367677b: Status 404 returned error can't find the container with id b5919bf88a86c7b16920074be7d9e0fde22ad254ffdb193421dac5e0b367677b Oct 30 03:41:31.218411 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:31.218380 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:41:31.218504 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:31.218440 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:41:31.722792 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:31.722733 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" event={"ID":"3b89f1c9-b82d-4bf9-9923-844990919dde","Type":"ContainerStarted","Data":"2877365544e74e090c756323c86e81d65fe7285498fcfd01dd55d2c7693a8955"} Oct 30 03:41:31.722954 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:31.722923 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:41:31.724479 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:31.724452 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9dccc70e-aa27-484d-924c-97335acd7b36","Type":"ContainerStarted","Data":"1ff01f18c4cfee1161abbbdbe434daddc8b94f7652874f1d7ed9d25f8eaf056e"} Oct 30 03:41:31.724589 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:31.724496 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9dccc70e-aa27-484d-924c-97335acd7b36","Type":"ContainerStarted","Data":"b5919bf88a86c7b16920074be7d9e0fde22ad254ffdb193421dac5e0b367677b"} Oct 30 03:41:31.764813 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:31.764773 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" podStartSLOduration=3.764740776 podStartE2EDuration="3.764740776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:31.764050058 +0000 UTC m=+3652.777328160" watchObservedRunningTime="2026-10-30 03:41:31.764740776 +0000 UTC m=+3652.778018877" Oct 30 03:41:31.807526 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:31.807482 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.807469546 podStartE2EDuration="1.807469546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:31.80601898 +0000 UTC m=+3652.819297084" watchObservedRunningTime="2026-10-30 03:41:31.807469546 +0000 UTC m=+3652.820747648" Oct 30 03:41:34.121593 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.121551 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 03:41:34.125537 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.125511 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.145551 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.145529 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 03:41:34.145663 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.145615 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-v4hz2\"" Oct 30 03:41:34.152957 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.152914 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 03:41:34.153121 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.152919 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 03:41:34.153295 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.152961 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 03:41:34.157198 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.157174 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 03:41:34.158698 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.158672 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.158832 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.158726 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/2d166da7-36f8-45ab-bdc9-2eb4347094f2-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.158951 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.158927 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.159114 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.159088 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4pwl\" (UniqueName: \"kubernetes.io/projected/2d166da7-36f8-45ab-bdc9-2eb4347094f2-kube-api-access-g4pwl\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.159229 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.159155 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d166da7-36f8-45ab-bdc9-2eb4347094f2-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.159229 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.159192 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d166da7-36f8-45ab-bdc9-2eb4347094f2-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.159338 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.159232 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.259795 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.259737 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.259795 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.259795 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.260045 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.259823 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/2d166da7-36f8-45ab-bdc9-2eb4347094f2-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.260045 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.259880 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.260045 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.260013 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4pwl\" (UniqueName: \"kubernetes.io/projected/2d166da7-36f8-45ab-bdc9-2eb4347094f2-kube-api-access-g4pwl\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.260213 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.260049 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d166da7-36f8-45ab-bdc9-2eb4347094f2-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.260213 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.260079 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d166da7-36f8-45ab-bdc9-2eb4347094f2-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.260367 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.260335 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/2d166da7-36f8-45ab-bdc9-2eb4347094f2-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.262300 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.262265 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d166da7-36f8-45ab-bdc9-2eb4347094f2-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.262435 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.262338 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.262435 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.262409 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.262624 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.262602 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d166da7-36f8-45ab-bdc9-2eb4347094f2-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.262710 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.262691 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.279180 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.279150 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 03:41:34.283814 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.283796 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.286523 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.286501 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4pwl\" (UniqueName: \"kubernetes.io/projected/2d166da7-36f8-45ab-bdc9-2eb4347094f2-kube-api-access-g4pwl\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.308833 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.308809 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 03:41:34.341402 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.341373 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 03:41:34.341797 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.341776 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 03:41:34.341866 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.341813 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 03:41:34.341927 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.341884 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 03:41:34.341927 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.341884 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 03:41:34.343308 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.343293 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-hmf28\"" Oct 30 03:41:34.360852 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.360829 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.360943 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.360867 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.360943 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.360888 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6436e346-6d25-4373-98f0-7b2eaedaf457-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.360943 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.360929 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.361056 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.361004 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.361056 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.361037 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.361056 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.361053 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.361192 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.361074 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.361192 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.361163 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.361259 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.361191 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvjs9\" (UniqueName: \"kubernetes.io/projected/6436e346-6d25-4373-98f0-7b2eaedaf457-kube-api-access-wvjs9\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.361259 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.361216 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6436e346-6d25-4373-98f0-7b2eaedaf457-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.361259 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.361230 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.378514 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.378457 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 03:41:34.378514 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.378462 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 03:41:34.378619 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.378590 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 03:41:34.443242 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.443220 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:34.462462 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.462438 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.462568 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.462466 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.462568 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.462483 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.462568 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.462503 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.462736 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.462586 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvjs9\" (UniqueName: \"kubernetes.io/projected/6436e346-6d25-4373-98f0-7b2eaedaf457-kube-api-access-wvjs9\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.462736 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.462615 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6436e346-6d25-4373-98f0-7b2eaedaf457-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.462736 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.462632 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.463254 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.462929 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.463254 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.462988 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.463254 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.463023 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6436e346-6d25-4373-98f0-7b2eaedaf457-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.463254 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.463048 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.463254 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.463115 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.463254 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.463141 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.463254 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.463230 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.463988 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.463960 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.466671 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.466651 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.468499 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.468275 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6436e346-6d25-4373-98f0-7b2eaedaf457-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.468499 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.468430 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.468499 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.468450 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.468499 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.468455 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6436e346-6d25-4373-98f0-7b2eaedaf457-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.468739 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.468570 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.468933 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.468911 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.469634 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.469612 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.486289 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.486268 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvjs9\" (UniqueName: \"kubernetes.io/projected/6436e346-6d25-4373-98f0-7b2eaedaf457-kube-api-access-wvjs9\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:34.763919 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:34.763895 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 03:41:34.765091 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:41:34.765065 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d166da7_36f8_45ab_bdc9_2eb4347094f2.slice/crio-5daaa2fbfc181e866b82de5df35fb4260f72c709bc42ddbfe05a02644b107ba7 WatchSource:0}: Error finding container 5daaa2fbfc181e866b82de5df35fb4260f72c709bc42ddbfe05a02644b107ba7: Status 404 returned error can't find the container with id 5daaa2fbfc181e866b82de5df35fb4260f72c709bc42ddbfe05a02644b107ba7 Oct 30 03:41:35.746412 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:35.746379 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2d166da7-36f8-45ab-bdc9-2eb4347094f2","Type":"ContainerStarted","Data":"73b1de764e0b3602bc7e247249fc507ebac8a4f274148a9e8e40876dcfd2c694"} Oct 30 03:41:35.746789 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:35.746441 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2d166da7-36f8-45ab-bdc9-2eb4347094f2","Type":"ContainerStarted","Data":"5daaa2fbfc181e866b82de5df35fb4260f72c709bc42ddbfe05a02644b107ba7"} Oct 30 03:41:37.732976 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:37.732903 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:41:38.596783 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:38.596682 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:38.626081 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:38.626051 4925 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:38.697481 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:38.697454 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:38.697624 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:38.697601 4925 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:38.712428 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:38.712404 4925 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") DevicePath \"csi-ec2ccb9dcc2629e5a59ea91641dbb6b15d80e87697977b7c41508ff0003b2f36\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:38.763986 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:38.763960 4925 generic.go:356] "Generic (PLEG): container finished" podID="2d166da7-36f8-45ab-bdc9-2eb4347094f2" containerID="73b1de764e0b3602bc7e247249fc507ebac8a4f274148a9e8e40876dcfd2c694" exitCode=0 Oct 30 03:41:38.764278 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:38.764033 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2d166da7-36f8-45ab-bdc9-2eb4347094f2","Type":"ContainerDied","Data":"73b1de764e0b3602bc7e247249fc507ebac8a4f274148a9e8e40876dcfd2c694"} Oct 30 03:41:38.815713 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:38.815688 4925 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a535375d22e374ba333968465471c7604a06f9d95a513910fe4bb083b629e986/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:38.842451 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:38.842427 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:39.126282 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:39.126215 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-hmf28\"" Oct 30 03:41:39.131304 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:39.131280 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:39.465933 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:39.465905 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 03:41:39.466841 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:41:39.466814 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6436e346_6d25_4373_98f0_7b2eaedaf457.slice/crio-6085eaa1052aa7a8695353df34f4a0184cc0011f5215dbac11d3f2fc1c69dbcb WatchSource:0}: Error finding container 6085eaa1052aa7a8695353df34f4a0184cc0011f5215dbac11d3f2fc1c69dbcb: Status 404 returned error can't find the container with id 6085eaa1052aa7a8695353df34f4a0184cc0011f5215dbac11d3f2fc1c69dbcb Oct 30 03:41:39.575079 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:39.575056 4925 scope.go:122] "RemoveContainer" containerID="727fc5c57caaecfb30f592d983ef54da71682e326dc53f642a87facbe46a1f59" Oct 30 03:41:39.583033 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:39.583011 4925 scope.go:122] "RemoveContainer" containerID="914a92b7f570547e2bc138c83c57dea930f0b56777754ee7fa94705333118437" Oct 30 03:41:39.591006 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:39.590991 4925 scope.go:122] "RemoveContainer" containerID="f43a319e2d26aef1419e890f952e3e192acfef64e1b5e277e10fb2fe254fd018" Oct 30 03:41:39.598420 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:39.598403 4925 scope.go:122] "RemoveContainer" containerID="45099e76f59a007d1598fd840a404d70b924c4805af6cafc58c467a22a26ddf8" Oct 30 03:41:39.771229 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:39.771149 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2d166da7-36f8-45ab-bdc9-2eb4347094f2","Type":"ContainerStarted","Data":"cd66980e9ac45eb8f8a14e29aa9bc83be7fc0e43d63487aa763cf43ab642914f"} Oct 30 03:41:39.771229 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:39.771215 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2d166da7-36f8-45ab-bdc9-2eb4347094f2","Type":"ContainerStarted","Data":"8dd05a7901c22fc9c847dd9d9779d924e9bb1e3e870923bf04a79d40dd3403ac"} Oct 30 03:41:39.771640 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:39.771406 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:39.772302 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:39.772282 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"6436e346-6d25-4373-98f0-7b2eaedaf457","Type":"ContainerStarted","Data":"6085eaa1052aa7a8695353df34f4a0184cc0011f5215dbac11d3f2fc1c69dbcb"} Oct 30 03:41:39.831255 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:39.831209 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=2.545515409 podStartE2EDuration="5.831197651s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.285682242s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:39.818299138 +0000 UTC m=+3660.831577279" watchObservedRunningTime="2026-10-30 03:41:39.831197651 +0000 UTC m=+3660.844475753" Oct 30 03:41:40.778816 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:40.778775 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"6436e346-6d25-4373-98f0-7b2eaedaf457","Type":"ContainerStarted","Data":"d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11"} Oct 30 03:41:41.726970 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:41.726938 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" Oct 30 03:41:43.794558 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:43.794526 4925 generic.go:356] "Generic (PLEG): container finished" podID="6436e346-6d25-4373-98f0-7b2eaedaf457" containerID="d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11" exitCode=0 Oct 30 03:41:43.794955 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:43.794573 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"6436e346-6d25-4373-98f0-7b2eaedaf457","Type":"ContainerDied","Data":"d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11"} Oct 30 03:41:44.802242 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:44.802149 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"6436e346-6d25-4373-98f0-7b2eaedaf457","Type":"ContainerStarted","Data":"ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4"} Oct 30 03:41:44.802242 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:44.802226 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"6436e346-6d25-4373-98f0-7b2eaedaf457","Type":"ContainerStarted","Data":"5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea"} Oct 30 03:41:44.802790 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:44.802264 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"6436e346-6d25-4373-98f0-7b2eaedaf457","Type":"ContainerStarted","Data":"96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e"} Oct 30 03:41:44.869746 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:44.869692 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=7.584383356 podStartE2EDuration="10.86967842s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.285295064s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:44.865582323 +0000 UTC m=+3665.878860426" watchObservedRunningTime="2026-10-30 03:41:44.86967842 +0000 UTC m=+3665.882956550" Oct 30 03:41:45.780507 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:45.780474 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:41:48.161760 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:41:48.161713 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:41:49.131844 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:49.131814 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:54.132313 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:54.132277 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:54.137854 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:54.137831 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:41:54.852851 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:41:54.852825 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:42:03.215886 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.215853 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:42:03.216375 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.216092 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="9dccc70e-aa27-484d-924c-97335acd7b36" containerName="otc-container" containerID="cri-o://1ff01f18c4cfee1161abbbdbe434daddc8b94f7652874f1d7ed9d25f8eaf056e" gracePeriod=30 Oct 30 03:42:03.460457 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.460432 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:42:03.514570 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.513722 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-tls-certs\" (UniqueName: \"kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-tls-certs\") pod \"9dccc70e-aa27-484d-924c-97335acd7b36\" (UID: \"9dccc70e-aa27-484d-924c-97335acd7b36\") " Oct 30 03:42:03.514570 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.513824 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dccc70e-aa27-484d-924c-97335acd7b36-kube-api-access-54869\" (UniqueName: \"kubernetes.io/projected/9dccc70e-aa27-484d-924c-97335acd7b36-kube-api-access-54869\") pod \"9dccc70e-aa27-484d-924c-97335acd7b36\" (UID: \"9dccc70e-aa27-484d-924c-97335acd7b36\") " Oct 30 03:42:03.514570 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.513875 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-data-science-collector-ta-client-cert\") pod \"9dccc70e-aa27-484d-924c-97335acd7b36\" (UID: \"9dccc70e-aa27-484d-924c-97335acd7b36\") " Oct 30 03:42:03.514570 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.513912 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-monitoring-tls-certs\") pod \"9dccc70e-aa27-484d-924c-97335acd7b36\" (UID: \"9dccc70e-aa27-484d-924c-97335acd7b36\") " Oct 30 03:42:03.514570 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.513987 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dccc70e-aa27-484d-924c-97335acd7b36-otc-internal\" (UniqueName: \"kubernetes.io/configmap/9dccc70e-aa27-484d-924c-97335acd7b36-otc-internal\") pod \"9dccc70e-aa27-484d-924c-97335acd7b36\" (UID: \"9dccc70e-aa27-484d-924c-97335acd7b36\") " Oct 30 03:42:03.515165 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.515093 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dccc70e-aa27-484d-924c-97335acd7b36-otc-internal" pod "9dccc70e-aa27-484d-924c-97335acd7b36" (UID: "9dccc70e-aa27-484d-924c-97335acd7b36"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:42:03.517617 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.517591 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-data-science-collector-ta-client-cert" pod "9dccc70e-aa27-484d-924c-97335acd7b36" (UID: "9dccc70e-aa27-484d-924c-97335acd7b36"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:42:03.517714 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.517616 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-monitoring-tls-certs" pod "9dccc70e-aa27-484d-924c-97335acd7b36" (UID: "9dccc70e-aa27-484d-924c-97335acd7b36"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:42:03.517795 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.517716 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-tls-certs" pod "9dccc70e-aa27-484d-924c-97335acd7b36" (UID: "9dccc70e-aa27-484d-924c-97335acd7b36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:42:03.519311 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.519290 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dccc70e-aa27-484d-924c-97335acd7b36-kube-api-access-54869" pod "9dccc70e-aa27-484d-924c-97335acd7b36" (UID: "9dccc70e-aa27-484d-924c-97335acd7b36"). InnerVolumeSpecName "kube-api-access-54869". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:42:03.614718 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.614690 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:42:03.614886 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.614732 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:42:03.614886 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.614743 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9dccc70e-aa27-484d-924c-97335acd7b36-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:42:03.614886 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.614777 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dccc70e-aa27-484d-924c-97335acd7b36-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:42:03.614886 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.614784 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54869\" (UniqueName: \"kubernetes.io/projected/9dccc70e-aa27-484d-924c-97335acd7b36-kube-api-access-54869\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:42:03.893092 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.893061 4925 generic.go:356] "Generic (PLEG): container finished" podID="9dccc70e-aa27-484d-924c-97335acd7b36" containerID="1ff01f18c4cfee1161abbbdbe434daddc8b94f7652874f1d7ed9d25f8eaf056e" exitCode=0 Oct 30 03:42:03.893284 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.893128 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:42:03.893284 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.893149 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9dccc70e-aa27-484d-924c-97335acd7b36","Type":"ContainerDied","Data":"1ff01f18c4cfee1161abbbdbe434daddc8b94f7652874f1d7ed9d25f8eaf056e"} Oct 30 03:42:03.893284 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.893250 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9dccc70e-aa27-484d-924c-97335acd7b36","Type":"ContainerDied","Data":"b5919bf88a86c7b16920074be7d9e0fde22ad254ffdb193421dac5e0b367677b"} Oct 30 03:42:03.893284 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.893274 4925 scope.go:122] "RemoveContainer" containerID="1ff01f18c4cfee1161abbbdbe434daddc8b94f7652874f1d7ed9d25f8eaf056e" Oct 30 03:42:03.902498 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.902480 4925 scope.go:122] "RemoveContainer" containerID="1ff01f18c4cfee1161abbbdbe434daddc8b94f7652874f1d7ed9d25f8eaf056e" Oct 30 03:42:03.902741 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:42:03.902723 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff01f18c4cfee1161abbbdbe434daddc8b94f7652874f1d7ed9d25f8eaf056e\": container with ID starting with 1ff01f18c4cfee1161abbbdbe434daddc8b94f7652874f1d7ed9d25f8eaf056e not found: ID does not exist" containerID="1ff01f18c4cfee1161abbbdbe434daddc8b94f7652874f1d7ed9d25f8eaf056e" Oct 30 03:42:03.902844 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.902774 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff01f18c4cfee1161abbbdbe434daddc8b94f7652874f1d7ed9d25f8eaf056e"} err="failed to get container status \"1ff01f18c4cfee1161abbbdbe434daddc8b94f7652874f1d7ed9d25f8eaf056e\": rpc error: code = NotFound desc = could not find container \"1ff01f18c4cfee1161abbbdbe434daddc8b94f7652874f1d7ed9d25f8eaf056e\": container with ID starting with 1ff01f18c4cfee1161abbbdbe434daddc8b94f7652874f1d7ed9d25f8eaf056e not found: ID does not exist" Oct 30 03:42:03.946466 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.946438 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:42:03.958964 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.958936 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:42:03.990213 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.990187 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:42:03.990570 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.990558 4925 cpu_manager.go:414] "containerMap: removing container" podUID="9dccc70e-aa27-484d-924c-97335acd7b36" containerName="otc-container" Oct 30 03:42:03.990618 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.990571 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dccc70e-aa27-484d-924c-97335acd7b36" containerName="otc-container" Oct 30 03:42:03.990658 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.990634 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dccc70e-aa27-484d-924c-97335acd7b36" containerName="otc-container" Oct 30 03:42:03.995109 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:03.995090 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:42:04.010652 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.010633 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:42:04.010786 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.010691 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 03:42:04.010786 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.010695 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 03:42:04.010786 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.010739 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 03:42:04.010786 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.010770 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\"" Oct 30 03:42:04.015723 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.015701 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:42:04.020779 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.019068 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"671da0dd-3d47-4bbc-9b78-491ba81d3007\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:42:04.020779 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.019192 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"671da0dd-3d47-4bbc-9b78-491ba81d3007\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:42:04.020779 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.019271 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"671da0dd-3d47-4bbc-9b78-491ba81d3007\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:42:04.020779 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.019305 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/671da0dd-3d47-4bbc-9b78-491ba81d3007-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"671da0dd-3d47-4bbc-9b78-491ba81d3007\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:42:04.020779 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.019327 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh8j4\" (UniqueName: \"kubernetes.io/projected/671da0dd-3d47-4bbc-9b78-491ba81d3007-kube-api-access-bh8j4\") pod \"data-science-collector-collector-0\" (UID: \"671da0dd-3d47-4bbc-9b78-491ba81d3007\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:42:04.119921 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.119887 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/671da0dd-3d47-4bbc-9b78-491ba81d3007-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"671da0dd-3d47-4bbc-9b78-491ba81d3007\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:42:04.119921 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.119920 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh8j4\" (UniqueName: \"kubernetes.io/projected/671da0dd-3d47-4bbc-9b78-491ba81d3007-kube-api-access-bh8j4\") pod \"data-science-collector-collector-0\" (UID: \"671da0dd-3d47-4bbc-9b78-491ba81d3007\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:42:04.120111 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.120050 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"671da0dd-3d47-4bbc-9b78-491ba81d3007\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:42:04.120111 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.120106 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"671da0dd-3d47-4bbc-9b78-491ba81d3007\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:42:04.120176 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.120142 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"671da0dd-3d47-4bbc-9b78-491ba81d3007\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:42:04.120625 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.120598 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/671da0dd-3d47-4bbc-9b78-491ba81d3007-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"671da0dd-3d47-4bbc-9b78-491ba81d3007\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:42:04.122563 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.122540 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"671da0dd-3d47-4bbc-9b78-491ba81d3007\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:42:04.122701 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.122608 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"671da0dd-3d47-4bbc-9b78-491ba81d3007\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:42:04.122701 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.122620 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"671da0dd-3d47-4bbc-9b78-491ba81d3007\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:42:04.147395 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.147342 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh8j4\" (UniqueName: \"kubernetes.io/projected/671da0dd-3d47-4bbc-9b78-491ba81d3007-kube-api-access-bh8j4\") pod \"data-science-collector-collector-0\" (UID: \"671da0dd-3d47-4bbc-9b78-491ba81d3007\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:42:04.307381 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.307348 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:42:04.607316 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.607292 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:42:04.609195 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:42:04.609148 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod671da0dd_3d47_4bbc_9b78_491ba81d3007.slice/crio-e9a1661e6f7a0857343529de38b5018824ea539cf9fd7b7d1008bfcc241285c8 WatchSource:0}: Error finding container e9a1661e6f7a0857343529de38b5018824ea539cf9fd7b7d1008bfcc241285c8: Status 404 returned error can't find the container with id e9a1661e6f7a0857343529de38b5018824ea539cf9fd7b7d1008bfcc241285c8 Oct 30 03:42:04.611351 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.611320 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:42:04.611450 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.611372 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:42:04.900532 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.900441 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"671da0dd-3d47-4bbc-9b78-491ba81d3007","Type":"ContainerStarted","Data":"1567c624fa947dc3babb8e8791592a41bddf2800fa92f9962c9018f739a9a23f"} Oct 30 03:42:04.900532 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.900503 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"671da0dd-3d47-4bbc-9b78-491ba81d3007","Type":"ContainerStarted","Data":"e9a1661e6f7a0857343529de38b5018824ea539cf9fd7b7d1008bfcc241285c8"} Oct 30 03:42:04.942671 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:04.942626 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.9426141540000001 podStartE2EDuration="1.942614154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:42:04.941953277 +0000 UTC m=+3685.955231382" watchObservedRunningTime="2026-10-30 03:42:04.942614154 +0000 UTC m=+3685.955892255" Oct 30 03:42:05.119909 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:42:05.119869 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dccc70e-aa27-484d-924c-97335acd7b36" path="/var/lib/kubelet/pods/9dccc70e-aa27-484d-924c-97335acd7b36/volumes" Oct 30 03:42:18.161046 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:42:18.161009 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:42:48.161620 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:42:48.161574 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:43:18.160343 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:43:18.160296 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:43:27.019984 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:27.019946 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv"] Oct 30 03:43:27.023576 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:27.023559 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" Oct 30 03:43:27.043764 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:27.043729 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv"] Oct 30 03:43:27.109602 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:27.109576 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqxwv\" (UniqueName: \"kubernetes.io/projected/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-kube-api-access-dqxwv\") pod \"korrel8r-66d6f7f696-grvkv\" (UID: \"cd1da09d-c287-4103-8a7a-0ebfd670e4d9\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" Oct 30 03:43:27.109783 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:27.109613 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-config\") pod \"korrel8r-66d6f7f696-grvkv\" (UID: \"cd1da09d-c287-4103-8a7a-0ebfd670e4d9\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" Oct 30 03:43:27.109783 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:27.109640 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-tls\") pod \"korrel8r-66d6f7f696-grvkv\" (UID: \"cd1da09d-c287-4103-8a7a-0ebfd670e4d9\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" Oct 30 03:43:27.210691 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:27.210659 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-tls\") pod \"korrel8r-66d6f7f696-grvkv\" (UID: \"cd1da09d-c287-4103-8a7a-0ebfd670e4d9\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" Oct 30 03:43:27.210891 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:27.210866 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqxwv\" (UniqueName: \"kubernetes.io/projected/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-kube-api-access-dqxwv\") pod \"korrel8r-66d6f7f696-grvkv\" (UID: \"cd1da09d-c287-4103-8a7a-0ebfd670e4d9\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" Oct 30 03:43:27.211085 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:27.211054 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-config\") pod \"korrel8r-66d6f7f696-grvkv\" (UID: \"cd1da09d-c287-4103-8a7a-0ebfd670e4d9\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" Oct 30 03:43:27.211694 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:27.211677 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-config\") pod \"korrel8r-66d6f7f696-grvkv\" (UID: \"cd1da09d-c287-4103-8a7a-0ebfd670e4d9\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" Oct 30 03:43:27.213106 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:27.213089 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-tls\") pod \"korrel8r-66d6f7f696-grvkv\" (UID: \"cd1da09d-c287-4103-8a7a-0ebfd670e4d9\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" Oct 30 03:43:27.240691 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:27.240666 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqxwv\" (UniqueName: \"kubernetes.io/projected/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-kube-api-access-dqxwv\") pod \"korrel8r-66d6f7f696-grvkv\" (UID: \"cd1da09d-c287-4103-8a7a-0ebfd670e4d9\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" Oct 30 03:43:27.336368 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:27.336345 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" Oct 30 03:43:27.634370 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:27.634332 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv"] Oct 30 03:43:27.636649 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:43:27.636614 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1da09d_c287_4103_8a7a_0ebfd670e4d9.slice/crio-4d9855b86f21d480c3096d782f6e808f7bbfaeadbb6f03cd4299ca8fe41d9a37 WatchSource:0}: Error finding container 4d9855b86f21d480c3096d782f6e808f7bbfaeadbb6f03cd4299ca8fe41d9a37: Status 404 returned error can't find the container with id 4d9855b86f21d480c3096d782f6e808f7bbfaeadbb6f03cd4299ca8fe41d9a37 Oct 30 03:43:28.296521 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:28.296490 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" event={"ID":"cd1da09d-c287-4103-8a7a-0ebfd670e4d9","Type":"ContainerStarted","Data":"5562f9ae497a91bea1efd7b4d64c12d2e5e0ff396fc87849222b9dea8988a66f"} Oct 30 03:43:28.296954 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:28.296539 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" event={"ID":"cd1da09d-c287-4103-8a7a-0ebfd670e4d9","Type":"ContainerStarted","Data":"4d9855b86f21d480c3096d782f6e808f7bbfaeadbb6f03cd4299ca8fe41d9a37"} Oct 30 03:43:28.296954 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:28.296594 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" Oct 30 03:43:28.340638 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:28.340596 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" podStartSLOduration=2.34058485 podStartE2EDuration="2.34058485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:28.339922298 +0000 UTC m=+3769.353200399" watchObservedRunningTime="2026-10-30 03:43:28.34058485 +0000 UTC m=+3769.353862953" Oct 30 03:43:32.573627 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:32.573592 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:43:32.574112 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:32.573835 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="671da0dd-3d47-4bbc-9b78-491ba81d3007" containerName="otc-container" containerID="cri-o://1567c624fa947dc3babb8e8791592a41bddf2800fa92f9962c9018f739a9a23f" gracePeriod=30 Oct 30 03:43:34.916423 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:34.916400 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:34.976591 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:34.976565 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-monitoring-tls-certs\") pod \"671da0dd-3d47-4bbc-9b78-491ba81d3007\" (UID: \"671da0dd-3d47-4bbc-9b78-491ba81d3007\") " Oct 30 03:43:34.976724 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:34.976605 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-data-science-collector-ta-client-cert\") pod \"671da0dd-3d47-4bbc-9b78-491ba81d3007\" (UID: \"671da0dd-3d47-4bbc-9b78-491ba81d3007\") " Oct 30 03:43:34.976724 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:34.976649 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/671da0dd-3d47-4bbc-9b78-491ba81d3007-otc-internal\" (UniqueName: \"kubernetes.io/configmap/671da0dd-3d47-4bbc-9b78-491ba81d3007-otc-internal\") pod \"671da0dd-3d47-4bbc-9b78-491ba81d3007\" (UID: \"671da0dd-3d47-4bbc-9b78-491ba81d3007\") " Oct 30 03:43:34.976724 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:34.976685 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-tls-certs\" (UniqueName: \"kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-tls-certs\") pod \"671da0dd-3d47-4bbc-9b78-491ba81d3007\" (UID: \"671da0dd-3d47-4bbc-9b78-491ba81d3007\") " Oct 30 03:43:34.976895 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:34.976817 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/671da0dd-3d47-4bbc-9b78-491ba81d3007-kube-api-access-bh8j4\" (UniqueName: \"kubernetes.io/projected/671da0dd-3d47-4bbc-9b78-491ba81d3007-kube-api-access-bh8j4\") pod \"671da0dd-3d47-4bbc-9b78-491ba81d3007\" (UID: \"671da0dd-3d47-4bbc-9b78-491ba81d3007\") " Oct 30 03:43:34.977072 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:34.977049 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/671da0dd-3d47-4bbc-9b78-491ba81d3007-otc-internal" pod "671da0dd-3d47-4bbc-9b78-491ba81d3007" (UID: "671da0dd-3d47-4bbc-9b78-491ba81d3007"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:43:34.978845 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:34.978788 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-monitoring-tls-certs" pod "671da0dd-3d47-4bbc-9b78-491ba81d3007" (UID: "671da0dd-3d47-4bbc-9b78-491ba81d3007"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:43:34.978845 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:34.978812 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-tls-certs" pod "671da0dd-3d47-4bbc-9b78-491ba81d3007" (UID: "671da0dd-3d47-4bbc-9b78-491ba81d3007"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:43:34.979073 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:34.979053 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-data-science-collector-ta-client-cert" pod "671da0dd-3d47-4bbc-9b78-491ba81d3007" (UID: "671da0dd-3d47-4bbc-9b78-491ba81d3007"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:43:34.979124 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:34.979075 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/671da0dd-3d47-4bbc-9b78-491ba81d3007-kube-api-access-bh8j4" pod "671da0dd-3d47-4bbc-9b78-491ba81d3007" (UID: "671da0dd-3d47-4bbc-9b78-491ba81d3007"). InnerVolumeSpecName "kube-api-access-bh8j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:43:35.078100 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.078069 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:35.078100 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.078097 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:35.078346 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.078107 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/671da0dd-3d47-4bbc-9b78-491ba81d3007-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:35.078346 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.078117 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671da0dd-3d47-4bbc-9b78-491ba81d3007-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:35.078346 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.078125 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh8j4\" (UniqueName: \"kubernetes.io/projected/671da0dd-3d47-4bbc-9b78-491ba81d3007-kube-api-access-bh8j4\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:35.332924 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.332835 4925 generic.go:356] "Generic (PLEG): container finished" podID="671da0dd-3d47-4bbc-9b78-491ba81d3007" containerID="1567c624fa947dc3babb8e8791592a41bddf2800fa92f9962c9018f739a9a23f" exitCode=0 Oct 30 03:43:35.332924 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.332886 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:35.333162 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.332929 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"671da0dd-3d47-4bbc-9b78-491ba81d3007","Type":"ContainerDied","Data":"1567c624fa947dc3babb8e8791592a41bddf2800fa92f9962c9018f739a9a23f"} Oct 30 03:43:35.333162 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.332990 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"671da0dd-3d47-4bbc-9b78-491ba81d3007","Type":"ContainerDied","Data":"e9a1661e6f7a0857343529de38b5018824ea539cf9fd7b7d1008bfcc241285c8"} Oct 30 03:43:35.333162 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.333018 4925 scope.go:122] "RemoveContainer" containerID="1567c624fa947dc3babb8e8791592a41bddf2800fa92f9962c9018f739a9a23f" Oct 30 03:43:35.341714 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.341697 4925 scope.go:122] "RemoveContainer" containerID="1567c624fa947dc3babb8e8791592a41bddf2800fa92f9962c9018f739a9a23f" Oct 30 03:43:35.342012 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:43:35.341991 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1567c624fa947dc3babb8e8791592a41bddf2800fa92f9962c9018f739a9a23f\": container with ID starting with 1567c624fa947dc3babb8e8791592a41bddf2800fa92f9962c9018f739a9a23f not found: ID does not exist" containerID="1567c624fa947dc3babb8e8791592a41bddf2800fa92f9962c9018f739a9a23f" Oct 30 03:43:35.342071 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.342020 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1567c624fa947dc3babb8e8791592a41bddf2800fa92f9962c9018f739a9a23f"} err="failed to get container status \"1567c624fa947dc3babb8e8791592a41bddf2800fa92f9962c9018f739a9a23f\": rpc error: code = NotFound desc = could not find container \"1567c624fa947dc3babb8e8791592a41bddf2800fa92f9962c9018f739a9a23f\": container with ID starting with 1567c624fa947dc3babb8e8791592a41bddf2800fa92f9962c9018f739a9a23f not found: ID does not exist" Oct 30 03:43:35.387170 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.387148 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:43:35.390507 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.390488 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:43:35.439546 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.439525 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:43:35.439891 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.439878 4925 cpu_manager.go:414] "containerMap: removing container" podUID="671da0dd-3d47-4bbc-9b78-491ba81d3007" containerName="otc-container" Oct 30 03:43:35.439942 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.439891 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="671da0dd-3d47-4bbc-9b78-491ba81d3007" containerName="otc-container" Oct 30 03:43:35.439977 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.439960 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="671da0dd-3d47-4bbc-9b78-491ba81d3007" containerName="otc-container" Oct 30 03:43:35.444326 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.444310 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:35.464628 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.464611 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\"" Oct 30 03:43:35.464781 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.464744 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 03:43:35.470020 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.470002 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-72c79cac\"" Oct 30 03:43:35.471779 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.471743 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:43:35.481174 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.481142 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:35.481267 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.481196 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:35.481315 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.481295 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:35.481353 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.481329 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:35.481398 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.481384 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zks2r\" (UniqueName: \"kubernetes.io/projected/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-kube-api-access-zks2r\") pod \"data-science-collector-collector-0\" (UID: \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:35.561734 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.561711 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 03:43:35.561893 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.561876 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:43:35.582376 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.582341 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:35.582473 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.582387 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:35.582473 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.582420 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zks2r\" (UniqueName: \"kubernetes.io/projected/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-kube-api-access-zks2r\") pod \"data-science-collector-collector-0\" (UID: \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:35.582473 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.582461 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:35.582633 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.582507 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:35.583080 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.583020 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:35.584952 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.584935 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:35.585027 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.584935 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:35.585027 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.584987 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:35.605943 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.605915 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zks2r\" (UniqueName: \"kubernetes.io/projected/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-kube-api-access-zks2r\") pod \"data-science-collector-collector-0\" (UID: \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:35.756189 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:35.756168 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:36.059837 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:36.059813 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:43:36.062030 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:43:36.061994 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0120503a_e4dd_4387_9cae_3b6bb3ef0b6e.slice/crio-6b9382e0f1e077368462d90f6bd5a87627f1e07798c44e92685e900eddb20fd9 WatchSource:0}: Error finding container 6b9382e0f1e077368462d90f6bd5a87627f1e07798c44e92685e900eddb20fd9: Status 404 returned error can't find the container with id 6b9382e0f1e077368462d90f6bd5a87627f1e07798c44e92685e900eddb20fd9 Oct 30 03:43:36.064150 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:36.064124 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:43:36.064205 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:36.064178 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:43:36.341087 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:36.341057 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e","Type":"ContainerStarted","Data":"39fad5f02dabba7813d90f652234210eab3697b767af126753e359a945a097a7"} Oct 30 03:43:36.341273 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:36.341115 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e","Type":"ContainerStarted","Data":"6b9382e0f1e077368462d90f6bd5a87627f1e07798c44e92685e900eddb20fd9"} Oct 30 03:43:36.391867 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:36.391823 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.391811416 podStartE2EDuration="1.391811416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:36.391111502 +0000 UTC m=+3777.404389615" watchObservedRunningTime="2026-10-30 03:43:36.391811416 +0000 UTC m=+3777.405089518" Oct 30 03:43:36.568103 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:36.568072 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv"] Oct 30 03:43:36.568290 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:36.568245 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" podUID="cd1da09d-c287-4103-8a7a-0ebfd670e4d9" containerName="korrel8r" containerID="cri-o://5562f9ae497a91bea1efd7b4d64c12d2e5e0ff396fc87849222b9dea8988a66f" gracePeriod=30 Oct 30 03:43:36.570141 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:36.570120 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" Oct 30 03:43:36.809646 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:36.809622 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" Oct 30 03:43:36.894170 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:36.894126 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-kube-api-access-dqxwv\" (UniqueName: \"kubernetes.io/projected/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-kube-api-access-dqxwv\") pod \"cd1da09d-c287-4103-8a7a-0ebfd670e4d9\" (UID: \"cd1da09d-c287-4103-8a7a-0ebfd670e4d9\") " Oct 30 03:43:36.894345 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:36.894203 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-config\" (UniqueName: \"kubernetes.io/configmap/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-config\") pod \"cd1da09d-c287-4103-8a7a-0ebfd670e4d9\" (UID: \"cd1da09d-c287-4103-8a7a-0ebfd670e4d9\") " Oct 30 03:43:36.894345 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:36.894225 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-tls\" (UniqueName: \"kubernetes.io/secret/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-tls\") pod \"cd1da09d-c287-4103-8a7a-0ebfd670e4d9\" (UID: \"cd1da09d-c287-4103-8a7a-0ebfd670e4d9\") " Oct 30 03:43:36.894563 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:36.894540 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-config" pod "cd1da09d-c287-4103-8a7a-0ebfd670e4d9" (UID: "cd1da09d-c287-4103-8a7a-0ebfd670e4d9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:43:36.896236 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:36.896200 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-kube-api-access-dqxwv" pod "cd1da09d-c287-4103-8a7a-0ebfd670e4d9" (UID: "cd1da09d-c287-4103-8a7a-0ebfd670e4d9"). InnerVolumeSpecName "kube-api-access-dqxwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:43:36.896338 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:36.896234 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-tls" pod "cd1da09d-c287-4103-8a7a-0ebfd670e4d9" (UID: "cd1da09d-c287-4103-8a7a-0ebfd670e4d9"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:43:36.994943 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:36.994880 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqxwv\" (UniqueName: \"kubernetes.io/projected/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-kube-api-access-dqxwv\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:36.994943 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:36.994904 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:36.994943 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:36.994914 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/cd1da09d-c287-4103-8a7a-0ebfd670e4d9-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:37.119127 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:37.119095 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="671da0dd-3d47-4bbc-9b78-491ba81d3007" path="/var/lib/kubelet/pods/671da0dd-3d47-4bbc-9b78-491ba81d3007/volumes" Oct 30 03:43:37.348499 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:37.348461 4925 generic.go:356] "Generic (PLEG): container finished" podID="cd1da09d-c287-4103-8a7a-0ebfd670e4d9" containerID="5562f9ae497a91bea1efd7b4d64c12d2e5e0ff396fc87849222b9dea8988a66f" exitCode=0 Oct 30 03:43:37.348678 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:37.348523 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" Oct 30 03:43:37.348678 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:37.348546 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" event={"ID":"cd1da09d-c287-4103-8a7a-0ebfd670e4d9","Type":"ContainerDied","Data":"5562f9ae497a91bea1efd7b4d64c12d2e5e0ff396fc87849222b9dea8988a66f"} Oct 30 03:43:37.348678 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:37.348612 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv" event={"ID":"cd1da09d-c287-4103-8a7a-0ebfd670e4d9","Type":"ContainerDied","Data":"4d9855b86f21d480c3096d782f6e808f7bbfaeadbb6f03cd4299ca8fe41d9a37"} Oct 30 03:43:37.348678 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:37.348641 4925 scope.go:122] "RemoveContainer" containerID="5562f9ae497a91bea1efd7b4d64c12d2e5e0ff396fc87849222b9dea8988a66f" Oct 30 03:43:37.357410 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:37.357380 4925 scope.go:122] "RemoveContainer" containerID="5562f9ae497a91bea1efd7b4d64c12d2e5e0ff396fc87849222b9dea8988a66f" Oct 30 03:43:37.360225 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:43:37.359851 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5562f9ae497a91bea1efd7b4d64c12d2e5e0ff396fc87849222b9dea8988a66f\": container with ID starting with 5562f9ae497a91bea1efd7b4d64c12d2e5e0ff396fc87849222b9dea8988a66f not found: ID does not exist" containerID="5562f9ae497a91bea1efd7b4d64c12d2e5e0ff396fc87849222b9dea8988a66f" Oct 30 03:43:37.360225 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:37.359887 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5562f9ae497a91bea1efd7b4d64c12d2e5e0ff396fc87849222b9dea8988a66f"} err="failed to get container status \"5562f9ae497a91bea1efd7b4d64c12d2e5e0ff396fc87849222b9dea8988a66f\": rpc error: code = NotFound desc = could not find container \"5562f9ae497a91bea1efd7b4d64c12d2e5e0ff396fc87849222b9dea8988a66f\": container with ID starting with 5562f9ae497a91bea1efd7b4d64c12d2e5e0ff396fc87849222b9dea8988a66f not found: ID does not exist" Oct 30 03:43:37.400801 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:37.400777 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv"] Oct 30 03:43:37.404613 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:37.404592 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-grvkv"] Oct 30 03:43:39.119036 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:39.119005 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd1da09d-c287-4103-8a7a-0ebfd670e4d9" path="/var/lib/kubelet/pods/cd1da09d-c287-4103-8a7a-0ebfd670e4d9/volumes" Oct 30 03:43:39.620292 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:39.620254 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:43:39.620499 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:39.620472 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="0120503a-e4dd-4387-9cae-3b6bb3ef0b6e" containerName="otc-container" containerID="cri-o://39fad5f02dabba7813d90f652234210eab3697b767af126753e359a945a097a7" gracePeriod=30 Oct 30 03:43:39.859625 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:39.859599 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:39.917796 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:39.917719 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-data-science-collector-ta-client-cert\") pod \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\" (UID: \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\") " Oct 30 03:43:39.917796 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:39.917782 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-tls-certs\" (UniqueName: \"kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-tls-certs\") pod \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\" (UID: \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\") " Oct 30 03:43:39.918017 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:39.917808 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-monitoring-tls-certs\") pod \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\" (UID: \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\") " Oct 30 03:43:39.918017 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:39.917832 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-otc-internal\" (UniqueName: \"kubernetes.io/configmap/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-otc-internal\") pod \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\" (UID: \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\") " Oct 30 03:43:39.918017 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:39.917861 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-kube-api-access-zks2r\" (UniqueName: \"kubernetes.io/projected/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-kube-api-access-zks2r\") pod \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\" (UID: \"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e\") " Oct 30 03:43:39.919666 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:39.919633 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-otc-internal" pod "0120503a-e4dd-4387-9cae-3b6bb3ef0b6e" (UID: "0120503a-e4dd-4387-9cae-3b6bb3ef0b6e"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:43:39.920189 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:39.920163 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-tls-certs" pod "0120503a-e4dd-4387-9cae-3b6bb3ef0b6e" (UID: "0120503a-e4dd-4387-9cae-3b6bb3ef0b6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:43:39.920361 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:39.920333 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-kube-api-access-zks2r" pod "0120503a-e4dd-4387-9cae-3b6bb3ef0b6e" (UID: "0120503a-e4dd-4387-9cae-3b6bb3ef0b6e"). InnerVolumeSpecName "kube-api-access-zks2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:43:39.920733 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:39.920708 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-data-science-collector-ta-client-cert" pod "0120503a-e4dd-4387-9cae-3b6bb3ef0b6e" (UID: "0120503a-e4dd-4387-9cae-3b6bb3ef0b6e"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:43:39.920818 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:39.920790 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-monitoring-tls-certs" pod "0120503a-e4dd-4387-9cae-3b6bb3ef0b6e" (UID: "0120503a-e4dd-4387-9cae-3b6bb3ef0b6e"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:43:40.019256 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.019225 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:40.019256 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.019249 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:40.019256 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.019262 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:40.019485 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.019271 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:40.019485 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.019280 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zks2r\" (UniqueName: \"kubernetes.io/projected/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e-kube-api-access-zks2r\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:40.367592 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.367557 4925 generic.go:356] "Generic (PLEG): container finished" podID="0120503a-e4dd-4387-9cae-3b6bb3ef0b6e" containerID="39fad5f02dabba7813d90f652234210eab3697b767af126753e359a945a097a7" exitCode=0 Oct 30 03:43:40.368036 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.367603 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:40.368036 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.367649 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e","Type":"ContainerDied","Data":"39fad5f02dabba7813d90f652234210eab3697b767af126753e359a945a097a7"} Oct 30 03:43:40.368036 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.367696 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"0120503a-e4dd-4387-9cae-3b6bb3ef0b6e","Type":"ContainerDied","Data":"6b9382e0f1e077368462d90f6bd5a87627f1e07798c44e92685e900eddb20fd9"} Oct 30 03:43:40.368036 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.367717 4925 scope.go:122] "RemoveContainer" containerID="39fad5f02dabba7813d90f652234210eab3697b767af126753e359a945a097a7" Oct 30 03:43:40.376541 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.376523 4925 scope.go:122] "RemoveContainer" containerID="39fad5f02dabba7813d90f652234210eab3697b767af126753e359a945a097a7" Oct 30 03:43:40.376827 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:43:40.376808 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39fad5f02dabba7813d90f652234210eab3697b767af126753e359a945a097a7\": container with ID starting with 39fad5f02dabba7813d90f652234210eab3697b767af126753e359a945a097a7 not found: ID does not exist" containerID="39fad5f02dabba7813d90f652234210eab3697b767af126753e359a945a097a7" Oct 30 03:43:40.376878 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.376835 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39fad5f02dabba7813d90f652234210eab3697b767af126753e359a945a097a7"} err="failed to get container status \"39fad5f02dabba7813d90f652234210eab3697b767af126753e359a945a097a7\": rpc error: code = NotFound desc = could not find container \"39fad5f02dabba7813d90f652234210eab3697b767af126753e359a945a097a7\": container with ID starting with 39fad5f02dabba7813d90f652234210eab3697b767af126753e359a945a097a7 not found: ID does not exist" Oct 30 03:43:40.424227 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.424205 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:43:40.436667 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.436645 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:43:40.470735 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.470702 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:43:40.471094 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.471077 4925 cpu_manager.go:414] "containerMap: removing container" podUID="0120503a-e4dd-4387-9cae-3b6bb3ef0b6e" containerName="otc-container" Oct 30 03:43:40.471183 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.471096 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0120503a-e4dd-4387-9cae-3b6bb3ef0b6e" containerName="otc-container" Oct 30 03:43:40.471183 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.471113 4925 cpu_manager.go:414] "containerMap: removing container" podUID="cd1da09d-c287-4103-8a7a-0ebfd670e4d9" containerName="korrel8r" Oct 30 03:43:40.471183 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.471120 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd1da09d-c287-4103-8a7a-0ebfd670e4d9" containerName="korrel8r" Oct 30 03:43:40.471313 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.471198 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="0120503a-e4dd-4387-9cae-3b6bb3ef0b6e" containerName="otc-container" Oct 30 03:43:40.471313 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.471206 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd1da09d-c287-4103-8a7a-0ebfd670e4d9" containerName="korrel8r" Oct 30 03:43:40.475781 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.475765 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:40.494895 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.494871 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 03:43:40.495157 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.495137 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 03:43:40.495222 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.495192 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 03:43:40.495275 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.495253 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\"" Oct 30 03:43:40.495467 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.495451 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:43:40.496967 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.496950 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:43:40.523352 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.523326 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"7f99763a-20c3-4709-89a0-79576893d62a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:40.523449 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.523354 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f99763a-20c3-4709-89a0-79576893d62a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:40.523449 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.523427 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9mpx\" (UniqueName: \"kubernetes.io/projected/7f99763a-20c3-4709-89a0-79576893d62a-kube-api-access-l9mpx\") pod \"data-science-collector-collector-0\" (UID: \"7f99763a-20c3-4709-89a0-79576893d62a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:40.523522 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.523470 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7f99763a-20c3-4709-89a0-79576893d62a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"7f99763a-20c3-4709-89a0-79576893d62a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:40.523576 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.523550 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f99763a-20c3-4709-89a0-79576893d62a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:40.625004 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.624942 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f99763a-20c3-4709-89a0-79576893d62a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:40.625107 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.625009 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"7f99763a-20c3-4709-89a0-79576893d62a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:40.625107 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.625028 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f99763a-20c3-4709-89a0-79576893d62a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:40.625107 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.625044 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9mpx\" (UniqueName: \"kubernetes.io/projected/7f99763a-20c3-4709-89a0-79576893d62a-kube-api-access-l9mpx\") pod \"data-science-collector-collector-0\" (UID: \"7f99763a-20c3-4709-89a0-79576893d62a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:40.625107 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.625065 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7f99763a-20c3-4709-89a0-79576893d62a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"7f99763a-20c3-4709-89a0-79576893d62a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:40.625778 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.625740 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7f99763a-20c3-4709-89a0-79576893d62a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"7f99763a-20c3-4709-89a0-79576893d62a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:40.627357 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.627338 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f99763a-20c3-4709-89a0-79576893d62a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:40.627442 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.627404 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"7f99763a-20c3-4709-89a0-79576893d62a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:40.627442 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.627419 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f99763a-20c3-4709-89a0-79576893d62a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:40.648765 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.648717 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9mpx\" (UniqueName: \"kubernetes.io/projected/7f99763a-20c3-4709-89a0-79576893d62a-kube-api-access-l9mpx\") pod \"data-science-collector-collector-0\" (UID: \"7f99763a-20c3-4709-89a0-79576893d62a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:40.787148 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:40.787110 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:41.087226 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:41.087194 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:43:41.090473 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:43:41.090445 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f99763a_20c3_4709_89a0_79576893d62a.slice/crio-71bfc2d165f825cbc20a1bef01f1a9bb80e843d296b6e8167f9b2b19b162a118 WatchSource:0}: Error finding container 71bfc2d165f825cbc20a1bef01f1a9bb80e843d296b6e8167f9b2b19b162a118: Status 404 returned error can't find the container with id 71bfc2d165f825cbc20a1bef01f1a9bb80e843d296b6e8167f9b2b19b162a118 Oct 30 03:43:41.092920 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:41.092886 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:43:41.092992 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:41.092960 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:43:41.119682 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:41.119652 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0120503a-e4dd-4387-9cae-3b6bb3ef0b6e" path="/var/lib/kubelet/pods/0120503a-e4dd-4387-9cae-3b6bb3ef0b6e/volumes" Oct 30 03:43:41.374204 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:41.374118 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7f99763a-20c3-4709-89a0-79576893d62a","Type":"ContainerStarted","Data":"78b77e4b478dd5ff4606363c2333a0059917d6b5856d934286603c9094e5da7a"} Oct 30 03:43:41.374204 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:41.374185 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7f99763a-20c3-4709-89a0-79576893d62a","Type":"ContainerStarted","Data":"71bfc2d165f825cbc20a1bef01f1a9bb80e843d296b6e8167f9b2b19b162a118"} Oct 30 03:43:41.419204 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:41.419160 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.419147959 podStartE2EDuration="1.419147959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:41.417880584 +0000 UTC m=+3782.431158685" watchObservedRunningTime="2026-10-30 03:43:41.419147959 +0000 UTC m=+3782.432426060" Oct 30 03:43:46.528322 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:46.528290 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6"] Oct 30 03:43:46.531804 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:46.531786 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" Oct 30 03:43:46.558271 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:46.558248 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6"] Oct 30 03:43:46.573503 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:46.573474 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdln7\" (UniqueName: \"kubernetes.io/projected/341c505d-43b8-43d6-97d3-da351a8f7cef-kube-api-access-xdln7\") pod \"korrel8r-66d6f7f696-c6wn6\" (UID: \"341c505d-43b8-43d6-97d3-da351a8f7cef\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" Oct 30 03:43:46.573613 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:46.573525 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/341c505d-43b8-43d6-97d3-da351a8f7cef-tls\") pod \"korrel8r-66d6f7f696-c6wn6\" (UID: \"341c505d-43b8-43d6-97d3-da351a8f7cef\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" Oct 30 03:43:46.573613 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:46.573574 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/341c505d-43b8-43d6-97d3-da351a8f7cef-config\") pod \"korrel8r-66d6f7f696-c6wn6\" (UID: \"341c505d-43b8-43d6-97d3-da351a8f7cef\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" Oct 30 03:43:46.674696 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:46.674658 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/341c505d-43b8-43d6-97d3-da351a8f7cef-tls\") pod \"korrel8r-66d6f7f696-c6wn6\" (UID: \"341c505d-43b8-43d6-97d3-da351a8f7cef\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" Oct 30 03:43:46.674696 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:46.674702 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/341c505d-43b8-43d6-97d3-da351a8f7cef-config\") pod \"korrel8r-66d6f7f696-c6wn6\" (UID: \"341c505d-43b8-43d6-97d3-da351a8f7cef\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" Oct 30 03:43:46.674901 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:46.674787 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdln7\" (UniqueName: \"kubernetes.io/projected/341c505d-43b8-43d6-97d3-da351a8f7cef-kube-api-access-xdln7\") pod \"korrel8r-66d6f7f696-c6wn6\" (UID: \"341c505d-43b8-43d6-97d3-da351a8f7cef\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" Oct 30 03:43:46.675613 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:46.675591 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/341c505d-43b8-43d6-97d3-da351a8f7cef-config\") pod \"korrel8r-66d6f7f696-c6wn6\" (UID: \"341c505d-43b8-43d6-97d3-da351a8f7cef\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" Oct 30 03:43:46.677171 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:46.677146 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/341c505d-43b8-43d6-97d3-da351a8f7cef-tls\") pod \"korrel8r-66d6f7f696-c6wn6\" (UID: \"341c505d-43b8-43d6-97d3-da351a8f7cef\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" Oct 30 03:43:46.699022 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:46.699000 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdln7\" (UniqueName: \"kubernetes.io/projected/341c505d-43b8-43d6-97d3-da351a8f7cef-kube-api-access-xdln7\") pod \"korrel8r-66d6f7f696-c6wn6\" (UID: \"341c505d-43b8-43d6-97d3-da351a8f7cef\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" Oct 30 03:43:46.844380 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:46.844334 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" Oct 30 03:43:47.151716 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:47.151692 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6"] Oct 30 03:43:47.154019 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:43:47.153987 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341c505d_43b8_43d6_97d3_da351a8f7cef.slice/crio-e33897b839a66b654cd653425e63ae43b3b0d78d7302c8d6ca4a4c9879943549 WatchSource:0}: Error finding container e33897b839a66b654cd653425e63ae43b3b0d78d7302c8d6ca4a4c9879943549: Status 404 returned error can't find the container with id e33897b839a66b654cd653425e63ae43b3b0d78d7302c8d6ca4a4c9879943549 Oct 30 03:43:47.405932 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:47.405842 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" event={"ID":"341c505d-43b8-43d6-97d3-da351a8f7cef","Type":"ContainerStarted","Data":"4db562d5e662c5979fb2acc107be72fa7c6cd4410880cd95d231f40ba1053a46"} Oct 30 03:43:47.405932 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:47.405901 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" event={"ID":"341c505d-43b8-43d6-97d3-da351a8f7cef","Type":"ContainerStarted","Data":"e33897b839a66b654cd653425e63ae43b3b0d78d7302c8d6ca4a4c9879943549"} Oct 30 03:43:47.406106 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:47.406059 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" Oct 30 03:43:47.452352 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:47.452303 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" podStartSLOduration=1.452290662 podStartE2EDuration="1.452290662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:47.451185259 +0000 UTC m=+3788.464463344" watchObservedRunningTime="2026-10-30 03:43:47.452290662 +0000 UTC m=+3788.465568764" Oct 30 03:43:48.162543 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:43:48.162505 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:43:50.683233 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:50.683197 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:43:50.683677 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:50.683410 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="7f99763a-20c3-4709-89a0-79576893d62a" containerName="otc-container" containerID="cri-o://78b77e4b478dd5ff4606363c2333a0059917d6b5856d934286603c9094e5da7a" gracePeriod=30 Oct 30 03:43:53.019299 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.019274 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:53.127413 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.127337 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-data-science-collector-ta-client-cert\") pod \"7f99763a-20c3-4709-89a0-79576893d62a\" (UID: \"7f99763a-20c3-4709-89a0-79576893d62a\") " Oct 30 03:43:53.127413 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.127376 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f99763a-20c3-4709-89a0-79576893d62a-kube-api-access-l9mpx\" (UniqueName: \"kubernetes.io/projected/7f99763a-20c3-4709-89a0-79576893d62a-kube-api-access-l9mpx\") pod \"7f99763a-20c3-4709-89a0-79576893d62a\" (UID: \"7f99763a-20c3-4709-89a0-79576893d62a\") " Oct 30 03:43:53.127579 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.127435 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f99763a-20c3-4709-89a0-79576893d62a-otc-internal\" (UniqueName: \"kubernetes.io/configmap/7f99763a-20c3-4709-89a0-79576893d62a-otc-internal\") pod \"7f99763a-20c3-4709-89a0-79576893d62a\" (UID: \"7f99763a-20c3-4709-89a0-79576893d62a\") " Oct 30 03:43:53.127579 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.127474 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-monitoring-tls-certs\") pod \"7f99763a-20c3-4709-89a0-79576893d62a\" (UID: \"7f99763a-20c3-4709-89a0-79576893d62a\") " Oct 30 03:43:53.127674 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.127570 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-tls-certs\") pod \"7f99763a-20c3-4709-89a0-79576893d62a\" (UID: \"7f99763a-20c3-4709-89a0-79576893d62a\") " Oct 30 03:43:53.127875 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.127845 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f99763a-20c3-4709-89a0-79576893d62a-otc-internal" pod "7f99763a-20c3-4709-89a0-79576893d62a" (UID: "7f99763a-20c3-4709-89a0-79576893d62a"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:43:53.129599 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.129571 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-tls-certs" pod "7f99763a-20c3-4709-89a0-79576893d62a" (UID: "7f99763a-20c3-4709-89a0-79576893d62a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:43:53.129703 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.129599 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-data-science-collector-ta-client-cert" pod "7f99763a-20c3-4709-89a0-79576893d62a" (UID: "7f99763a-20c3-4709-89a0-79576893d62a"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:43:53.129703 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.129664 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-monitoring-tls-certs" pod "7f99763a-20c3-4709-89a0-79576893d62a" (UID: "7f99763a-20c3-4709-89a0-79576893d62a"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:43:53.129802 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.129711 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f99763a-20c3-4709-89a0-79576893d62a-kube-api-access-l9mpx" pod "7f99763a-20c3-4709-89a0-79576893d62a" (UID: "7f99763a-20c3-4709-89a0-79576893d62a"). InnerVolumeSpecName "kube-api-access-l9mpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:43:53.228637 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.228613 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7f99763a-20c3-4709-89a0-79576893d62a-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:53.228637 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.228634 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:53.228801 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.228643 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:53.228801 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.228654 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7f99763a-20c3-4709-89a0-79576893d62a-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:53.228801 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.228661 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9mpx\" (UniqueName: \"kubernetes.io/projected/7f99763a-20c3-4709-89a0-79576893d62a-kube-api-access-l9mpx\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:53.442241 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.442154 4925 generic.go:356] "Generic (PLEG): container finished" podID="7f99763a-20c3-4709-89a0-79576893d62a" containerID="78b77e4b478dd5ff4606363c2333a0059917d6b5856d934286603c9094e5da7a" exitCode=0 Oct 30 03:43:53.442241 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.442212 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:53.442417 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.442233 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7f99763a-20c3-4709-89a0-79576893d62a","Type":"ContainerDied","Data":"78b77e4b478dd5ff4606363c2333a0059917d6b5856d934286603c9094e5da7a"} Oct 30 03:43:53.442417 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.442320 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7f99763a-20c3-4709-89a0-79576893d62a","Type":"ContainerDied","Data":"71bfc2d165f825cbc20a1bef01f1a9bb80e843d296b6e8167f9b2b19b162a118"} Oct 30 03:43:53.442417 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.442347 4925 scope.go:122] "RemoveContainer" containerID="78b77e4b478dd5ff4606363c2333a0059917d6b5856d934286603c9094e5da7a" Oct 30 03:43:53.451659 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.451639 4925 scope.go:122] "RemoveContainer" containerID="78b77e4b478dd5ff4606363c2333a0059917d6b5856d934286603c9094e5da7a" Oct 30 03:43:53.451976 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:43:53.451952 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b77e4b478dd5ff4606363c2333a0059917d6b5856d934286603c9094e5da7a\": container with ID starting with 78b77e4b478dd5ff4606363c2333a0059917d6b5856d934286603c9094e5da7a not found: ID does not exist" containerID="78b77e4b478dd5ff4606363c2333a0059917d6b5856d934286603c9094e5da7a" Oct 30 03:43:53.452037 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.451988 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b77e4b478dd5ff4606363c2333a0059917d6b5856d934286603c9094e5da7a"} err="failed to get container status \"78b77e4b478dd5ff4606363c2333a0059917d6b5856d934286603c9094e5da7a\": rpc error: code = NotFound desc = could not find container \"78b77e4b478dd5ff4606363c2333a0059917d6b5856d934286603c9094e5da7a\": container with ID starting with 78b77e4b478dd5ff4606363c2333a0059917d6b5856d934286603c9094e5da7a not found: ID does not exist" Oct 30 03:43:53.501595 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.501564 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:43:53.509573 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.509551 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:43:53.550915 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.550889 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:43:53.551328 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.551310 4925 cpu_manager.go:414] "containerMap: removing container" podUID="7f99763a-20c3-4709-89a0-79576893d62a" containerName="otc-container" Oct 30 03:43:53.551411 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.551329 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f99763a-20c3-4709-89a0-79576893d62a" containerName="otc-container" Oct 30 03:43:53.551468 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.551444 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f99763a-20c3-4709-89a0-79576893d62a" containerName="otc-container" Oct 30 03:43:53.555942 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.555924 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:53.575978 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.575950 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 03:43:53.576230 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.575968 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 03:43:53.576326 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.576020 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:43:53.576326 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.576139 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\"" Oct 30 03:43:53.577738 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.577719 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:43:53.631926 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.631900 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8c7aa059-e19a-488e-bced-47437b5b0c43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:53.632069 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.631956 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sxqf\" (UniqueName: \"kubernetes.io/projected/8c7aa059-e19a-488e-bced-47437b5b0c43-kube-api-access-8sxqf\") pod \"data-science-collector-collector-0\" (UID: \"8c7aa059-e19a-488e-bced-47437b5b0c43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:53.632069 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.632035 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8c7aa059-e19a-488e-bced-47437b5b0c43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:53.632155 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.632090 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8c7aa059-e19a-488e-bced-47437b5b0c43-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"8c7aa059-e19a-488e-bced-47437b5b0c43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:53.632224 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.632203 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"8c7aa059-e19a-488e-bced-47437b5b0c43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:53.654444 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.654423 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-72c79cac\"" Oct 30 03:43:53.732690 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.732631 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8c7aa059-e19a-488e-bced-47437b5b0c43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:53.732690 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.732670 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sxqf\" (UniqueName: \"kubernetes.io/projected/8c7aa059-e19a-488e-bced-47437b5b0c43-kube-api-access-8sxqf\") pod \"data-science-collector-collector-0\" (UID: \"8c7aa059-e19a-488e-bced-47437b5b0c43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:53.732690 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.732691 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8c7aa059-e19a-488e-bced-47437b5b0c43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:53.732932 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.732719 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8c7aa059-e19a-488e-bced-47437b5b0c43-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"8c7aa059-e19a-488e-bced-47437b5b0c43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:53.732932 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.732814 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"8c7aa059-e19a-488e-bced-47437b5b0c43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:53.733522 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.733499 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8c7aa059-e19a-488e-bced-47437b5b0c43-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"8c7aa059-e19a-488e-bced-47437b5b0c43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:53.735224 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.735206 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"8c7aa059-e19a-488e-bced-47437b5b0c43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:53.735224 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.735218 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8c7aa059-e19a-488e-bced-47437b5b0c43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:53.735383 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.735365 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8c7aa059-e19a-488e-bced-47437b5b0c43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:53.755705 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.755684 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sxqf\" (UniqueName: \"kubernetes.io/projected/8c7aa059-e19a-488e-bced-47437b5b0c43-kube-api-access-8sxqf\") pod \"data-science-collector-collector-0\" (UID: \"8c7aa059-e19a-488e-bced-47437b5b0c43\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:53.868768 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:53.868727 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:43:54.174000 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:54.173963 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:43:54.176356 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:43:54.176334 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c7aa059_e19a_488e_bced_47437b5b0c43.slice/crio-28325e85639a9f0850df8bfc38d505b95733c1eaf4303a0aff9d0bb238ee6def WatchSource:0}: Error finding container 28325e85639a9f0850df8bfc38d505b95733c1eaf4303a0aff9d0bb238ee6def: Status 404 returned error can't find the container with id 28325e85639a9f0850df8bfc38d505b95733c1eaf4303a0aff9d0bb238ee6def Oct 30 03:43:54.178808 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:54.178777 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:43:54.178884 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:54.178835 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:43:54.448878 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:54.448792 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"8c7aa059-e19a-488e-bced-47437b5b0c43","Type":"ContainerStarted","Data":"13210c2108f849f233b240e328709b62bfaab2a70590016a8cd35683e0b644f2"} Oct 30 03:43:54.448878 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:54.448855 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"8c7aa059-e19a-488e-bced-47437b5b0c43","Type":"ContainerStarted","Data":"28325e85639a9f0850df8bfc38d505b95733c1eaf4303a0aff9d0bb238ee6def"} Oct 30 03:43:54.499816 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:54.499762 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.499733693 podStartE2EDuration="1.499733693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:54.498851776 +0000 UTC m=+3795.512129877" watchObservedRunningTime="2026-10-30 03:43:54.499733693 +0000 UTC m=+3795.513011794" Oct 30 03:43:55.119958 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:55.119922 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f99763a-20c3-4709-89a0-79576893d62a" path="/var/lib/kubelet/pods/7f99763a-20c3-4709-89a0-79576893d62a/volumes" Oct 30 03:43:56.229050 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:56.229015 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6"] Oct 30 03:43:56.229528 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:56.229248 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" podUID="341c505d-43b8-43d6-97d3-da351a8f7cef" containerName="korrel8r" containerID="cri-o://4db562d5e662c5979fb2acc107be72fa7c6cd4410880cd95d231f40ba1053a46" gracePeriod=30 Oct 30 03:43:56.230926 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:56.230904 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" Oct 30 03:43:56.461957 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:56.461930 4925 generic.go:356] "Generic (PLEG): container finished" podID="341c505d-43b8-43d6-97d3-da351a8f7cef" containerID="4db562d5e662c5979fb2acc107be72fa7c6cd4410880cd95d231f40ba1053a46" exitCode=0 Oct 30 03:43:56.462068 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:56.462006 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" event={"ID":"341c505d-43b8-43d6-97d3-da351a8f7cef","Type":"ContainerDied","Data":"4db562d5e662c5979fb2acc107be72fa7c6cd4410880cd95d231f40ba1053a46"} Oct 30 03:43:56.462068 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:56.462057 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" event={"ID":"341c505d-43b8-43d6-97d3-da351a8f7cef","Type":"ContainerDied","Data":"e33897b839a66b654cd653425e63ae43b3b0d78d7302c8d6ca4a4c9879943549"} Oct 30 03:43:56.462136 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:56.462074 4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e33897b839a66b654cd653425e63ae43b3b0d78d7302c8d6ca4a4c9879943549" Oct 30 03:43:56.472245 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:56.472227 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" Oct 30 03:43:56.555329 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:56.555241 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/341c505d-43b8-43d6-97d3-da351a8f7cef-config\" (UniqueName: \"kubernetes.io/configmap/341c505d-43b8-43d6-97d3-da351a8f7cef-config\") pod \"341c505d-43b8-43d6-97d3-da351a8f7cef\" (UID: \"341c505d-43b8-43d6-97d3-da351a8f7cef\") " Oct 30 03:43:56.555473 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:56.555341 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/341c505d-43b8-43d6-97d3-da351a8f7cef-tls\" (UniqueName: \"kubernetes.io/secret/341c505d-43b8-43d6-97d3-da351a8f7cef-tls\") pod \"341c505d-43b8-43d6-97d3-da351a8f7cef\" (UID: \"341c505d-43b8-43d6-97d3-da351a8f7cef\") " Oct 30 03:43:56.555473 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:56.555378 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/341c505d-43b8-43d6-97d3-da351a8f7cef-kube-api-access-xdln7\" (UniqueName: \"kubernetes.io/projected/341c505d-43b8-43d6-97d3-da351a8f7cef-kube-api-access-xdln7\") pod \"341c505d-43b8-43d6-97d3-da351a8f7cef\" (UID: \"341c505d-43b8-43d6-97d3-da351a8f7cef\") " Oct 30 03:43:56.555666 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:56.555641 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/341c505d-43b8-43d6-97d3-da351a8f7cef-config" pod "341c505d-43b8-43d6-97d3-da351a8f7cef" (UID: "341c505d-43b8-43d6-97d3-da351a8f7cef"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:43:56.557382 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:56.557355 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341c505d-43b8-43d6-97d3-da351a8f7cef-kube-api-access-xdln7" pod "341c505d-43b8-43d6-97d3-da351a8f7cef" (UID: "341c505d-43b8-43d6-97d3-da351a8f7cef"). InnerVolumeSpecName "kube-api-access-xdln7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:43:56.557382 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:56.557376 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/341c505d-43b8-43d6-97d3-da351a8f7cef-tls" pod "341c505d-43b8-43d6-97d3-da351a8f7cef" (UID: "341c505d-43b8-43d6-97d3-da351a8f7cef"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:43:56.656102 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:56.656069 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/341c505d-43b8-43d6-97d3-da351a8f7cef-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:56.656102 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:56.656096 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdln7\" (UniqueName: \"kubernetes.io/projected/341c505d-43b8-43d6-97d3-da351a8f7cef-kube-api-access-xdln7\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:56.656266 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:56.656113 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/341c505d-43b8-43d6-97d3-da351a8f7cef-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:43:57.466370 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:57.466286 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6" Oct 30 03:43:57.528909 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:57.528880 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6"] Oct 30 03:43:57.540740 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:57.540715 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-c6wn6"] Oct 30 03:43:59.126034 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:43:59.126000 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="341c505d-43b8-43d6-97d3-da351a8f7cef" path="/var/lib/kubelet/pods/341c505d-43b8-43d6-97d3-da351a8f7cef/volumes" Oct 30 03:44:01.736481 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:01.736438 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:01.737116 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:01.736646 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="8c7aa059-e19a-488e-bced-47437b5b0c43" containerName="otc-container" containerID="cri-o://13210c2108f849f233b240e328709b62bfaab2a70590016a8cd35683e0b644f2" gracePeriod=30 Oct 30 03:44:01.986731 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:01.986671 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:02.103218 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.103186 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-tls-certs\") pod \"8c7aa059-e19a-488e-bced-47437b5b0c43\" (UID: \"8c7aa059-e19a-488e-bced-47437b5b0c43\") " Oct 30 03:44:02.103399 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.103252 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-data-science-collector-ta-client-cert\") pod \"8c7aa059-e19a-488e-bced-47437b5b0c43\" (UID: \"8c7aa059-e19a-488e-bced-47437b5b0c43\") " Oct 30 03:44:02.103399 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.103389 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c7aa059-e19a-488e-bced-47437b5b0c43-kube-api-access-8sxqf\" (UniqueName: \"kubernetes.io/projected/8c7aa059-e19a-488e-bced-47437b5b0c43-kube-api-access-8sxqf\") pod \"8c7aa059-e19a-488e-bced-47437b5b0c43\" (UID: \"8c7aa059-e19a-488e-bced-47437b5b0c43\") " Oct 30 03:44:02.103534 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.103415 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-monitoring-tls-certs\") pod \"8c7aa059-e19a-488e-bced-47437b5b0c43\" (UID: \"8c7aa059-e19a-488e-bced-47437b5b0c43\") " Oct 30 03:44:02.103534 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.103433 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c7aa059-e19a-488e-bced-47437b5b0c43-otc-internal\" (UniqueName: \"kubernetes.io/configmap/8c7aa059-e19a-488e-bced-47437b5b0c43-otc-internal\") pod \"8c7aa059-e19a-488e-bced-47437b5b0c43\" (UID: \"8c7aa059-e19a-488e-bced-47437b5b0c43\") " Oct 30 03:44:02.103903 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.103875 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c7aa059-e19a-488e-bced-47437b5b0c43-otc-internal" pod "8c7aa059-e19a-488e-bced-47437b5b0c43" (UID: "8c7aa059-e19a-488e-bced-47437b5b0c43"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:02.105453 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.105429 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-monitoring-tls-certs" pod "8c7aa059-e19a-488e-bced-47437b5b0c43" (UID: "8c7aa059-e19a-488e-bced-47437b5b0c43"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:02.105538 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.105502 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7aa059-e19a-488e-bced-47437b5b0c43-kube-api-access-8sxqf" pod "8c7aa059-e19a-488e-bced-47437b5b0c43" (UID: "8c7aa059-e19a-488e-bced-47437b5b0c43"). InnerVolumeSpecName "kube-api-access-8sxqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:02.105700 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.105682 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-tls-certs" pod "8c7aa059-e19a-488e-bced-47437b5b0c43" (UID: "8c7aa059-e19a-488e-bced-47437b5b0c43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:02.105784 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.105704 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-data-science-collector-ta-client-cert" pod "8c7aa059-e19a-488e-bced-47437b5b0c43" (UID: "8c7aa059-e19a-488e-bced-47437b5b0c43"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:02.204106 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.204068 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:02.204106 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.204090 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:02.204106 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.204102 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sxqf\" (UniqueName: \"kubernetes.io/projected/8c7aa059-e19a-488e-bced-47437b5b0c43-kube-api-access-8sxqf\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:02.204296 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.204115 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7aa059-e19a-488e-bced-47437b5b0c43-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:02.204296 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.204140 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8c7aa059-e19a-488e-bced-47437b5b0c43-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:02.496144 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.496108 4925 generic.go:356] "Generic (PLEG): container finished" podID="8c7aa059-e19a-488e-bced-47437b5b0c43" containerID="13210c2108f849f233b240e328709b62bfaab2a70590016a8cd35683e0b644f2" exitCode=0 Oct 30 03:44:02.496336 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.496160 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:02.496336 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.496187 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"8c7aa059-e19a-488e-bced-47437b5b0c43","Type":"ContainerDied","Data":"13210c2108f849f233b240e328709b62bfaab2a70590016a8cd35683e0b644f2"} Oct 30 03:44:02.496336 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.496243 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"8c7aa059-e19a-488e-bced-47437b5b0c43","Type":"ContainerDied","Data":"28325e85639a9f0850df8bfc38d505b95733c1eaf4303a0aff9d0bb238ee6def"} Oct 30 03:44:02.496336 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.496265 4925 scope.go:122] "RemoveContainer" containerID="13210c2108f849f233b240e328709b62bfaab2a70590016a8cd35683e0b644f2" Oct 30 03:44:02.505511 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.505492 4925 scope.go:122] "RemoveContainer" containerID="13210c2108f849f233b240e328709b62bfaab2a70590016a8cd35683e0b644f2" Oct 30 03:44:02.505819 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:44:02.505791 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13210c2108f849f233b240e328709b62bfaab2a70590016a8cd35683e0b644f2\": container with ID starting with 13210c2108f849f233b240e328709b62bfaab2a70590016a8cd35683e0b644f2 not found: ID does not exist" containerID="13210c2108f849f233b240e328709b62bfaab2a70590016a8cd35683e0b644f2" Oct 30 03:44:02.505922 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.505832 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13210c2108f849f233b240e328709b62bfaab2a70590016a8cd35683e0b644f2"} err="failed to get container status \"13210c2108f849f233b240e328709b62bfaab2a70590016a8cd35683e0b644f2\": rpc error: code = NotFound desc = could not find container \"13210c2108f849f233b240e328709b62bfaab2a70590016a8cd35683e0b644f2\": container with ID starting with 13210c2108f849f233b240e328709b62bfaab2a70590016a8cd35683e0b644f2 not found: ID does not exist" Oct 30 03:44:02.550983 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.550955 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:02.555853 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.555830 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:02.597664 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.597641 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:02.598010 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.597997 4925 cpu_manager.go:414] "containerMap: removing container" podUID="341c505d-43b8-43d6-97d3-da351a8f7cef" containerName="korrel8r" Oct 30 03:44:02.598057 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.598012 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="341c505d-43b8-43d6-97d3-da351a8f7cef" containerName="korrel8r" Oct 30 03:44:02.598057 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.598024 4925 cpu_manager.go:414] "containerMap: removing container" podUID="8c7aa059-e19a-488e-bced-47437b5b0c43" containerName="otc-container" Oct 30 03:44:02.598057 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.598029 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c7aa059-e19a-488e-bced-47437b5b0c43" containerName="otc-container" Oct 30 03:44:02.598145 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.598110 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="341c505d-43b8-43d6-97d3-da351a8f7cef" containerName="korrel8r" Oct 30 03:44:02.598145 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.598117 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c7aa059-e19a-488e-bced-47437b5b0c43" containerName="otc-container" Oct 30 03:44:02.602488 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.602471 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:02.622105 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.622082 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 03:44:02.629038 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.629013 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 03:44:02.629038 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.629018 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\"" Oct 30 03:44:02.629228 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.629022 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 03:44:02.629228 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.629026 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:44:02.629915 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.629896 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:02.708439 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.708415 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"30e8450b-13f4-46ac-a8a1-6c898e477203\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:02.708567 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.708473 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"30e8450b-13f4-46ac-a8a1-6c898e477203\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:02.708567 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.708491 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/30e8450b-13f4-46ac-a8a1-6c898e477203-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"30e8450b-13f4-46ac-a8a1-6c898e477203\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:02.708694 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.708571 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cnbw\" (UniqueName: \"kubernetes.io/projected/30e8450b-13f4-46ac-a8a1-6c898e477203-kube-api-access-4cnbw\") pod \"data-science-collector-collector-0\" (UID: \"30e8450b-13f4-46ac-a8a1-6c898e477203\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:02.708694 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.708600 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"30e8450b-13f4-46ac-a8a1-6c898e477203\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:02.809302 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.809244 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"30e8450b-13f4-46ac-a8a1-6c898e477203\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:02.809302 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.809272 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/30e8450b-13f4-46ac-a8a1-6c898e477203-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"30e8450b-13f4-46ac-a8a1-6c898e477203\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:02.809302 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.809294 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cnbw\" (UniqueName: \"kubernetes.io/projected/30e8450b-13f4-46ac-a8a1-6c898e477203-kube-api-access-4cnbw\") pod \"data-science-collector-collector-0\" (UID: \"30e8450b-13f4-46ac-a8a1-6c898e477203\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:02.809686 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.809312 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"30e8450b-13f4-46ac-a8a1-6c898e477203\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:02.809686 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.809353 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"30e8450b-13f4-46ac-a8a1-6c898e477203\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:02.810024 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.810001 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/30e8450b-13f4-46ac-a8a1-6c898e477203-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"30e8450b-13f4-46ac-a8a1-6c898e477203\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:02.811715 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.811692 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"30e8450b-13f4-46ac-a8a1-6c898e477203\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:02.811862 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.811843 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"30e8450b-13f4-46ac-a8a1-6c898e477203\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:02.811939 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.811918 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"30e8450b-13f4-46ac-a8a1-6c898e477203\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:02.834351 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.834332 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cnbw\" (UniqueName: \"kubernetes.io/projected/30e8450b-13f4-46ac-a8a1-6c898e477203-kube-api-access-4cnbw\") pod \"data-science-collector-collector-0\" (UID: \"30e8450b-13f4-46ac-a8a1-6c898e477203\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:02.914436 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:02.914413 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:03.126417 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:03.121880 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c7aa059-e19a-488e-bced-47437b5b0c43" path="/var/lib/kubelet/pods/8c7aa059-e19a-488e-bced-47437b5b0c43/volumes" Oct 30 03:44:03.219274 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:03.219250 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:03.220426 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:44:03.220400 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e8450b_13f4_46ac_a8a1_6c898e477203.slice/crio-7864e9c0470bd7ee3507055c26a5785afe1e6038cabc898b71162d61e1cccdd2 WatchSource:0}: Error finding container 7864e9c0470bd7ee3507055c26a5785afe1e6038cabc898b71162d61e1cccdd2: Status 404 returned error can't find the container with id 7864e9c0470bd7ee3507055c26a5785afe1e6038cabc898b71162d61e1cccdd2 Oct 30 03:44:03.222395 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:03.222365 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:03.222463 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:03.222420 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:03.502294 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:03.502204 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"30e8450b-13f4-46ac-a8a1-6c898e477203","Type":"ContainerStarted","Data":"b3059cbbc4aaea6a10f2c8dc3e8706c68e4a268dd1f821c3b98ce138f35fdd12"} Oct 30 03:44:03.502294 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:03.502271 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"30e8450b-13f4-46ac-a8a1-6c898e477203","Type":"ContainerStarted","Data":"7864e9c0470bd7ee3507055c26a5785afe1e6038cabc898b71162d61e1cccdd2"} Oct 30 03:44:03.547879 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:03.547832 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.5478182409999999 podStartE2EDuration="1.547818241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:03.546936095 +0000 UTC m=+3804.560214197" watchObservedRunningTime="2026-10-30 03:44:03.547818241 +0000 UTC m=+3804.561096342" Oct 30 03:44:08.771916 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:08.771887 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:08.772398 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:08.772061 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="30e8450b-13f4-46ac-a8a1-6c898e477203" containerName="otc-container" containerID="cri-o://b3059cbbc4aaea6a10f2c8dc3e8706c68e4a268dd1f821c3b98ce138f35fdd12" gracePeriod=30 Oct 30 03:44:10.011841 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.011811 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:10.169604 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.169518 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e8450b-13f4-46ac-a8a1-6c898e477203-kube-api-access-4cnbw\" (UniqueName: \"kubernetes.io/projected/30e8450b-13f4-46ac-a8a1-6c898e477203-kube-api-access-4cnbw\") pod \"30e8450b-13f4-46ac-a8a1-6c898e477203\" (UID: \"30e8450b-13f4-46ac-a8a1-6c898e477203\") " Oct 30 03:44:10.169604 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.169578 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-monitoring-tls-certs\") pod \"30e8450b-13f4-46ac-a8a1-6c898e477203\" (UID: \"30e8450b-13f4-46ac-a8a1-6c898e477203\") " Oct 30 03:44:10.169862 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.169650 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-data-science-collector-ta-client-cert\") pod \"30e8450b-13f4-46ac-a8a1-6c898e477203\" (UID: \"30e8450b-13f4-46ac-a8a1-6c898e477203\") " Oct 30 03:44:10.169862 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.169686 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30e8450b-13f4-46ac-a8a1-6c898e477203-otc-internal\" (UniqueName: \"kubernetes.io/configmap/30e8450b-13f4-46ac-a8a1-6c898e477203-otc-internal\") pod \"30e8450b-13f4-46ac-a8a1-6c898e477203\" (UID: \"30e8450b-13f4-46ac-a8a1-6c898e477203\") " Oct 30 03:44:10.169862 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.169769 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-tls-certs\" (UniqueName: \"kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-tls-certs\") pod \"30e8450b-13f4-46ac-a8a1-6c898e477203\" (UID: \"30e8450b-13f4-46ac-a8a1-6c898e477203\") " Oct 30 03:44:10.170410 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.170311 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e8450b-13f4-46ac-a8a1-6c898e477203-otc-internal" pod "30e8450b-13f4-46ac-a8a1-6c898e477203" (UID: "30e8450b-13f4-46ac-a8a1-6c898e477203"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:10.172045 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.172009 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e8450b-13f4-46ac-a8a1-6c898e477203-kube-api-access-4cnbw" pod "30e8450b-13f4-46ac-a8a1-6c898e477203" (UID: "30e8450b-13f4-46ac-a8a1-6c898e477203"). InnerVolumeSpecName "kube-api-access-4cnbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:10.172172 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.172152 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-data-science-collector-ta-client-cert" pod "30e8450b-13f4-46ac-a8a1-6c898e477203" (UID: "30e8450b-13f4-46ac-a8a1-6c898e477203"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:10.172230 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.172215 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-tls-certs" pod "30e8450b-13f4-46ac-a8a1-6c898e477203" (UID: "30e8450b-13f4-46ac-a8a1-6c898e477203"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:10.172324 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.172305 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-monitoring-tls-certs" pod "30e8450b-13f4-46ac-a8a1-6c898e477203" (UID: "30e8450b-13f4-46ac-a8a1-6c898e477203"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:10.270469 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.270445 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:10.270469 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.270468 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/30e8450b-13f4-46ac-a8a1-6c898e477203-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:10.270628 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.270477 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:10.270628 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.270487 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cnbw\" (UniqueName: \"kubernetes.io/projected/30e8450b-13f4-46ac-a8a1-6c898e477203-kube-api-access-4cnbw\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:10.270628 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.270495 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/30e8450b-13f4-46ac-a8a1-6c898e477203-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:10.539974 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.539897 4925 generic.go:356] "Generic (PLEG): container finished" podID="30e8450b-13f4-46ac-a8a1-6c898e477203" containerID="b3059cbbc4aaea6a10f2c8dc3e8706c68e4a268dd1f821c3b98ce138f35fdd12" exitCode=0 Oct 30 03:44:10.539974 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.539939 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:10.540185 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.539937 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"30e8450b-13f4-46ac-a8a1-6c898e477203","Type":"ContainerDied","Data":"b3059cbbc4aaea6a10f2c8dc3e8706c68e4a268dd1f821c3b98ce138f35fdd12"} Oct 30 03:44:10.540185 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.540064 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"30e8450b-13f4-46ac-a8a1-6c898e477203","Type":"ContainerDied","Data":"7864e9c0470bd7ee3507055c26a5785afe1e6038cabc898b71162d61e1cccdd2"} Oct 30 03:44:10.540185 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.540094 4925 scope.go:122] "RemoveContainer" containerID="b3059cbbc4aaea6a10f2c8dc3e8706c68e4a268dd1f821c3b98ce138f35fdd12" Oct 30 03:44:10.548977 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.548960 4925 scope.go:122] "RemoveContainer" containerID="b3059cbbc4aaea6a10f2c8dc3e8706c68e4a268dd1f821c3b98ce138f35fdd12" Oct 30 03:44:10.549235 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:44:10.549215 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3059cbbc4aaea6a10f2c8dc3e8706c68e4a268dd1f821c3b98ce138f35fdd12\": container with ID starting with b3059cbbc4aaea6a10f2c8dc3e8706c68e4a268dd1f821c3b98ce138f35fdd12 not found: ID does not exist" containerID="b3059cbbc4aaea6a10f2c8dc3e8706c68e4a268dd1f821c3b98ce138f35fdd12" Oct 30 03:44:10.549300 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.549241 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3059cbbc4aaea6a10f2c8dc3e8706c68e4a268dd1f821c3b98ce138f35fdd12"} err="failed to get container status \"b3059cbbc4aaea6a10f2c8dc3e8706c68e4a268dd1f821c3b98ce138f35fdd12\": rpc error: code = NotFound desc = could not find container \"b3059cbbc4aaea6a10f2c8dc3e8706c68e4a268dd1f821c3b98ce138f35fdd12\": container with ID starting with b3059cbbc4aaea6a10f2c8dc3e8706c68e4a268dd1f821c3b98ce138f35fdd12 not found: ID does not exist" Oct 30 03:44:10.601998 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.601974 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:10.613124 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.613099 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:10.649056 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.649033 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:10.649438 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.649426 4925 cpu_manager.go:414] "containerMap: removing container" podUID="30e8450b-13f4-46ac-a8a1-6c898e477203" containerName="otc-container" Oct 30 03:44:10.649482 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.649439 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e8450b-13f4-46ac-a8a1-6c898e477203" containerName="otc-container" Oct 30 03:44:10.649512 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.649500 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="30e8450b-13f4-46ac-a8a1-6c898e477203" containerName="otc-container" Oct 30 03:44:10.654011 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.653993 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:10.669929 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.669912 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:44:10.672899 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.672880 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-2dfe8e70\"" Oct 30 03:44:10.675257 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.675237 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:10.770880 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.770849 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 03:44:10.771034 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.770997 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\"" Oct 30 03:44:10.771086 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.771049 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 03:44:10.774424 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.774408 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:10.774484 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.774466 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:10.774540 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.774524 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:10.774578 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.774558 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-562df\" (UniqueName: \"kubernetes.io/projected/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-kube-api-access-562df\") pod \"data-science-collector-collector-0\" (UID: \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:10.774625 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.774594 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:10.875600 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.875561 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:10.875600 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.875604 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:10.875889 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.875637 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-562df\" (UniqueName: \"kubernetes.io/projected/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-kube-api-access-562df\") pod \"data-science-collector-collector-0\" (UID: \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:10.875889 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.875682 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:10.875889 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.875727 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:10.876560 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.876527 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:10.878196 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.878165 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:10.878306 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.878288 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:10.878369 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.878335 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:10.900007 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.899978 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-562df\" (UniqueName: \"kubernetes.io/projected/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-kube-api-access-562df\") pod \"data-science-collector-collector-0\" (UID: \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:10.965744 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:10.965718 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:11.119739 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:11.119708 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30e8450b-13f4-46ac-a8a1-6c898e477203" path="/var/lib/kubelet/pods/30e8450b-13f4-46ac-a8a1-6c898e477203/volumes" Oct 30 03:44:11.272533 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:11.272510 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:11.273588 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:44:11.273563 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e4bdc6_9bcd_4882_9ce4_a259668cf4f8.slice/crio-7283bf6f061e18c4dd9d72cd7f0b36c9627e52747df25e519ee686c5aaf4ed9b WatchSource:0}: Error finding container 7283bf6f061e18c4dd9d72cd7f0b36c9627e52747df25e519ee686c5aaf4ed9b: Status 404 returned error can't find the container with id 7283bf6f061e18c4dd9d72cd7f0b36c9627e52747df25e519ee686c5aaf4ed9b Oct 30 03:44:11.275785 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:11.275743 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:11.275848 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:11.275812 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:11.548039 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:11.547956 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8","Type":"ContainerStarted","Data":"58d7d1d6ca5032643a7a595d92c69a352be197db847784080ed7ad3c29c9fb86"} Oct 30 03:44:11.548039 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:11.548014 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8","Type":"ContainerStarted","Data":"7283bf6f061e18c4dd9d72cd7f0b36c9627e52747df25e519ee686c5aaf4ed9b"} Oct 30 03:44:11.592433 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:11.592382 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.592368457 podStartE2EDuration="1.592368457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:11.590861788 +0000 UTC m=+3812.604139914" watchObservedRunningTime="2026-10-30 03:44:11.592368457 +0000 UTC m=+3812.605646604" Oct 30 03:44:17.830288 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:17.830253 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:17.830649 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:17.830462 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="88e4bdc6-9bcd-4882-9ce4-a259668cf4f8" containerName="otc-container" containerID="cri-o://58d7d1d6ca5032643a7a595d92c69a352be197db847784080ed7ad3c29c9fb86" gracePeriod=30 Oct 30 03:44:18.081298 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.081240 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:18.163455 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:44:18.163422 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:44:18.240779 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.240738 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-otc-internal\" (UniqueName: \"kubernetes.io/configmap/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-otc-internal\") pod \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\" (UID: \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\") " Oct 30 03:44:18.240918 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.240801 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-monitoring-tls-certs\") pod \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\" (UID: \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\") " Oct 30 03:44:18.240918 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.240898 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-tls-certs\") pod \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\" (UID: \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\") " Oct 30 03:44:18.241028 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.240929 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-data-science-collector-ta-client-cert\") pod \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\" (UID: \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\") " Oct 30 03:44:18.241028 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.240962 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-kube-api-access-562df\" (UniqueName: \"kubernetes.io/projected/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-kube-api-access-562df\") pod \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\" (UID: \"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8\") " Oct 30 03:44:18.241141 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.241089 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-otc-internal" pod "88e4bdc6-9bcd-4882-9ce4-a259668cf4f8" (UID: "88e4bdc6-9bcd-4882-9ce4-a259668cf4f8"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:18.241216 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.241200 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:18.242868 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.242842 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-tls-certs" pod "88e4bdc6-9bcd-4882-9ce4-a259668cf4f8" (UID: "88e4bdc6-9bcd-4882-9ce4-a259668cf4f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:18.242972 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.242863 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-data-science-collector-ta-client-cert" pod "88e4bdc6-9bcd-4882-9ce4-a259668cf4f8" (UID: "88e4bdc6-9bcd-4882-9ce4-a259668cf4f8"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:18.242972 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.242941 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-kube-api-access-562df" pod "88e4bdc6-9bcd-4882-9ce4-a259668cf4f8" (UID: "88e4bdc6-9bcd-4882-9ce4-a259668cf4f8"). InnerVolumeSpecName "kube-api-access-562df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:18.242972 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.242964 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-monitoring-tls-certs" pod "88e4bdc6-9bcd-4882-9ce4-a259668cf4f8" (UID: "88e4bdc6-9bcd-4882-9ce4-a259668cf4f8"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:18.341860 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.341800 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:18.341860 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.341823 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:18.341860 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.341833 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-562df\" (UniqueName: \"kubernetes.io/projected/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-kube-api-access-562df\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:18.341860 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.341842 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:18.584191 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.584155 4925 generic.go:356] "Generic (PLEG): container finished" podID="88e4bdc6-9bcd-4882-9ce4-a259668cf4f8" containerID="58d7d1d6ca5032643a7a595d92c69a352be197db847784080ed7ad3c29c9fb86" exitCode=0 Oct 30 03:44:18.584326 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.584209 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:18.584326 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.584233 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8","Type":"ContainerDied","Data":"58d7d1d6ca5032643a7a595d92c69a352be197db847784080ed7ad3c29c9fb86"} Oct 30 03:44:18.584326 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.584287 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"88e4bdc6-9bcd-4882-9ce4-a259668cf4f8","Type":"ContainerDied","Data":"7283bf6f061e18c4dd9d72cd7f0b36c9627e52747df25e519ee686c5aaf4ed9b"} Oct 30 03:44:18.584326 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.584308 4925 scope.go:122] "RemoveContainer" containerID="58d7d1d6ca5032643a7a595d92c69a352be197db847784080ed7ad3c29c9fb86" Oct 30 03:44:18.594552 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.594538 4925 scope.go:122] "RemoveContainer" containerID="58d7d1d6ca5032643a7a595d92c69a352be197db847784080ed7ad3c29c9fb86" Oct 30 03:44:18.594761 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:44:18.594732 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58d7d1d6ca5032643a7a595d92c69a352be197db847784080ed7ad3c29c9fb86\": container with ID starting with 58d7d1d6ca5032643a7a595d92c69a352be197db847784080ed7ad3c29c9fb86 not found: ID does not exist" containerID="58d7d1d6ca5032643a7a595d92c69a352be197db847784080ed7ad3c29c9fb86" Oct 30 03:44:18.594818 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.594778 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d7d1d6ca5032643a7a595d92c69a352be197db847784080ed7ad3c29c9fb86"} err="failed to get container status \"58d7d1d6ca5032643a7a595d92c69a352be197db847784080ed7ad3c29c9fb86\": rpc error: code = NotFound desc = could not find container \"58d7d1d6ca5032643a7a595d92c69a352be197db847784080ed7ad3c29c9fb86\": container with ID starting with 58d7d1d6ca5032643a7a595d92c69a352be197db847784080ed7ad3c29c9fb86 not found: ID does not exist" Oct 30 03:44:18.642167 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.642142 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:18.654705 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.654686 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:18.687568 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.687548 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:18.687925 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.687912 4925 cpu_manager.go:414] "containerMap: removing container" podUID="88e4bdc6-9bcd-4882-9ce4-a259668cf4f8" containerName="otc-container" Oct 30 03:44:18.687925 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.687926 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e4bdc6-9bcd-4882-9ce4-a259668cf4f8" containerName="otc-container" Oct 30 03:44:18.688020 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.687985 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e4bdc6-9bcd-4882-9ce4-a259668cf4f8" containerName="otc-container" Oct 30 03:44:18.692361 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.692336 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:18.708813 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.708783 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 03:44:18.708925 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.708807 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 03:44:18.708925 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.708877 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:44:18.709025 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.708928 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 03:44:18.709025 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.709009 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\"" Oct 30 03:44:18.722534 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.722503 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:18.846273 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.846195 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:18.846273 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.846251 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:18.846637 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.846377 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:18.846637 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.846407 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f68zk\" (UniqueName: \"kubernetes.io/projected/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-kube-api-access-f68zk\") pod \"data-science-collector-collector-0\" (UID: \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:18.846637 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.846432 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:18.947527 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.947494 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:18.947527 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.947529 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f68zk\" (UniqueName: \"kubernetes.io/projected/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-kube-api-access-f68zk\") pod \"data-science-collector-collector-0\" (UID: \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:18.947776 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.947553 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:18.947776 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.947591 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:18.947776 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.947618 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:18.948367 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.948345 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:18.950226 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.950205 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:18.950316 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.950259 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:18.950357 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.950339 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:18.971555 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:18.971533 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f68zk\" (UniqueName: \"kubernetes.io/projected/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-kube-api-access-f68zk\") pod \"data-science-collector-collector-0\" (UID: \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:19.004369 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:19.004332 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:19.119129 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:19.119067 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e4bdc6-9bcd-4882-9ce4-a259668cf4f8" path="/var/lib/kubelet/pods/88e4bdc6-9bcd-4882-9ce4-a259668cf4f8/volumes" Oct 30 03:44:19.310840 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:19.310816 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:19.312129 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:44:19.312101 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a87d846_587d_4eb2_b2c0_8c41bd6149a7.slice/crio-e81fff0d4b1d2631055d16cf5af1ab1ea6f3a79fcb95c0557b64dd4d7181649a WatchSource:0}: Error finding container e81fff0d4b1d2631055d16cf5af1ab1ea6f3a79fcb95c0557b64dd4d7181649a: Status 404 returned error can't find the container with id e81fff0d4b1d2631055d16cf5af1ab1ea6f3a79fcb95c0557b64dd4d7181649a Oct 30 03:44:19.314111 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:19.314084 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:19.314194 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:19.314137 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:19.599916 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:19.599879 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5a87d846-587d-4eb2-b2c0-8c41bd6149a7","Type":"ContainerStarted","Data":"c57616883dcf35da61672827446d8d2d926c906ec5e834971168a9fb05f671df"} Oct 30 03:44:19.600075 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:19.599971 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5a87d846-587d-4eb2-b2c0-8c41bd6149a7","Type":"ContainerStarted","Data":"e81fff0d4b1d2631055d16cf5af1ab1ea6f3a79fcb95c0557b64dd4d7181649a"} Oct 30 03:44:19.644637 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:19.644547 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.644533601 podStartE2EDuration="1.644533601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:19.642250292 +0000 UTC m=+3820.655528394" watchObservedRunningTime="2026-10-30 03:44:19.644533601 +0000 UTC m=+3820.657811702" Oct 30 03:44:23.465369 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:23.465291 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts"] Oct 30 03:44:23.468921 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:23.468900 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" Oct 30 03:44:23.491033 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:23.491001 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts"] Oct 30 03:44:23.583829 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:23.583795 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-config\") pod \"korrel8r-66d6f7f696-2j7ts\" (UID: \"e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" Oct 30 03:44:23.583829 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:23.583828 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwdhc\" (UniqueName: \"kubernetes.io/projected/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-kube-api-access-dwdhc\") pod \"korrel8r-66d6f7f696-2j7ts\" (UID: \"e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" Oct 30 03:44:23.584009 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:23.583918 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-tls\") pod \"korrel8r-66d6f7f696-2j7ts\" (UID: \"e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" Oct 30 03:44:23.684939 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:23.684915 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-config\") pod \"korrel8r-66d6f7f696-2j7ts\" (UID: \"e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" Oct 30 03:44:23.685066 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:23.684945 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwdhc\" (UniqueName: \"kubernetes.io/projected/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-kube-api-access-dwdhc\") pod \"korrel8r-66d6f7f696-2j7ts\" (UID: \"e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" Oct 30 03:44:23.685066 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:23.684993 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-tls\") pod \"korrel8r-66d6f7f696-2j7ts\" (UID: \"e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" Oct 30 03:44:23.685676 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:23.685654 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-config\") pod \"korrel8r-66d6f7f696-2j7ts\" (UID: \"e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" Oct 30 03:44:23.687370 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:23.687348 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-tls\") pod \"korrel8r-66d6f7f696-2j7ts\" (UID: \"e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" Oct 30 03:44:23.710629 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:23.710610 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwdhc\" (UniqueName: \"kubernetes.io/projected/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-kube-api-access-dwdhc\") pod \"korrel8r-66d6f7f696-2j7ts\" (UID: \"e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" Oct 30 03:44:23.781542 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:23.781491 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" Oct 30 03:44:24.078106 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:24.078083 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts"] Oct 30 03:44:24.079976 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:44:24.079948 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e4a9e8_6bba_4c28_ade0_2f8fae497b3c.slice/crio-6f1dc6eefd6d7aa1c82b8ee5489df17e8d157279ddbb0e3d89d68301269c130e WatchSource:0}: Error finding container 6f1dc6eefd6d7aa1c82b8ee5489df17e8d157279ddbb0e3d89d68301269c130e: Status 404 returned error can't find the container with id 6f1dc6eefd6d7aa1c82b8ee5489df17e8d157279ddbb0e3d89d68301269c130e Oct 30 03:44:24.628657 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:24.628624 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" event={"ID":"e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c","Type":"ContainerStarted","Data":"6d9002f67b8adb7951e0e0c94a76ffe43dea419696485ced9fff1cdf087207e8"} Oct 30 03:44:24.629052 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:24.628673 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" event={"ID":"e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c","Type":"ContainerStarted","Data":"6f1dc6eefd6d7aa1c82b8ee5489df17e8d157279ddbb0e3d89d68301269c130e"} Oct 30 03:44:24.629052 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:24.628703 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" Oct 30 03:44:24.671347 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:24.671303 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" podStartSLOduration=1.671290236 podStartE2EDuration="1.671290236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:24.670068094 +0000 UTC m=+3825.683346198" watchObservedRunningTime="2026-10-30 03:44:24.671290236 +0000 UTC m=+3825.684568336" Oct 30 03:44:28.895328 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:28.895294 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:28.895821 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:28.895515 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="5a87d846-587d-4eb2-b2c0-8c41bd6149a7" containerName="otc-container" containerID="cri-o://c57616883dcf35da61672827446d8d2d926c906ec5e834971168a9fb05f671df" gracePeriod=30 Oct 30 03:44:29.144349 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.144320 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:29.229545 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.229473 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-kube-api-access-f68zk\" (UniqueName: \"kubernetes.io/projected/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-kube-api-access-f68zk\") pod \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\" (UID: \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\") " Oct 30 03:44:29.229545 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.229513 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-data-science-collector-ta-client-cert\") pod \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\" (UID: \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\") " Oct 30 03:44:29.229708 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.229557 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-tls-certs\") pod \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\" (UID: \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\") " Oct 30 03:44:29.229708 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.229637 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-otc-internal\") pod \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\" (UID: \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\") " Oct 30 03:44:29.229829 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.229707 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-monitoring-tls-certs\") pod \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\" (UID: \"5a87d846-587d-4eb2-b2c0-8c41bd6149a7\") " Oct 30 03:44:29.230099 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.230040 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-otc-internal" pod "5a87d846-587d-4eb2-b2c0-8c41bd6149a7" (UID: "5a87d846-587d-4eb2-b2c0-8c41bd6149a7"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:29.231822 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.231795 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-data-science-collector-ta-client-cert" pod "5a87d846-587d-4eb2-b2c0-8c41bd6149a7" (UID: "5a87d846-587d-4eb2-b2c0-8c41bd6149a7"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:29.231966 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.231945 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-monitoring-tls-certs" pod "5a87d846-587d-4eb2-b2c0-8c41bd6149a7" (UID: "5a87d846-587d-4eb2-b2c0-8c41bd6149a7"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:29.232162 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.232142 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-tls-certs" pod "5a87d846-587d-4eb2-b2c0-8c41bd6149a7" (UID: "5a87d846-587d-4eb2-b2c0-8c41bd6149a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:29.232265 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.232167 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-kube-api-access-f68zk" pod "5a87d846-587d-4eb2-b2c0-8c41bd6149a7" (UID: "5a87d846-587d-4eb2-b2c0-8c41bd6149a7"). InnerVolumeSpecName "kube-api-access-f68zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:29.330263 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.330235 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:29.330263 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.330258 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f68zk\" (UniqueName: \"kubernetes.io/projected/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-kube-api-access-f68zk\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:29.330263 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.330268 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:29.330596 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.330278 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:29.330596 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.330287 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5a87d846-587d-4eb2-b2c0-8c41bd6149a7-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:29.656049 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.656018 4925 generic.go:356] "Generic (PLEG): container finished" podID="5a87d846-587d-4eb2-b2c0-8c41bd6149a7" containerID="c57616883dcf35da61672827446d8d2d926c906ec5e834971168a9fb05f671df" exitCode=0 Oct 30 03:44:29.656204 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.656066 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:29.656204 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.656101 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5a87d846-587d-4eb2-b2c0-8c41bd6149a7","Type":"ContainerDied","Data":"c57616883dcf35da61672827446d8d2d926c906ec5e834971168a9fb05f671df"} Oct 30 03:44:29.656204 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.656154 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5a87d846-587d-4eb2-b2c0-8c41bd6149a7","Type":"ContainerDied","Data":"e81fff0d4b1d2631055d16cf5af1ab1ea6f3a79fcb95c0557b64dd4d7181649a"} Oct 30 03:44:29.656204 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.656177 4925 scope.go:122] "RemoveContainer" containerID="c57616883dcf35da61672827446d8d2d926c906ec5e834971168a9fb05f671df" Oct 30 03:44:29.665280 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.665264 4925 scope.go:122] "RemoveContainer" containerID="c57616883dcf35da61672827446d8d2d926c906ec5e834971168a9fb05f671df" Oct 30 03:44:29.665554 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:44:29.665535 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c57616883dcf35da61672827446d8d2d926c906ec5e834971168a9fb05f671df\": container with ID starting with c57616883dcf35da61672827446d8d2d926c906ec5e834971168a9fb05f671df not found: ID does not exist" containerID="c57616883dcf35da61672827446d8d2d926c906ec5e834971168a9fb05f671df" Oct 30 03:44:29.665622 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.665561 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57616883dcf35da61672827446d8d2d926c906ec5e834971168a9fb05f671df"} err="failed to get container status \"c57616883dcf35da61672827446d8d2d926c906ec5e834971168a9fb05f671df\": rpc error: code = NotFound desc = could not find container \"c57616883dcf35da61672827446d8d2d926c906ec5e834971168a9fb05f671df\": container with ID starting with c57616883dcf35da61672827446d8d2d926c906ec5e834971168a9fb05f671df not found: ID does not exist" Oct 30 03:44:29.719656 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.719635 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:29.723985 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.723966 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:29.763415 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.763389 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:29.763776 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.763763 4925 cpu_manager.go:414] "containerMap: removing container" podUID="5a87d846-587d-4eb2-b2c0-8c41bd6149a7" containerName="otc-container" Oct 30 03:44:29.763823 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.763778 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a87d846-587d-4eb2-b2c0-8c41bd6149a7" containerName="otc-container" Oct 30 03:44:29.763854 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.763841 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a87d846-587d-4eb2-b2c0-8c41bd6149a7" containerName="otc-container" Oct 30 03:44:29.768464 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.768444 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:29.790311 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.790291 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:44:29.790430 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.790354 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 03:44:29.790430 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.790362 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 03:44:29.790430 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.790414 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\"" Oct 30 03:44:29.791057 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.791036 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:29.797923 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.797904 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-45fbfa60\"" Oct 30 03:44:29.832726 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.832702 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:29.832842 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.832735 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:29.832842 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.832828 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:29.832916 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.832899 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7s98\" (UniqueName: \"kubernetes.io/projected/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-kube-api-access-l7s98\") pod \"data-science-collector-collector-0\" (UID: \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:29.832953 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.832920 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:29.933377 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.933301 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:29.933377 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.933344 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:29.933879 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.933388 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7s98\" (UniqueName: \"kubernetes.io/projected/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-kube-api-access-l7s98\") pod \"data-science-collector-collector-0\" (UID: \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:29.933879 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.933416 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:29.933879 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.933502 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:29.934224 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.934205 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:29.935765 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.935736 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:29.935937 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.935921 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:29.935974 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.935938 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:29.957598 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:29.957578 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7s98\" (UniqueName: \"kubernetes.io/projected/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-kube-api-access-l7s98\") pod \"data-science-collector-collector-0\" (UID: \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:30.082495 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:30.082461 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:30.387051 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:30.387027 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:30.389184 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:44:30.389154 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b50ab9_a519_4227_84ea_91d8eb83ed2b.slice/crio-cf7a3d34a255462e946f69abcebce39bafd4201b91becaa3f09bb8013a07c95b WatchSource:0}: Error finding container cf7a3d34a255462e946f69abcebce39bafd4201b91becaa3f09bb8013a07c95b: Status 404 returned error can't find the container with id cf7a3d34a255462e946f69abcebce39bafd4201b91becaa3f09bb8013a07c95b Oct 30 03:44:30.391201 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:30.391173 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:30.391266 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:30.391230 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:30.664507 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:30.664421 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c6b50ab9-a519-4227-84ea-91d8eb83ed2b","Type":"ContainerStarted","Data":"8b84a5fbe3532232b272315e48cef936c01e6376c2bcb6d1cfb999e74f03de94"} Oct 30 03:44:30.664648 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:30.664509 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c6b50ab9-a519-4227-84ea-91d8eb83ed2b","Type":"ContainerStarted","Data":"cf7a3d34a255462e946f69abcebce39bafd4201b91becaa3f09bb8013a07c95b"} Oct 30 03:44:30.709913 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:30.709867 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.709853402 podStartE2EDuration="1.709853402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:30.708914395 +0000 UTC m=+3831.722192510" watchObservedRunningTime="2026-10-30 03:44:30.709853402 +0000 UTC m=+3831.723131506" Oct 30 03:44:31.119857 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:31.119826 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a87d846-587d-4eb2-b2c0-8c41bd6149a7" path="/var/lib/kubelet/pods/5a87d846-587d-4eb2-b2c0-8c41bd6149a7/volumes" Oct 30 03:44:33.265426 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.265393 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts"] Oct 30 03:44:33.265842 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.265562 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" podUID="e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c" containerName="korrel8r" containerID="cri-o://6d9002f67b8adb7951e0e0c94a76ffe43dea419696485ced9fff1cdf087207e8" gracePeriod=30 Oct 30 03:44:33.267179 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.267158 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" Oct 30 03:44:33.513728 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.513705 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" Oct 30 03:44:33.560846 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.560776 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-kube-api-access-dwdhc\" (UniqueName: \"kubernetes.io/projected/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-kube-api-access-dwdhc\") pod \"e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c\" (UID: \"e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c\") " Oct 30 03:44:33.560846 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.560805 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-tls\" (UniqueName: \"kubernetes.io/secret/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-tls\") pod \"e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c\" (UID: \"e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c\") " Oct 30 03:44:33.561007 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.560871 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-config\" (UniqueName: \"kubernetes.io/configmap/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-config\") pod \"e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c\" (UID: \"e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c\") " Oct 30 03:44:33.561349 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.561325 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-config" pod "e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c" (UID: "e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:33.562805 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.562784 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-tls" pod "e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c" (UID: "e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:33.562894 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.562865 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-kube-api-access-dwdhc" pod "e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c" (UID: "e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c"). InnerVolumeSpecName "kube-api-access-dwdhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:33.662064 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.662039 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:33.662064 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.662061 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwdhc\" (UniqueName: \"kubernetes.io/projected/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-kube-api-access-dwdhc\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:33.662211 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.662069 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:33.679999 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.679971 4925 generic.go:356] "Generic (PLEG): container finished" podID="e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c" containerID="6d9002f67b8adb7951e0e0c94a76ffe43dea419696485ced9fff1cdf087207e8" exitCode=0 Oct 30 03:44:33.680104 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.680008 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" Oct 30 03:44:33.680104 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.680050 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" event={"ID":"e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c","Type":"ContainerDied","Data":"6d9002f67b8adb7951e0e0c94a76ffe43dea419696485ced9fff1cdf087207e8"} Oct 30 03:44:33.680104 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.680100 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts" event={"ID":"e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c","Type":"ContainerDied","Data":"6f1dc6eefd6d7aa1c82b8ee5489df17e8d157279ddbb0e3d89d68301269c130e"} Oct 30 03:44:33.680206 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.680124 4925 scope.go:122] "RemoveContainer" containerID="6d9002f67b8adb7951e0e0c94a76ffe43dea419696485ced9fff1cdf087207e8" Oct 30 03:44:33.690164 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.690146 4925 scope.go:122] "RemoveContainer" containerID="6d9002f67b8adb7951e0e0c94a76ffe43dea419696485ced9fff1cdf087207e8" Oct 30 03:44:33.690418 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:44:33.690400 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d9002f67b8adb7951e0e0c94a76ffe43dea419696485ced9fff1cdf087207e8\": container with ID starting with 6d9002f67b8adb7951e0e0c94a76ffe43dea419696485ced9fff1cdf087207e8 not found: ID does not exist" containerID="6d9002f67b8adb7951e0e0c94a76ffe43dea419696485ced9fff1cdf087207e8" Oct 30 03:44:33.690487 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.690424 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9002f67b8adb7951e0e0c94a76ffe43dea419696485ced9fff1cdf087207e8"} err="failed to get container status \"6d9002f67b8adb7951e0e0c94a76ffe43dea419696485ced9fff1cdf087207e8\": rpc error: code = NotFound desc = could not find container \"6d9002f67b8adb7951e0e0c94a76ffe43dea419696485ced9fff1cdf087207e8\": container with ID starting with 6d9002f67b8adb7951e0e0c94a76ffe43dea419696485ced9fff1cdf087207e8 not found: ID does not exist" Oct 30 03:44:33.734559 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.734536 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts"] Oct 30 03:44:33.738344 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:33.738325 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-2j7ts"] Oct 30 03:44:35.120085 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:35.120051 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c" path="/var/lib/kubelet/pods/e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c/volumes" Oct 30 03:44:36.931321 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:36.931187 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:36.931788 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:36.931680 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="c6b50ab9-a519-4227-84ea-91d8eb83ed2b" containerName="otc-container" containerID="cri-o://8b84a5fbe3532232b272315e48cef936c01e6376c2bcb6d1cfb999e74f03de94" gracePeriod=30 Oct 30 03:44:37.192520 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.192459 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:37.290004 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.289975 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-tls-certs\") pod \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\" (UID: \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\") " Oct 30 03:44:37.290180 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.290029 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-otc-internal\" (UniqueName: \"kubernetes.io/configmap/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-otc-internal\") pod \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\" (UID: \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\") " Oct 30 03:44:37.290180 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.290066 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-kube-api-access-l7s98\" (UniqueName: \"kubernetes.io/projected/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-kube-api-access-l7s98\") pod \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\" (UID: \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\") " Oct 30 03:44:37.290180 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.290098 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-data-science-collector-ta-client-cert\") pod \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\" (UID: \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\") " Oct 30 03:44:37.290180 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.290119 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-monitoring-tls-certs\") pod \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\" (UID: \"c6b50ab9-a519-4227-84ea-91d8eb83ed2b\") " Oct 30 03:44:37.290427 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.290404 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-otc-internal" pod "c6b50ab9-a519-4227-84ea-91d8eb83ed2b" (UID: "c6b50ab9-a519-4227-84ea-91d8eb83ed2b"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:37.292143 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.292116 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-data-science-collector-ta-client-cert" pod "c6b50ab9-a519-4227-84ea-91d8eb83ed2b" (UID: "c6b50ab9-a519-4227-84ea-91d8eb83ed2b"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:37.292260 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.292147 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-kube-api-access-l7s98" pod "c6b50ab9-a519-4227-84ea-91d8eb83ed2b" (UID: "c6b50ab9-a519-4227-84ea-91d8eb83ed2b"). InnerVolumeSpecName "kube-api-access-l7s98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:37.292260 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.292204 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-tls-certs" pod "c6b50ab9-a519-4227-84ea-91d8eb83ed2b" (UID: "c6b50ab9-a519-4227-84ea-91d8eb83ed2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:37.292260 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.292236 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-monitoring-tls-certs" pod "c6b50ab9-a519-4227-84ea-91d8eb83ed2b" (UID: "c6b50ab9-a519-4227-84ea-91d8eb83ed2b"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:37.391251 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.391217 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:37.391251 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.391243 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:37.391251 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.391252 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7s98\" (UniqueName: \"kubernetes.io/projected/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-kube-api-access-l7s98\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:37.391481 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.391262 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:37.391481 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.391271 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6b50ab9-a519-4227-84ea-91d8eb83ed2b-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:37.702386 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.702351 4925 generic.go:356] "Generic (PLEG): container finished" podID="c6b50ab9-a519-4227-84ea-91d8eb83ed2b" containerID="8b84a5fbe3532232b272315e48cef936c01e6376c2bcb6d1cfb999e74f03de94" exitCode=0 Oct 30 03:44:37.702555 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.702397 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:37.702555 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.702431 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c6b50ab9-a519-4227-84ea-91d8eb83ed2b","Type":"ContainerDied","Data":"8b84a5fbe3532232b272315e48cef936c01e6376c2bcb6d1cfb999e74f03de94"} Oct 30 03:44:37.702555 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.702483 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c6b50ab9-a519-4227-84ea-91d8eb83ed2b","Type":"ContainerDied","Data":"cf7a3d34a255462e946f69abcebce39bafd4201b91becaa3f09bb8013a07c95b"} Oct 30 03:44:37.702555 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.702504 4925 scope.go:122] "RemoveContainer" containerID="8b84a5fbe3532232b272315e48cef936c01e6376c2bcb6d1cfb999e74f03de94" Oct 30 03:44:37.712087 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.712069 4925 scope.go:122] "RemoveContainer" containerID="8b84a5fbe3532232b272315e48cef936c01e6376c2bcb6d1cfb999e74f03de94" Oct 30 03:44:37.712349 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:44:37.712332 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b84a5fbe3532232b272315e48cef936c01e6376c2bcb6d1cfb999e74f03de94\": container with ID starting with 8b84a5fbe3532232b272315e48cef936c01e6376c2bcb6d1cfb999e74f03de94 not found: ID does not exist" containerID="8b84a5fbe3532232b272315e48cef936c01e6376c2bcb6d1cfb999e74f03de94" Oct 30 03:44:37.712390 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.712357 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b84a5fbe3532232b272315e48cef936c01e6376c2bcb6d1cfb999e74f03de94"} err="failed to get container status \"8b84a5fbe3532232b272315e48cef936c01e6376c2bcb6d1cfb999e74f03de94\": rpc error: code = NotFound desc = could not find container \"8b84a5fbe3532232b272315e48cef936c01e6376c2bcb6d1cfb999e74f03de94\": container with ID starting with 8b84a5fbe3532232b272315e48cef936c01e6376c2bcb6d1cfb999e74f03de94 not found: ID does not exist" Oct 30 03:44:37.755962 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.755942 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:37.767980 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.767958 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:37.825247 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.825219 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:37.825592 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.825579 4925 cpu_manager.go:414] "containerMap: removing container" podUID="e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c" containerName="korrel8r" Oct 30 03:44:37.825644 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.825594 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c" containerName="korrel8r" Oct 30 03:44:37.825644 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.825612 4925 cpu_manager.go:414] "containerMap: removing container" podUID="c6b50ab9-a519-4227-84ea-91d8eb83ed2b" containerName="otc-container" Oct 30 03:44:37.825644 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.825617 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6b50ab9-a519-4227-84ea-91d8eb83ed2b" containerName="otc-container" Oct 30 03:44:37.825731 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.825694 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3e4a9e8-6bba-4c28-ade0-2f8fae497b3c" containerName="korrel8r" Oct 30 03:44:37.825731 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.825704 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6b50ab9-a519-4227-84ea-91d8eb83ed2b" containerName="otc-container" Oct 30 03:44:37.830139 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.830124 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:37.859130 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.859110 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:44:37.859130 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.859110 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 03:44:37.859313 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.859300 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 03:44:37.859371 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.859350 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\"" Oct 30 03:44:37.859547 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.859527 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 03:44:37.869675 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.869653 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:37.895411 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.895382 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"636b68e8-0b60-4662-a5f2-ac20c01c869c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:37.895524 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.895414 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/636b68e8-0b60-4662-a5f2-ac20c01c869c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"636b68e8-0b60-4662-a5f2-ac20c01c869c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:37.895524 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.895435 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fth8\" (UniqueName: \"kubernetes.io/projected/636b68e8-0b60-4662-a5f2-ac20c01c869c-kube-api-access-8fth8\") pod \"data-science-collector-collector-0\" (UID: \"636b68e8-0b60-4662-a5f2-ac20c01c869c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:37.895605 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.895523 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"636b68e8-0b60-4662-a5f2-ac20c01c869c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:37.895641 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.895610 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"636b68e8-0b60-4662-a5f2-ac20c01c869c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:37.996722 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.996660 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"636b68e8-0b60-4662-a5f2-ac20c01c869c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:37.997043 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.996739 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"636b68e8-0b60-4662-a5f2-ac20c01c869c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:37.997043 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.996790 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/636b68e8-0b60-4662-a5f2-ac20c01c869c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"636b68e8-0b60-4662-a5f2-ac20c01c869c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:37.997043 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.996819 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fth8\" (UniqueName: \"kubernetes.io/projected/636b68e8-0b60-4662-a5f2-ac20c01c869c-kube-api-access-8fth8\") pod \"data-science-collector-collector-0\" (UID: \"636b68e8-0b60-4662-a5f2-ac20c01c869c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:37.997043 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.996858 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"636b68e8-0b60-4662-a5f2-ac20c01c869c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:37.997458 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.997436 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/636b68e8-0b60-4662-a5f2-ac20c01c869c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"636b68e8-0b60-4662-a5f2-ac20c01c869c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:37.999137 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.999110 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"636b68e8-0b60-4662-a5f2-ac20c01c869c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:37.999252 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.999234 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"636b68e8-0b60-4662-a5f2-ac20c01c869c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:37.999383 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:37.999365 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"636b68e8-0b60-4662-a5f2-ac20c01c869c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:38.021691 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:38.021666 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fth8\" (UniqueName: \"kubernetes.io/projected/636b68e8-0b60-4662-a5f2-ac20c01c869c-kube-api-access-8fth8\") pod \"data-science-collector-collector-0\" (UID: \"636b68e8-0b60-4662-a5f2-ac20c01c869c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:38.142309 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:38.142281 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:38.444490 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:38.444465 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:38.447177 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:44:38.447150 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636b68e8_0b60_4662_a5f2_ac20c01c869c.slice/crio-bee6629938b93afcd5050ae28aaca894cf05194b3999dbf9706513ff22067458 WatchSource:0}: Error finding container bee6629938b93afcd5050ae28aaca894cf05194b3999dbf9706513ff22067458: Status 404 returned error can't find the container with id bee6629938b93afcd5050ae28aaca894cf05194b3999dbf9706513ff22067458 Oct 30 03:44:38.449084 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:38.449059 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:38.449166 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:38.449115 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:38.710701 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:38.710618 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"636b68e8-0b60-4662-a5f2-ac20c01c869c","Type":"ContainerStarted","Data":"e96a6cd7195e8705684b9e45749e3bce56f949030aae060a611effed76c303ba"} Oct 30 03:44:38.710701 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:38.710678 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"636b68e8-0b60-4662-a5f2-ac20c01c869c","Type":"ContainerStarted","Data":"bee6629938b93afcd5050ae28aaca894cf05194b3999dbf9706513ff22067458"} Oct 30 03:44:38.755614 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:38.755568 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.7555555109999998 podStartE2EDuration="1.755555511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:38.753422072 +0000 UTC m=+3839.766700175" watchObservedRunningTime="2026-10-30 03:44:38.755555511 +0000 UTC m=+3839.768833614" Oct 30 03:44:39.120102 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:39.120074 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6b50ab9-a519-4227-84ea-91d8eb83ed2b" path="/var/lib/kubelet/pods/c6b50ab9-a519-4227-84ea-91d8eb83ed2b/volumes" Oct 30 03:44:45.984630 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:45.984595 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:45.985067 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:45.984838 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="636b68e8-0b60-4662-a5f2-ac20c01c869c" containerName="otc-container" containerID="cri-o://e96a6cd7195e8705684b9e45749e3bce56f949030aae060a611effed76c303ba" gracePeriod=30 Oct 30 03:44:46.252005 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.251937 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:46.275359 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.275333 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-data-science-collector-ta-client-cert\") pod \"636b68e8-0b60-4662-a5f2-ac20c01c869c\" (UID: \"636b68e8-0b60-4662-a5f2-ac20c01c869c\") " Oct 30 03:44:46.275495 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.275418 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-tls-certs\" (UniqueName: \"kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-tls-certs\") pod \"636b68e8-0b60-4662-a5f2-ac20c01c869c\" (UID: \"636b68e8-0b60-4662-a5f2-ac20c01c869c\") " Oct 30 03:44:46.275495 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.275444 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636b68e8-0b60-4662-a5f2-ac20c01c869c-kube-api-access-8fth8\" (UniqueName: \"kubernetes.io/projected/636b68e8-0b60-4662-a5f2-ac20c01c869c-kube-api-access-8fth8\") pod \"636b68e8-0b60-4662-a5f2-ac20c01c869c\" (UID: \"636b68e8-0b60-4662-a5f2-ac20c01c869c\") " Oct 30 03:44:46.275495 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.275481 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636b68e8-0b60-4662-a5f2-ac20c01c869c-otc-internal\" (UniqueName: \"kubernetes.io/configmap/636b68e8-0b60-4662-a5f2-ac20c01c869c-otc-internal\") pod \"636b68e8-0b60-4662-a5f2-ac20c01c869c\" (UID: \"636b68e8-0b60-4662-a5f2-ac20c01c869c\") " Oct 30 03:44:46.275657 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.275503 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-monitoring-tls-certs\") pod \"636b68e8-0b60-4662-a5f2-ac20c01c869c\" (UID: \"636b68e8-0b60-4662-a5f2-ac20c01c869c\") " Oct 30 03:44:46.275887 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.275864 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636b68e8-0b60-4662-a5f2-ac20c01c869c-otc-internal" pod "636b68e8-0b60-4662-a5f2-ac20c01c869c" (UID: "636b68e8-0b60-4662-a5f2-ac20c01c869c"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:46.277541 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.277514 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-data-science-collector-ta-client-cert" pod "636b68e8-0b60-4662-a5f2-ac20c01c869c" (UID: "636b68e8-0b60-4662-a5f2-ac20c01c869c"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:46.277634 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.277566 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-tls-certs" pod "636b68e8-0b60-4662-a5f2-ac20c01c869c" (UID: "636b68e8-0b60-4662-a5f2-ac20c01c869c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:46.277634 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.277587 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-monitoring-tls-certs" pod "636b68e8-0b60-4662-a5f2-ac20c01c869c" (UID: "636b68e8-0b60-4662-a5f2-ac20c01c869c"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:46.277634 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.277614 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636b68e8-0b60-4662-a5f2-ac20c01c869c-kube-api-access-8fth8" pod "636b68e8-0b60-4662-a5f2-ac20c01c869c" (UID: "636b68e8-0b60-4662-a5f2-ac20c01c869c"). InnerVolumeSpecName "kube-api-access-8fth8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:46.376569 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.376531 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:46.376569 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.376562 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fth8\" (UniqueName: \"kubernetes.io/projected/636b68e8-0b60-4662-a5f2-ac20c01c869c-kube-api-access-8fth8\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:46.376569 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.376571 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/636b68e8-0b60-4662-a5f2-ac20c01c869c-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:46.376823 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.376581 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:46.376823 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.376590 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/636b68e8-0b60-4662-a5f2-ac20c01c869c-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:46.751732 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.751692 4925 generic.go:356] "Generic (PLEG): container finished" podID="636b68e8-0b60-4662-a5f2-ac20c01c869c" containerID="e96a6cd7195e8705684b9e45749e3bce56f949030aae060a611effed76c303ba" exitCode=0 Oct 30 03:44:46.751920 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.751745 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:46.751920 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.751780 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"636b68e8-0b60-4662-a5f2-ac20c01c869c","Type":"ContainerDied","Data":"e96a6cd7195e8705684b9e45749e3bce56f949030aae060a611effed76c303ba"} Oct 30 03:44:46.751920 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.751835 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"636b68e8-0b60-4662-a5f2-ac20c01c869c","Type":"ContainerDied","Data":"bee6629938b93afcd5050ae28aaca894cf05194b3999dbf9706513ff22067458"} Oct 30 03:44:46.751920 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.751862 4925 scope.go:122] "RemoveContainer" containerID="e96a6cd7195e8705684b9e45749e3bce56f949030aae060a611effed76c303ba" Oct 30 03:44:46.761267 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.761251 4925 scope.go:122] "RemoveContainer" containerID="e96a6cd7195e8705684b9e45749e3bce56f949030aae060a611effed76c303ba" Oct 30 03:44:46.761527 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:44:46.761502 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e96a6cd7195e8705684b9e45749e3bce56f949030aae060a611effed76c303ba\": container with ID starting with e96a6cd7195e8705684b9e45749e3bce56f949030aae060a611effed76c303ba not found: ID does not exist" containerID="e96a6cd7195e8705684b9e45749e3bce56f949030aae060a611effed76c303ba" Oct 30 03:44:46.761601 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.761535 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e96a6cd7195e8705684b9e45749e3bce56f949030aae060a611effed76c303ba"} err="failed to get container status \"e96a6cd7195e8705684b9e45749e3bce56f949030aae060a611effed76c303ba\": rpc error: code = NotFound desc = could not find container \"e96a6cd7195e8705684b9e45749e3bce56f949030aae060a611effed76c303ba\": container with ID starting with e96a6cd7195e8705684b9e45749e3bce56f949030aae060a611effed76c303ba not found: ID does not exist" Oct 30 03:44:46.804853 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.804823 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:46.817294 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.817264 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:46.846939 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.846915 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:46.847323 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.847307 4925 cpu_manager.go:414] "containerMap: removing container" podUID="636b68e8-0b60-4662-a5f2-ac20c01c869c" containerName="otc-container" Oct 30 03:44:46.847400 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.847325 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636b68e8-0b60-4662-a5f2-ac20c01c869c" containerName="otc-container" Oct 30 03:44:46.847509 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.847424 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="636b68e8-0b60-4662-a5f2-ac20c01c869c" containerName="otc-container" Oct 30 03:44:46.855318 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.855299 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:46.870848 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.870826 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 03:44:46.872624 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.872603 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:46.875010 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.874247 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\"" Oct 30 03:44:46.875010 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.874299 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:44:46.875010 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.874331 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 03:44:46.875010 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.874608 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 03:44:46.882093 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.881844 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrjpx\" (UniqueName: \"kubernetes.io/projected/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-kube-api-access-xrjpx\") pod \"data-science-collector-collector-0\" (UID: \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:46.882180 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.882126 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:46.882180 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.882161 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:46.882286 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.882191 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:46.882574 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.882553 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:46.983397 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.983364 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:46.983557 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.983422 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrjpx\" (UniqueName: \"kubernetes.io/projected/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-kube-api-access-xrjpx\") pod \"data-science-collector-collector-0\" (UID: \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:46.983557 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.983442 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:46.983557 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.983460 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:46.983557 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.983475 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:46.984258 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.984233 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:46.985883 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.985861 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:46.985883 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.985871 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:46.986181 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:46.985932 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:47.007289 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:47.007231 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrjpx\" (UniqueName: \"kubernetes.io/projected/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-kube-api-access-xrjpx\") pod \"data-science-collector-collector-0\" (UID: \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:47.119563 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:47.119537 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636b68e8-0b60-4662-a5f2-ac20c01c869c" path="/var/lib/kubelet/pods/636b68e8-0b60-4662-a5f2-ac20c01c869c/volumes" Oct 30 03:44:47.168040 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:47.168015 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:47.470565 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:47.470539 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:47.472351 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:44:47.472327 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9861ff9d_6690_41c5_ac9e_ab3ced4d48bd.slice/crio-08a828a6a34205d50a37fc4b0fc35bb547271db6e3bdf7832d16fc39f484e218 WatchSource:0}: Error finding container 08a828a6a34205d50a37fc4b0fc35bb547271db6e3bdf7832d16fc39f484e218: Status 404 returned error can't find the container with id 08a828a6a34205d50a37fc4b0fc35bb547271db6e3bdf7832d16fc39f484e218 Oct 30 03:44:47.474545 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:47.474513 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:47.474631 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:47.474571 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:47.758686 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:47.758604 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd","Type":"ContainerStarted","Data":"57cf25f8824626d3405a99b4bd71bd8ca55b5e88881aaa00ee32386ef1a13649"} Oct 30 03:44:47.758686 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:47.758663 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd","Type":"ContainerStarted","Data":"08a828a6a34205d50a37fc4b0fc35bb547271db6e3bdf7832d16fc39f484e218"} Oct 30 03:44:47.805266 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:47.805213 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.805201472 podStartE2EDuration="1.805201472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:47.803385011 +0000 UTC m=+3848.816663145" watchObservedRunningTime="2026-10-30 03:44:47.805201472 +0000 UTC m=+3848.818479641" Oct 30 03:44:48.161472 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:44:48.161433 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:44:54.002086 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.002053 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:54.002535 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.002220 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="9861ff9d-6690-41c5-ac9e-ab3ced4d48bd" containerName="otc-container" containerID="cri-o://57cf25f8824626d3405a99b4bd71bd8ca55b5e88881aaa00ee32386ef1a13649" gracePeriod=30 Oct 30 03:44:54.255943 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.255890 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:54.343181 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.343155 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-monitoring-tls-certs\") pod \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\" (UID: \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\") " Oct 30 03:44:54.343322 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.343213 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-kube-api-access-xrjpx\" (UniqueName: \"kubernetes.io/projected/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-kube-api-access-xrjpx\") pod \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\" (UID: \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\") " Oct 30 03:44:54.343322 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.343262 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-data-science-collector-ta-client-cert\") pod \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\" (UID: \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\") " Oct 30 03:44:54.343322 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.343286 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-tls-certs\") pod \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\" (UID: \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\") " Oct 30 03:44:54.343489 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.343325 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-otc-internal\" (UniqueName: \"kubernetes.io/configmap/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-otc-internal\") pod \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\" (UID: \"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd\") " Oct 30 03:44:54.343804 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.343743 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-otc-internal" pod "9861ff9d-6690-41c5-ac9e-ab3ced4d48bd" (UID: "9861ff9d-6690-41c5-ac9e-ab3ced4d48bd"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:54.345306 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.345282 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-data-science-collector-ta-client-cert" pod "9861ff9d-6690-41c5-ac9e-ab3ced4d48bd" (UID: "9861ff9d-6690-41c5-ac9e-ab3ced4d48bd"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:54.345676 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.345652 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-kube-api-access-xrjpx" pod "9861ff9d-6690-41c5-ac9e-ab3ced4d48bd" (UID: "9861ff9d-6690-41c5-ac9e-ab3ced4d48bd"). InnerVolumeSpecName "kube-api-access-xrjpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:54.345738 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.345688 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-monitoring-tls-certs" pod "9861ff9d-6690-41c5-ac9e-ab3ced4d48bd" (UID: "9861ff9d-6690-41c5-ac9e-ab3ced4d48bd"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:54.345738 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.345708 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-tls-certs" pod "9861ff9d-6690-41c5-ac9e-ab3ced4d48bd" (UID: "9861ff9d-6690-41c5-ac9e-ab3ced4d48bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:44:54.444032 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.444002 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:54.444032 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.444032 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:54.444190 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.444044 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:54.444190 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.444063 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:54.444190 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.444071 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrjpx\" (UniqueName: \"kubernetes.io/projected/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd-kube-api-access-xrjpx\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:44:54.795371 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.795333 4925 generic.go:356] "Generic (PLEG): container finished" podID="9861ff9d-6690-41c5-ac9e-ab3ced4d48bd" containerID="57cf25f8824626d3405a99b4bd71bd8ca55b5e88881aaa00ee32386ef1a13649" exitCode=0 Oct 30 03:44:54.795525 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.795380 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:54.795525 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.795424 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd","Type":"ContainerDied","Data":"57cf25f8824626d3405a99b4bd71bd8ca55b5e88881aaa00ee32386ef1a13649"} Oct 30 03:44:54.795525 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.795479 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9861ff9d-6690-41c5-ac9e-ab3ced4d48bd","Type":"ContainerDied","Data":"08a828a6a34205d50a37fc4b0fc35bb547271db6e3bdf7832d16fc39f484e218"} Oct 30 03:44:54.795525 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.795501 4925 scope.go:122] "RemoveContainer" containerID="57cf25f8824626d3405a99b4bd71bd8ca55b5e88881aaa00ee32386ef1a13649" Oct 30 03:44:54.806223 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.805649 4925 scope.go:122] "RemoveContainer" containerID="57cf25f8824626d3405a99b4bd71bd8ca55b5e88881aaa00ee32386ef1a13649" Oct 30 03:44:54.807382 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:44:54.807219 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57cf25f8824626d3405a99b4bd71bd8ca55b5e88881aaa00ee32386ef1a13649\": container with ID starting with 57cf25f8824626d3405a99b4bd71bd8ca55b5e88881aaa00ee32386ef1a13649 not found: ID does not exist" containerID="57cf25f8824626d3405a99b4bd71bd8ca55b5e88881aaa00ee32386ef1a13649" Oct 30 03:44:54.807382 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.807254 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57cf25f8824626d3405a99b4bd71bd8ca55b5e88881aaa00ee32386ef1a13649"} err="failed to get container status \"57cf25f8824626d3405a99b4bd71bd8ca55b5e88881aaa00ee32386ef1a13649\": rpc error: code = NotFound desc = could not find container \"57cf25f8824626d3405a99b4bd71bd8ca55b5e88881aaa00ee32386ef1a13649\": container with ID starting with 57cf25f8824626d3405a99b4bd71bd8ca55b5e88881aaa00ee32386ef1a13649 not found: ID does not exist" Oct 30 03:44:54.849263 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.849228 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:54.861101 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.861076 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:54.893927 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.893904 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:54.894313 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.894299 4925 cpu_manager.go:414] "containerMap: removing container" podUID="9861ff9d-6690-41c5-ac9e-ab3ced4d48bd" containerName="otc-container" Oct 30 03:44:54.894379 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.894314 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9861ff9d-6690-41c5-ac9e-ab3ced4d48bd" containerName="otc-container" Oct 30 03:44:54.894422 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.894382 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9861ff9d-6690-41c5-ac9e-ab3ced4d48bd" containerName="otc-container" Oct 30 03:44:54.898847 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.898827 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:54.914351 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.914327 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:44:54.914351 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.914342 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\"" Oct 30 03:44:54.914483 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.914334 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 03:44:54.914483 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.914327 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 03:44:54.914483 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.914348 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 03:44:54.919010 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.918973 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:54.948039 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.948014 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:54.948160 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.948079 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:54.948160 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.948141 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txpr4\" (UniqueName: \"kubernetes.io/projected/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-kube-api-access-txpr4\") pod \"data-science-collector-collector-0\" (UID: \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:54.948267 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.948164 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:54.948267 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:54.948187 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:55.049134 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:55.049062 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:55.049134 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:55.049134 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:55.049616 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:55.049163 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txpr4\" (UniqueName: \"kubernetes.io/projected/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-kube-api-access-txpr4\") pod \"data-science-collector-collector-0\" (UID: \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:55.049616 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:55.049182 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:55.049616 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:55.049208 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:55.050002 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:55.049973 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:55.051625 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:55.051601 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:55.051766 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:55.051737 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:55.051836 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:55.051821 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:55.073495 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:55.073468 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txpr4\" (UniqueName: \"kubernetes.io/projected/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-kube-api-access-txpr4\") pod \"data-science-collector-collector-0\" (UID: \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:55.119721 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:55.119537 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9861ff9d-6690-41c5-ac9e-ab3ced4d48bd" path="/var/lib/kubelet/pods/9861ff9d-6690-41c5-ac9e-ab3ced4d48bd/volumes" Oct 30 03:44:55.211089 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:55.211061 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:44:55.508728 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:55.508706 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:44:55.510323 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:44:55.510293 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b8bbfb_564f_4c26_a1d4_53e76f41cb51.slice/crio-1e67779ba3edea6074ce14d0b09c1ef90c4facb42bc51522a1944a3f4e824d18 WatchSource:0}: Error finding container 1e67779ba3edea6074ce14d0b09c1ef90c4facb42bc51522a1944a3f4e824d18: Status 404 returned error can't find the container with id 1e67779ba3edea6074ce14d0b09c1ef90c4facb42bc51522a1944a3f4e824d18 Oct 30 03:44:55.512360 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:55.512330 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:55.512425 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:55.512387 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:44:55.803224 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:55.803131 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"67b8bbfb-564f-4c26-a1d4-53e76f41cb51","Type":"ContainerStarted","Data":"33be876706dd240b01c344964001175ab0847dea6a7855bd27cd141090d1ce5b"} Oct 30 03:44:55.803224 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:55.803194 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"67b8bbfb-564f-4c26-a1d4-53e76f41cb51","Type":"ContainerStarted","Data":"1e67779ba3edea6074ce14d0b09c1ef90c4facb42bc51522a1944a3f4e824d18"} Oct 30 03:44:55.849681 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:44:55.849626 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.84961384 podStartE2EDuration="1.84961384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:55.849267558 +0000 UTC m=+3856.862545659" watchObservedRunningTime="2026-10-30 03:44:55.84961384 +0000 UTC m=+3856.862891941" Oct 30 03:45:13.102874 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:13.102834 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:45:13.104022 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:13.103136 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="67b8bbfb-564f-4c26-a1d4-53e76f41cb51" containerName="otc-container" containerID="cri-o://33be876706dd240b01c344964001175ab0847dea6a7855bd27cd141090d1ce5b" gracePeriod=30 Oct 30 03:45:14.350239 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.350215 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:14.412451 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.412421 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-otc-internal\" (UniqueName: \"kubernetes.io/configmap/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-otc-internal\") pod \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\" (UID: \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\") " Oct 30 03:45:14.412625 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.412488 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-tls-certs\" (UniqueName: \"kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-tls-certs\") pod \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\" (UID: \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\") " Oct 30 03:45:14.412625 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.412523 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-data-science-collector-ta-client-cert\") pod \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\" (UID: \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\") " Oct 30 03:45:14.412625 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.412573 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-monitoring-tls-certs\") pod \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\" (UID: \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\") " Oct 30 03:45:14.412625 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.412608 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-kube-api-access-txpr4\" (UniqueName: \"kubernetes.io/projected/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-kube-api-access-txpr4\") pod \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\" (UID: \"67b8bbfb-564f-4c26-a1d4-53e76f41cb51\") " Oct 30 03:45:14.412882 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.412817 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-otc-internal" pod "67b8bbfb-564f-4c26-a1d4-53e76f41cb51" (UID: "67b8bbfb-564f-4c26-a1d4-53e76f41cb51"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:14.412930 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.412888 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:45:14.414732 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.414703 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-monitoring-tls-certs" pod "67b8bbfb-564f-4c26-a1d4-53e76f41cb51" (UID: "67b8bbfb-564f-4c26-a1d4-53e76f41cb51"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:45:14.414732 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.414728 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-tls-certs" pod "67b8bbfb-564f-4c26-a1d4-53e76f41cb51" (UID: "67b8bbfb-564f-4c26-a1d4-53e76f41cb51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:45:14.414939 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.414870 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-data-science-collector-ta-client-cert" pod "67b8bbfb-564f-4c26-a1d4-53e76f41cb51" (UID: "67b8bbfb-564f-4c26-a1d4-53e76f41cb51"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:45:14.415128 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.415110 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-kube-api-access-txpr4" pod "67b8bbfb-564f-4c26-a1d4-53e76f41cb51" (UID: "67b8bbfb-564f-4c26-a1d4-53e76f41cb51"). InnerVolumeSpecName "kube-api-access-txpr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:14.513359 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.513291 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:45:14.513359 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.513315 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txpr4\" (UniqueName: \"kubernetes.io/projected/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-kube-api-access-txpr4\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:45:14.513359 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.513329 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:45:14.513359 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.513342 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/67b8bbfb-564f-4c26-a1d4-53e76f41cb51-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:45:14.902827 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.902794 4925 generic.go:356] "Generic (PLEG): container finished" podID="67b8bbfb-564f-4c26-a1d4-53e76f41cb51" containerID="33be876706dd240b01c344964001175ab0847dea6a7855bd27cd141090d1ce5b" exitCode=0 Oct 30 03:45:14.903009 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.902842 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:14.903009 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.902855 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"67b8bbfb-564f-4c26-a1d4-53e76f41cb51","Type":"ContainerDied","Data":"33be876706dd240b01c344964001175ab0847dea6a7855bd27cd141090d1ce5b"} Oct 30 03:45:14.903009 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.902897 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"67b8bbfb-564f-4c26-a1d4-53e76f41cb51","Type":"ContainerDied","Data":"1e67779ba3edea6074ce14d0b09c1ef90c4facb42bc51522a1944a3f4e824d18"} Oct 30 03:45:14.903009 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.902917 4925 scope.go:122] "RemoveContainer" containerID="33be876706dd240b01c344964001175ab0847dea6a7855bd27cd141090d1ce5b" Oct 30 03:45:14.913112 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.913089 4925 scope.go:122] "RemoveContainer" containerID="33be876706dd240b01c344964001175ab0847dea6a7855bd27cd141090d1ce5b" Oct 30 03:45:14.913393 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:45:14.913374 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33be876706dd240b01c344964001175ab0847dea6a7855bd27cd141090d1ce5b\": container with ID starting with 33be876706dd240b01c344964001175ab0847dea6a7855bd27cd141090d1ce5b not found: ID does not exist" containerID="33be876706dd240b01c344964001175ab0847dea6a7855bd27cd141090d1ce5b" Oct 30 03:45:14.913439 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.913402 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33be876706dd240b01c344964001175ab0847dea6a7855bd27cd141090d1ce5b"} err="failed to get container status \"33be876706dd240b01c344964001175ab0847dea6a7855bd27cd141090d1ce5b\": rpc error: code = NotFound desc = could not find container \"33be876706dd240b01c344964001175ab0847dea6a7855bd27cd141090d1ce5b\": container with ID starting with 33be876706dd240b01c344964001175ab0847dea6a7855bd27cd141090d1ce5b not found: ID does not exist" Oct 30 03:45:14.954709 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.954682 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:45:14.967182 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.967158 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:45:14.999283 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.999253 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:45:14.999654 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.999637 4925 cpu_manager.go:414] "containerMap: removing container" podUID="67b8bbfb-564f-4c26-a1d4-53e76f41cb51" containerName="otc-container" Oct 30 03:45:14.999729 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.999657 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b8bbfb-564f-4c26-a1d4-53e76f41cb51" containerName="otc-container" Oct 30 03:45:14.999835 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:14.999821 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b8bbfb-564f-4c26-a1d4-53e76f41cb51" containerName="otc-container" Oct 30 03:45:15.004098 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.004079 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:15.019622 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.019602 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 03:45:15.019622 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.019607 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 03:45:15.019813 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.019691 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\"" Oct 30 03:45:15.019875 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.019829 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:45:15.020218 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.020201 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 03:45:15.026715 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.026695 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:45:15.117761 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.117702 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"193feb13-e681-498a-b105-3faa70c6c5cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:15.117761 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.117740 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/193feb13-e681-498a-b105-3faa70c6c5cd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"193feb13-e681-498a-b105-3faa70c6c5cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:15.117996 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.117819 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"193feb13-e681-498a-b105-3faa70c6c5cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:15.117996 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.117856 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"193feb13-e681-498a-b105-3faa70c6c5cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:15.117996 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.117966 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gtm4\" (UniqueName: \"kubernetes.io/projected/193feb13-e681-498a-b105-3faa70c6c5cd-kube-api-access-5gtm4\") pod \"data-science-collector-collector-0\" (UID: \"193feb13-e681-498a-b105-3faa70c6c5cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:15.119544 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.119519 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b8bbfb-564f-4c26-a1d4-53e76f41cb51" path="/var/lib/kubelet/pods/67b8bbfb-564f-4c26-a1d4-53e76f41cb51/volumes" Oct 30 03:45:15.219001 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.218937 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gtm4\" (UniqueName: \"kubernetes.io/projected/193feb13-e681-498a-b105-3faa70c6c5cd-kube-api-access-5gtm4\") pod \"data-science-collector-collector-0\" (UID: \"193feb13-e681-498a-b105-3faa70c6c5cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:15.219001 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.218994 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"193feb13-e681-498a-b105-3faa70c6c5cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:15.219157 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.219038 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/193feb13-e681-498a-b105-3faa70c6c5cd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"193feb13-e681-498a-b105-3faa70c6c5cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:15.219157 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.219060 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"193feb13-e681-498a-b105-3faa70c6c5cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:15.219157 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.219106 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"193feb13-e681-498a-b105-3faa70c6c5cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:15.219705 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.219678 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/193feb13-e681-498a-b105-3faa70c6c5cd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"193feb13-e681-498a-b105-3faa70c6c5cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:15.221651 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.221631 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"193feb13-e681-498a-b105-3faa70c6c5cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:15.221744 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.221669 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"193feb13-e681-498a-b105-3faa70c6c5cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:15.221744 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.221689 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"193feb13-e681-498a-b105-3faa70c6c5cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:15.243526 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.243501 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gtm4\" (UniqueName: \"kubernetes.io/projected/193feb13-e681-498a-b105-3faa70c6c5cd-kube-api-access-5gtm4\") pod \"data-science-collector-collector-0\" (UID: \"193feb13-e681-498a-b105-3faa70c6c5cd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:15.318650 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.318611 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:15.617366 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.617339 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:45:15.619162 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:45:15.619136 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193feb13_e681_498a_b105_3faa70c6c5cd.slice/crio-cb90ad2131c754afc03998e253771132144643cd5d519b4c60847aa386eaa279 WatchSource:0}: Error finding container cb90ad2131c754afc03998e253771132144643cd5d519b4c60847aa386eaa279: Status 404 returned error can't find the container with id cb90ad2131c754afc03998e253771132144643cd5d519b4c60847aa386eaa279 Oct 30 03:45:15.621314 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.621289 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:45:15.621395 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.621340 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:45:15.910714 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.910631 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"193feb13-e681-498a-b105-3faa70c6c5cd","Type":"ContainerStarted","Data":"163b2ff52e5b51cf2d550d129def4f5f4df7822afdad800c68495683be566288"} Oct 30 03:45:15.910714 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.910682 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"193feb13-e681-498a-b105-3faa70c6c5cd","Type":"ContainerStarted","Data":"cb90ad2131c754afc03998e253771132144643cd5d519b4c60847aa386eaa279"} Oct 30 03:45:15.965348 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:15.965280 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.965263816 podStartE2EDuration="1.965263816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:15.963344918 +0000 UTC m=+3876.976623032" watchObservedRunningTime="2026-10-30 03:45:15.965263816 +0000 UTC m=+3876.978541918" Oct 30 03:45:18.161287 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:45:18.161251 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:45:30.203497 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:30.203452 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:45:30.204157 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:30.203677 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="193feb13-e681-498a-b105-3faa70c6c5cd" containerName="otc-container" containerID="cri-o://163b2ff52e5b51cf2d550d129def4f5f4df7822afdad800c68495683be566288" gracePeriod=30 Oct 30 03:45:31.444459 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.444433 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:31.549816 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.549771 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193feb13-e681-498a-b105-3faa70c6c5cd-kube-api-access-5gtm4\" (UniqueName: \"kubernetes.io/projected/193feb13-e681-498a-b105-3faa70c6c5cd-kube-api-access-5gtm4\") pod \"193feb13-e681-498a-b105-3faa70c6c5cd\" (UID: \"193feb13-e681-498a-b105-3faa70c6c5cd\") " Oct 30 03:45:31.549974 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.549830 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-tls-certs\") pod \"193feb13-e681-498a-b105-3faa70c6c5cd\" (UID: \"193feb13-e681-498a-b105-3faa70c6c5cd\") " Oct 30 03:45:31.549974 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.549861 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-data-science-collector-ta-client-cert\") pod \"193feb13-e681-498a-b105-3faa70c6c5cd\" (UID: \"193feb13-e681-498a-b105-3faa70c6c5cd\") " Oct 30 03:45:31.549974 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.549888 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-monitoring-tls-certs\") pod \"193feb13-e681-498a-b105-3faa70c6c5cd\" (UID: \"193feb13-e681-498a-b105-3faa70c6c5cd\") " Oct 30 03:45:31.549974 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.549943 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/193feb13-e681-498a-b105-3faa70c6c5cd-otc-internal\" (UniqueName: \"kubernetes.io/configmap/193feb13-e681-498a-b105-3faa70c6c5cd-otc-internal\") pod \"193feb13-e681-498a-b105-3faa70c6c5cd\" (UID: \"193feb13-e681-498a-b105-3faa70c6c5cd\") " Oct 30 03:45:31.550358 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.550326 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/193feb13-e681-498a-b105-3faa70c6c5cd-otc-internal" pod "193feb13-e681-498a-b105-3faa70c6c5cd" (UID: "193feb13-e681-498a-b105-3faa70c6c5cd"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:31.551973 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.551939 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-data-science-collector-ta-client-cert" pod "193feb13-e681-498a-b105-3faa70c6c5cd" (UID: "193feb13-e681-498a-b105-3faa70c6c5cd"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:45:31.552093 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.552011 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-tls-certs" pod "193feb13-e681-498a-b105-3faa70c6c5cd" (UID: "193feb13-e681-498a-b105-3faa70c6c5cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:45:31.552234 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.552214 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-monitoring-tls-certs" pod "193feb13-e681-498a-b105-3faa70c6c5cd" (UID: "193feb13-e681-498a-b105-3faa70c6c5cd"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:45:31.552295 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.552241 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193feb13-e681-498a-b105-3faa70c6c5cd-kube-api-access-5gtm4" pod "193feb13-e681-498a-b105-3faa70c6c5cd" (UID: "193feb13-e681-498a-b105-3faa70c6c5cd"). InnerVolumeSpecName "kube-api-access-5gtm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:31.650985 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.650956 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/193feb13-e681-498a-b105-3faa70c6c5cd-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:45:31.650985 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.650984 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gtm4\" (UniqueName: \"kubernetes.io/projected/193feb13-e681-498a-b105-3faa70c6c5cd-kube-api-access-5gtm4\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:45:31.651127 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.650996 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:45:31.651127 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.651007 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:45:31.651127 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.651016 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/193feb13-e681-498a-b105-3faa70c6c5cd-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:45:31.995065 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.995030 4925 generic.go:356] "Generic (PLEG): container finished" podID="193feb13-e681-498a-b105-3faa70c6c5cd" containerID="163b2ff52e5b51cf2d550d129def4f5f4df7822afdad800c68495683be566288" exitCode=0 Oct 30 03:45:31.995214 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.995076 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:31.995214 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.995088 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"193feb13-e681-498a-b105-3faa70c6c5cd","Type":"ContainerDied","Data":"163b2ff52e5b51cf2d550d129def4f5f4df7822afdad800c68495683be566288"} Oct 30 03:45:31.995214 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.995129 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"193feb13-e681-498a-b105-3faa70c6c5cd","Type":"ContainerDied","Data":"cb90ad2131c754afc03998e253771132144643cd5d519b4c60847aa386eaa279"} Oct 30 03:45:31.995214 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:31.995148 4925 scope.go:122] "RemoveContainer" containerID="163b2ff52e5b51cf2d550d129def4f5f4df7822afdad800c68495683be566288" Oct 30 03:45:32.004039 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.004019 4925 scope.go:122] "RemoveContainer" containerID="163b2ff52e5b51cf2d550d129def4f5f4df7822afdad800c68495683be566288" Oct 30 03:45:32.004298 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:45:32.004274 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163b2ff52e5b51cf2d550d129def4f5f4df7822afdad800c68495683be566288\": container with ID starting with 163b2ff52e5b51cf2d550d129def4f5f4df7822afdad800c68495683be566288 not found: ID does not exist" containerID="163b2ff52e5b51cf2d550d129def4f5f4df7822afdad800c68495683be566288" Oct 30 03:45:32.004363 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.004305 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163b2ff52e5b51cf2d550d129def4f5f4df7822afdad800c68495683be566288"} err="failed to get container status \"163b2ff52e5b51cf2d550d129def4f5f4df7822afdad800c68495683be566288\": rpc error: code = NotFound desc = could not find container \"163b2ff52e5b51cf2d550d129def4f5f4df7822afdad800c68495683be566288\": container with ID starting with 163b2ff52e5b51cf2d550d129def4f5f4df7822afdad800c68495683be566288 not found: ID does not exist" Oct 30 03:45:32.047456 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.047430 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:45:32.060140 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.060115 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:45:32.095031 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.094999 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:45:32.095346 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.095333 4925 cpu_manager.go:414] "containerMap: removing container" podUID="193feb13-e681-498a-b105-3faa70c6c5cd" containerName="otc-container" Oct 30 03:45:32.095346 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.095346 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="193feb13-e681-498a-b105-3faa70c6c5cd" containerName="otc-container" Oct 30 03:45:32.095470 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.095425 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="193feb13-e681-498a-b105-3faa70c6c5cd" containerName="otc-container" Oct 30 03:45:32.099900 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.099880 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:32.115457 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.115436 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 03:45:32.115550 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.115506 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\"" Oct 30 03:45:32.118153 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.118129 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:45:32.118255 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.118192 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 03:45:32.120667 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.120643 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:45:32.216086 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.216059 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 03:45:32.256615 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.256546 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psqz6\" (UniqueName: \"kubernetes.io/projected/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-kube-api-access-psqz6\") pod \"data-science-collector-collector-0\" (UID: \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:32.256615 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.256609 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:32.256795 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.256681 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:32.256795 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.256708 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:32.256795 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.256726 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:32.358188 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.358153 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psqz6\" (UniqueName: \"kubernetes.io/projected/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-kube-api-access-psqz6\") pod \"data-science-collector-collector-0\" (UID: \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:32.358352 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.358214 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:32.358352 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.358248 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:32.358352 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.358271 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:32.358352 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.358297 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:32.358954 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.358928 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:32.360742 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.360715 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:32.360742 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.360740 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:32.360896 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.360812 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:32.382231 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.382208 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psqz6\" (UniqueName: \"kubernetes.io/projected/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-kube-api-access-psqz6\") pod \"data-science-collector-collector-0\" (UID: \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:32.416129 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.416106 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:45:32.714456 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.714430 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:45:32.715832 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:45:32.715798 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35261b43_9fe0_45f2_b23e_d4a6f12cf1a1.slice/crio-a7a3548b06a356dd3a84155162e91bbc9e85596920b556c76b9e8e880fe6a24f WatchSource:0}: Error finding container a7a3548b06a356dd3a84155162e91bbc9e85596920b556c76b9e8e880fe6a24f: Status 404 returned error can't find the container with id a7a3548b06a356dd3a84155162e91bbc9e85596920b556c76b9e8e880fe6a24f Oct 30 03:45:32.718043 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.718015 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:45:32.718131 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:32.718081 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:45:33.002452 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:33.002368 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1","Type":"ContainerStarted","Data":"46c17f4b065ae9b2ed08c06b0d071d79e469baf32bd61e19b243e465b496e517"} Oct 30 03:45:33.002452 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:33.002416 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1","Type":"ContainerStarted","Data":"a7a3548b06a356dd3a84155162e91bbc9e85596920b556c76b9e8e880fe6a24f"} Oct 30 03:45:33.049935 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:33.049882 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.049868149 podStartE2EDuration="1.049868149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:33.047851037 +0000 UTC m=+3894.061129138" watchObservedRunningTime="2026-10-30 03:45:33.049868149 +0000 UTC m=+3894.063146251" Oct 30 03:45:33.119679 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:33.119647 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="193feb13-e681-498a-b105-3faa70c6c5cd" path="/var/lib/kubelet/pods/193feb13-e681-498a-b105-3faa70c6c5cd/volumes" Oct 30 03:45:39.515493 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:39.515385 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:45:39.535014 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:39.524316 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:45:39.535014 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:39.524408 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:45:39.535014 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:45:39.532946 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:45:48.161443 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:45:48.161397 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:46:18.160630 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:46:18.160596 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:46:33.479975 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:46:33.479939 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb"] Oct 30 03:46:33.485383 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:46:33.485352 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb" Oct 30 03:46:33.513310 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:46:33.513282 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"tests-monitoring-tenant-metrics\"/\"default-dockercfg-mtgj8\"" Oct 30 03:46:33.513427 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:46:33.513342 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"tests-monitoring-tenant-metrics\"/\"openshift-service-ca.crt\"" Oct 30 03:46:33.513427 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:46:33.513374 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"tests-monitoring-tenant-metrics\"/\"kube-root-ca.crt\"" Oct 30 03:46:33.515245 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:46:33.515220 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb"] Oct 30 03:46:33.547230 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:46:33.547200 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ed58ad6e-968e-416c-a532-1bff3f97eef1-data\") pod \"tenant-metrics-exporter-5cf544fb97-rcppb\" (UID: \"ed58ad6e-968e-416c-a532-1bff3f97eef1\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb" Oct 30 03:46:33.547347 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:46:33.547282 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fs6q\" (UniqueName: \"kubernetes.io/projected/ed58ad6e-968e-416c-a532-1bff3f97eef1-kube-api-access-8fs6q\") pod \"tenant-metrics-exporter-5cf544fb97-rcppb\" (UID: \"ed58ad6e-968e-416c-a532-1bff3f97eef1\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb" Oct 30 03:46:33.648146 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:46:33.648122 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ed58ad6e-968e-416c-a532-1bff3f97eef1-data\") pod \"tenant-metrics-exporter-5cf544fb97-rcppb\" (UID: \"ed58ad6e-968e-416c-a532-1bff3f97eef1\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb" Oct 30 03:46:33.648272 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:46:33.648203 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fs6q\" (UniqueName: \"kubernetes.io/projected/ed58ad6e-968e-416c-a532-1bff3f97eef1-kube-api-access-8fs6q\") pod \"tenant-metrics-exporter-5cf544fb97-rcppb\" (UID: \"ed58ad6e-968e-416c-a532-1bff3f97eef1\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb" Oct 30 03:46:33.648581 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:46:33.648561 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ed58ad6e-968e-416c-a532-1bff3f97eef1-data\") pod \"tenant-metrics-exporter-5cf544fb97-rcppb\" (UID: \"ed58ad6e-968e-416c-a532-1bff3f97eef1\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb" Oct 30 03:46:33.686158 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:46:33.686131 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fs6q\" (UniqueName: \"kubernetes.io/projected/ed58ad6e-968e-416c-a532-1bff3f97eef1-kube-api-access-8fs6q\") pod \"tenant-metrics-exporter-5cf544fb97-rcppb\" (UID: \"ed58ad6e-968e-416c-a532-1bff3f97eef1\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb" Oct 30 03:46:33.799180 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:46:33.799121 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb" Oct 30 03:46:34.102683 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:46:34.102654 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb"] Oct 30 03:46:34.104550 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:46:34.104523 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded58ad6e_968e_416c_a532_1bff3f97eef1.slice/crio-0a4bc20f9a968bd5ffdae47621d0fbbfebcf0a0cc6c5e5cb0e3b8ed0e83a2a94 WatchSource:0}: Error finding container 0a4bc20f9a968bd5ffdae47621d0fbbfebcf0a0cc6c5e5cb0e3b8ed0e83a2a94: Status 404 returned error can't find the container with id 0a4bc20f9a968bd5ffdae47621d0fbbfebcf0a0cc6c5e5cb0e3b8ed0e83a2a94 Oct 30 03:46:34.106296 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:46:34.106281 4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:46:34.294743 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:46:34.294711 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb" event={"ID":"ed58ad6e-968e-416c-a532-1bff3f97eef1","Type":"ContainerStarted","Data":"0a4bc20f9a968bd5ffdae47621d0fbbfebcf0a0cc6c5e5cb0e3b8ed0e83a2a94"} Oct 30 03:46:38.317373 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:46:38.317337 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb" event={"ID":"ed58ad6e-968e-416c-a532-1bff3f97eef1","Type":"ContainerStarted","Data":"245120bc146c51ff3c1ff945a1cace078a3c470370cff4d1a7dca99862b497d9"} Oct 30 03:46:38.360673 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:46:38.360622 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb" podStartSLOduration=2.0377148370000002 podStartE2EDuration="5.360608646s" totalImagesPullingTime="3.322893809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:38.359864654 +0000 UTC m=+3959.373142757" watchObservedRunningTime="2026-10-30 03:46:38.360608646 +0000 UTC m=+3959.373886747" Oct 30 03:46:48.161549 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:46:48.161512 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:47:18.160394 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:47:18.160355 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:47:48.162216 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:47:48.162180 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:48:18.160630 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:48:18.160593 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:48:48.160077 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:48:48.160038 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:49:18.160166 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:49:18.160133 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:49:48.160773 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:49:48.160712 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:50:18.160133 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:50:18.160092 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:50:39.548494 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:39.548398 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:50:39.557987 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:39.557964 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:50:39.559813 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:39.559795 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:50:39.568838 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:39.568820 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:50:39.829025 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:39.829000 4925 scope.go:122] "RemoveContainer" containerID="4db562d5e662c5979fb2acc107be72fa7c6cd4410880cd95d231f40ba1053a46" Oct 30 03:50:40.128517 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:40.128431 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb"] Oct 30 03:50:40.128666 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:40.128628 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb" podUID="ed58ad6e-968e-416c-a532-1bff3f97eef1" containerName="tenant-metrics-exporter" containerID="cri-o://245120bc146c51ff3c1ff945a1cace078a3c470370cff4d1a7dca99862b497d9" gracePeriod=30 Oct 30 03:50:40.472845 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:40.472823 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb" Oct 30 03:50:40.473040 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:40.473018 4925 generic.go:356] "Generic (PLEG): container finished" podID="ed58ad6e-968e-416c-a532-1bff3f97eef1" containerID="245120bc146c51ff3c1ff945a1cace078a3c470370cff4d1a7dca99862b497d9" exitCode=0 Oct 30 03:50:40.473119 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:40.473090 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb" event={"ID":"ed58ad6e-968e-416c-a532-1bff3f97eef1","Type":"ContainerDied","Data":"245120bc146c51ff3c1ff945a1cace078a3c470370cff4d1a7dca99862b497d9"} Oct 30 03:50:40.473183 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:40.473163 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb" event={"ID":"ed58ad6e-968e-416c-a532-1bff3f97eef1","Type":"ContainerDied","Data":"0a4bc20f9a968bd5ffdae47621d0fbbfebcf0a0cc6c5e5cb0e3b8ed0e83a2a94"} Oct 30 03:50:40.473237 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:40.473198 4925 scope.go:122] "RemoveContainer" containerID="245120bc146c51ff3c1ff945a1cace078a3c470370cff4d1a7dca99862b497d9" Oct 30 03:50:40.481074 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:40.481048 4925 scope.go:122] "RemoveContainer" containerID="245120bc146c51ff3c1ff945a1cace078a3c470370cff4d1a7dca99862b497d9" Oct 30 03:50:40.481323 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:50:40.481307 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"245120bc146c51ff3c1ff945a1cace078a3c470370cff4d1a7dca99862b497d9\": container with ID starting with 245120bc146c51ff3c1ff945a1cace078a3c470370cff4d1a7dca99862b497d9 not found: ID does not exist" containerID="245120bc146c51ff3c1ff945a1cace078a3c470370cff4d1a7dca99862b497d9" Oct 30 03:50:40.481371 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:40.481330 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245120bc146c51ff3c1ff945a1cace078a3c470370cff4d1a7dca99862b497d9"} err="failed to get container status \"245120bc146c51ff3c1ff945a1cace078a3c470370cff4d1a7dca99862b497d9\": rpc error: code = NotFound desc = could not find container \"245120bc146c51ff3c1ff945a1cace078a3c470370cff4d1a7dca99862b497d9\": container with ID starting with 245120bc146c51ff3c1ff945a1cace078a3c470370cff4d1a7dca99862b497d9 not found: ID does not exist" Oct 30 03:50:40.507739 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:40.507715 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed58ad6e-968e-416c-a532-1bff3f97eef1-kube-api-access-8fs6q\" (UniqueName: \"kubernetes.io/projected/ed58ad6e-968e-416c-a532-1bff3f97eef1-kube-api-access-8fs6q\") pod \"ed58ad6e-968e-416c-a532-1bff3f97eef1\" (UID: \"ed58ad6e-968e-416c-a532-1bff3f97eef1\") " Oct 30 03:50:40.509647 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:40.509621 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed58ad6e-968e-416c-a532-1bff3f97eef1-kube-api-access-8fs6q" pod "ed58ad6e-968e-416c-a532-1bff3f97eef1" (UID: "ed58ad6e-968e-416c-a532-1bff3f97eef1"). InnerVolumeSpecName "kube-api-access-8fs6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:50:40.608856 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:40.608831 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed58ad6e-968e-416c-a532-1bff3f97eef1-data\" (UniqueName: \"kubernetes.io/empty-dir/ed58ad6e-968e-416c-a532-1bff3f97eef1-data\") pod \"ed58ad6e-968e-416c-a532-1bff3f97eef1\" (UID: \"ed58ad6e-968e-416c-a532-1bff3f97eef1\") " Oct 30 03:50:40.609168 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:40.609080 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fs6q\" (UniqueName: \"kubernetes.io/projected/ed58ad6e-968e-416c-a532-1bff3f97eef1-kube-api-access-8fs6q\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:50:40.609237 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:40.609220 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed58ad6e-968e-416c-a532-1bff3f97eef1-data" pod "ed58ad6e-968e-416c-a532-1bff3f97eef1" (UID: "ed58ad6e-968e-416c-a532-1bff3f97eef1"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:50:40.710144 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:40.710088 4925 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ed58ad6e-968e-416c-a532-1bff3f97eef1-data\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:50:41.478166 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:41.478086 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb" Oct 30 03:50:41.530614 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:41.530584 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb"] Oct 30 03:50:41.534347 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:41.534322 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-rcppb"] Oct 30 03:50:43.119410 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:50:43.119375 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed58ad6e-968e-416c-a532-1bff3f97eef1" path="/var/lib/kubelet/pods/ed58ad6e-968e-416c-a532-1bff3f97eef1/volumes" Oct 30 03:50:48.161489 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:50:48.161455 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:51:02.562550 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:02.562511 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:51:02.563113 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:02.562782 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="35261b43-9fe0-45f2-b23e-d4a6f12cf1a1" containerName="otc-container" containerID="cri-o://46c17f4b065ae9b2ed08c06b0d071d79e469baf32bd61e19b243e465b496e517" gracePeriod=30 Oct 30 03:51:04.904522 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:04.904498 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:51:05.011215 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.011137 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-data-science-collector-ta-client-cert\") pod \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\" (UID: \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\") " Oct 30 03:51:05.011215 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.011178 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-otc-internal\" (UniqueName: \"kubernetes.io/configmap/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-otc-internal\") pod \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\" (UID: \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\") " Oct 30 03:51:05.011421 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.011262 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-monitoring-tls-certs\") pod \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\" (UID: \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\") " Oct 30 03:51:05.011421 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.011317 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-kube-api-access-psqz6\" (UniqueName: \"kubernetes.io/projected/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-kube-api-access-psqz6\") pod \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\" (UID: \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\") " Oct 30 03:51:05.011421 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.011340 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-tls-certs\") pod \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\" (UID: \"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1\") " Oct 30 03:51:05.011628 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.011602 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-otc-internal" pod "35261b43-9fe0-45f2-b23e-d4a6f12cf1a1" (UID: "35261b43-9fe0-45f2-b23e-d4a6f12cf1a1"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:51:05.013430 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.013403 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-monitoring-tls-certs" pod "35261b43-9fe0-45f2-b23e-d4a6f12cf1a1" (UID: "35261b43-9fe0-45f2-b23e-d4a6f12cf1a1"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:51:05.013561 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.013424 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-data-science-collector-ta-client-cert" pod "35261b43-9fe0-45f2-b23e-d4a6f12cf1a1" (UID: "35261b43-9fe0-45f2-b23e-d4a6f12cf1a1"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:51:05.013561 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.013429 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-kube-api-access-psqz6" pod "35261b43-9fe0-45f2-b23e-d4a6f12cf1a1" (UID: "35261b43-9fe0-45f2-b23e-d4a6f12cf1a1"). InnerVolumeSpecName "kube-api-access-psqz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:51:05.013561 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.013452 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-tls-certs" pod "35261b43-9fe0-45f2-b23e-d4a6f12cf1a1" (UID: "35261b43-9fe0-45f2-b23e-d4a6f12cf1a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:51:05.112928 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.112900 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:51:05.112928 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.112927 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:51:05.113119 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.112937 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:51:05.113119 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.112945 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psqz6\" (UniqueName: \"kubernetes.io/projected/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-kube-api-access-psqz6\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:51:05.113119 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.112954 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:51:05.610255 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.610218 4925 generic.go:356] "Generic (PLEG): container finished" podID="35261b43-9fe0-45f2-b23e-d4a6f12cf1a1" containerID="46c17f4b065ae9b2ed08c06b0d071d79e469baf32bd61e19b243e465b496e517" exitCode=0 Oct 30 03:51:05.610424 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.610287 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:51:05.610424 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.610305 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1","Type":"ContainerDied","Data":"46c17f4b065ae9b2ed08c06b0d071d79e469baf32bd61e19b243e465b496e517"} Oct 30 03:51:05.610424 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.610381 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"35261b43-9fe0-45f2-b23e-d4a6f12cf1a1","Type":"ContainerDied","Data":"a7a3548b06a356dd3a84155162e91bbc9e85596920b556c76b9e8e880fe6a24f"} Oct 30 03:51:05.610424 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.610405 4925 scope.go:122] "RemoveContainer" containerID="46c17f4b065ae9b2ed08c06b0d071d79e469baf32bd61e19b243e465b496e517" Oct 30 03:51:05.619085 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.619069 4925 scope.go:122] "RemoveContainer" containerID="46c17f4b065ae9b2ed08c06b0d071d79e469baf32bd61e19b243e465b496e517" Oct 30 03:51:05.619321 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:51:05.619303 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46c17f4b065ae9b2ed08c06b0d071d79e469baf32bd61e19b243e465b496e517\": container with ID starting with 46c17f4b065ae9b2ed08c06b0d071d79e469baf32bd61e19b243e465b496e517 not found: ID does not exist" containerID="46c17f4b065ae9b2ed08c06b0d071d79e469baf32bd61e19b243e465b496e517" Oct 30 03:51:05.619381 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.619326 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46c17f4b065ae9b2ed08c06b0d071d79e469baf32bd61e19b243e465b496e517"} err="failed to get container status \"46c17f4b065ae9b2ed08c06b0d071d79e469baf32bd61e19b243e465b496e517\": rpc error: code = NotFound desc = could not find container \"46c17f4b065ae9b2ed08c06b0d071d79e469baf32bd61e19b243e465b496e517\": container with ID starting with 46c17f4b065ae9b2ed08c06b0d071d79e469baf32bd61e19b243e465b496e517 not found: ID does not exist" Oct 30 03:51:05.662426 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.662399 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:51:05.675247 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.675224 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:51:05.702963 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.702926 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:51:05.703284 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.703272 4925 cpu_manager.go:414] "containerMap: removing container" podUID="35261b43-9fe0-45f2-b23e-d4a6f12cf1a1" containerName="otc-container" Oct 30 03:51:05.703325 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.703285 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35261b43-9fe0-45f2-b23e-d4a6f12cf1a1" containerName="otc-container" Oct 30 03:51:05.703325 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.703304 4925 cpu_manager.go:414] "containerMap: removing container" podUID="ed58ad6e-968e-416c-a532-1bff3f97eef1" containerName="tenant-metrics-exporter" Oct 30 03:51:05.703325 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.703309 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed58ad6e-968e-416c-a532-1bff3f97eef1" containerName="tenant-metrics-exporter" Oct 30 03:51:05.703417 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.703366 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="35261b43-9fe0-45f2-b23e-d4a6f12cf1a1" containerName="otc-container" Oct 30 03:51:05.703417 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.703375 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed58ad6e-968e-416c-a532-1bff3f97eef1" containerName="tenant-metrics-exporter" Oct 30 03:51:05.707781 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.707759 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:51:05.723448 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.723429 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\"" Oct 30 03:51:05.723548 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.723517 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 03:51:05.723548 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.723542 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 03:51:05.723706 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.723589 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:51:05.723706 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.723664 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 03:51:05.728337 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.728316 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:51:05.820025 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.819993 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6tqw\" (UniqueName: \"kubernetes.io/projected/ebd20321-4299-431a-afd2-e56c1b110b63-kube-api-access-c6tqw\") pod \"data-science-collector-collector-0\" (UID: \"ebd20321-4299-431a-afd2-e56c1b110b63\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:51:05.820146 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.820032 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ebd20321-4299-431a-afd2-e56c1b110b63\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:51:05.820219 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.820144 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ebd20321-4299-431a-afd2-e56c1b110b63\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:51:05.820219 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.820176 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ebd20321-4299-431a-afd2-e56c1b110b63-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ebd20321-4299-431a-afd2-e56c1b110b63\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:51:05.820298 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.820223 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ebd20321-4299-431a-afd2-e56c1b110b63\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:51:05.920891 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.920826 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6tqw\" (UniqueName: \"kubernetes.io/projected/ebd20321-4299-431a-afd2-e56c1b110b63-kube-api-access-c6tqw\") pod \"data-science-collector-collector-0\" (UID: \"ebd20321-4299-431a-afd2-e56c1b110b63\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:51:05.920891 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.920857 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ebd20321-4299-431a-afd2-e56c1b110b63\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:51:05.921366 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.920895 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ebd20321-4299-431a-afd2-e56c1b110b63\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:51:05.921366 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.920924 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ebd20321-4299-431a-afd2-e56c1b110b63-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ebd20321-4299-431a-afd2-e56c1b110b63\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:51:05.921366 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.920954 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ebd20321-4299-431a-afd2-e56c1b110b63\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:51:05.921622 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.921599 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ebd20321-4299-431a-afd2-e56c1b110b63-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ebd20321-4299-431a-afd2-e56c1b110b63\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:51:05.923303 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.923280 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ebd20321-4299-431a-afd2-e56c1b110b63\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:51:05.923793 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.923775 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ebd20321-4299-431a-afd2-e56c1b110b63\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:51:05.923886 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.923867 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ebd20321-4299-431a-afd2-e56c1b110b63\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:51:05.945082 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:05.945062 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6tqw\" (UniqueName: \"kubernetes.io/projected/ebd20321-4299-431a-afd2-e56c1b110b63-kube-api-access-c6tqw\") pod \"data-science-collector-collector-0\" (UID: \"ebd20321-4299-431a-afd2-e56c1b110b63\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:51:06.022657 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:06.022634 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:51:06.325100 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:06.325069 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:51:06.326659 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:51:06.326633 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd20321_4299_431a_afd2_e56c1b110b63.slice/crio-a88fe651e511d08c7588db8691f274e24c8037b1e242cb522bba24f4e38808a9 WatchSource:0}: Error finding container a88fe651e511d08c7588db8691f274e24c8037b1e242cb522bba24f4e38808a9: Status 404 returned error can't find the container with id a88fe651e511d08c7588db8691f274e24c8037b1e242cb522bba24f4e38808a9 Oct 30 03:51:06.328682 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:06.328656 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:51:06.328740 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:06.328708 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:51:06.617880 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:06.617838 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ebd20321-4299-431a-afd2-e56c1b110b63","Type":"ContainerStarted","Data":"58919e99e1d31ddfab14be9e41fc7fed78664bd3e23958a7a05061cd17298796"} Oct 30 03:51:06.618044 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:06.617898 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ebd20321-4299-431a-afd2-e56c1b110b63","Type":"ContainerStarted","Data":"a88fe651e511d08c7588db8691f274e24c8037b1e242cb522bba24f4e38808a9"} Oct 30 03:51:06.664110 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:06.664065 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.664053692 podStartE2EDuration="1.664053692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:06.662036452 +0000 UTC m=+4227.675314550" watchObservedRunningTime="2026-10-30 03:51:06.664053692 +0000 UTC m=+4227.677331794" Oct 30 03:51:07.118620 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:51:07.118590 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35261b43-9fe0-45f2-b23e-d4a6f12cf1a1" path="/var/lib/kubelet/pods/35261b43-9fe0-45f2-b23e-d4a6f12cf1a1/volumes" Oct 30 03:51:18.161172 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:51:18.161138 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:51:48.160593 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:51:48.160561 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:52:18.160395 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:52:18.160359 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:52:48.161400 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:52:48.161363 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:53:18.160810 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:53:18.160776 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:53:48.161295 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:53:48.161256 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:54:18.161862 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:54:18.161821 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:54:48.161347 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:54:48.161309 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:55:18.160208 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:55:18.160170 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:55:39.581229 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:55:39.581200 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:55:39.593224 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:55:39.593204 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:55:39.596955 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:55:39.596938 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 03:55:39.605323 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:55:39.605308 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 03:55:48.160567 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:55:48.160533 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:56:11.609564 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:11.609528 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7"] Oct 30 03:56:11.610108 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:11.609768 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" podUID="3b89f1c9-b82d-4bf9-9923-844990919dde" containerName="kube-rbac-proxy" containerID="cri-o://2877365544e74e090c756323c86e81d65fe7285498fcfd01dd55d2c7693a8955" gracePeriod=30 Oct 30 03:56:11.668929 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:11.668897 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw"] Oct 30 03:56:11.669101 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:11.669081 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" podUID="4609d443-3e1a-4d52-92e5-939f0f90fce7" containerName="korrel8r" containerID="cri-o://d7981097195b342a37b9e857cfb7711ca60f33b99debe24e59b4703d51b830fa" gracePeriod=30 Oct 30 03:56:11.725589 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:11.725555 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" podUID="4609d443-3e1a-4d52-92e5-939f0f90fce7" containerName="korrel8r" probeResult="failure" output="dial tcp 10.132.0.59:8443: connect: connection refused" Oct 30 03:56:11.908996 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:11.908973 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:56:11.940214 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:11.939798 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" Oct 30 03:56:12.050255 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.050226 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b89f1c9-b82d-4bf9-9923-844990919dde-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/3b89f1c9-b82d-4bf9-9923-844990919dde-prometheus-ca\") pod \"3b89f1c9-b82d-4bf9-9923-844990919dde\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " Oct 30 03:56:12.050551 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.050298 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-prometheus-client-cert\") pod \"3b89f1c9-b82d-4bf9-9923-844990919dde\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " Oct 30 03:56:12.050551 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.050344 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4609d443-3e1a-4d52-92e5-939f0f90fce7-config\" (UniqueName: \"kubernetes.io/configmap/4609d443-3e1a-4d52-92e5-939f0f90fce7-config\") pod \"4609d443-3e1a-4d52-92e5-939f0f90fce7\" (UID: \"4609d443-3e1a-4d52-92e5-939f0f90fce7\") " Oct 30 03:56:12.050551 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.050396 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4609d443-3e1a-4d52-92e5-939f0f90fce7-tls\" (UniqueName: \"kubernetes.io/secret/4609d443-3e1a-4d52-92e5-939f0f90fce7-tls\") pod \"4609d443-3e1a-4d52-92e5-939f0f90fce7\" (UID: \"4609d443-3e1a-4d52-92e5-939f0f90fce7\") " Oct 30 03:56:12.050551 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.050433 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-kube-rbac-proxy-config\") pod \"3b89f1c9-b82d-4bf9-9923-844990919dde\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " Oct 30 03:56:12.050551 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.050466 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4609d443-3e1a-4d52-92e5-939f0f90fce7-kube-api-access-fb2bg\" (UniqueName: \"kubernetes.io/projected/4609d443-3e1a-4d52-92e5-939f0f90fce7-kube-api-access-fb2bg\") pod \"4609d443-3e1a-4d52-92e5-939f0f90fce7\" (UID: \"4609d443-3e1a-4d52-92e5-939f0f90fce7\") " Oct 30 03:56:12.050551 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.050520 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-tls\" (UniqueName: \"kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-tls\") pod \"3b89f1c9-b82d-4bf9-9923-844990919dde\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " Oct 30 03:56:12.050930 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.050558 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b89f1c9-b82d-4bf9-9923-844990919dde-kube-api-access-g9mlc\" (UniqueName: \"kubernetes.io/projected/3b89f1c9-b82d-4bf9-9923-844990919dde-kube-api-access-g9mlc\") pod \"3b89f1c9-b82d-4bf9-9923-844990919dde\" (UID: \"3b89f1c9-b82d-4bf9-9923-844990919dde\") " Oct 30 03:56:12.050930 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.050679 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b89f1c9-b82d-4bf9-9923-844990919dde-prometheus-ca" pod "3b89f1c9-b82d-4bf9-9923-844990919dde" (UID: "3b89f1c9-b82d-4bf9-9923-844990919dde"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:12.050930 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.050856 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/3b89f1c9-b82d-4bf9-9923-844990919dde-prometheus-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.051279 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.051243 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4609d443-3e1a-4d52-92e5-939f0f90fce7-config" pod "4609d443-3e1a-4d52-92e5-939f0f90fce7" (UID: "4609d443-3e1a-4d52-92e5-939f0f90fce7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:12.053035 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.053014 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-prometheus-client-cert" pod "3b89f1c9-b82d-4bf9-9923-844990919dde" (UID: "3b89f1c9-b82d-4bf9-9923-844990919dde"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:12.053145 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.053049 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4609d443-3e1a-4d52-92e5-939f0f90fce7-tls" pod "4609d443-3e1a-4d52-92e5-939f0f90fce7" (UID: "4609d443-3e1a-4d52-92e5-939f0f90fce7"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:12.053145 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.053114 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-tls" pod "3b89f1c9-b82d-4bf9-9923-844990919dde" (UID: "3b89f1c9-b82d-4bf9-9923-844990919dde"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:12.053241 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.053190 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-kube-rbac-proxy-config" pod "3b89f1c9-b82d-4bf9-9923-844990919dde" (UID: "3b89f1c9-b82d-4bf9-9923-844990919dde"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:12.054451 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.054429 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b89f1c9-b82d-4bf9-9923-844990919dde-kube-api-access-g9mlc" pod "3b89f1c9-b82d-4bf9-9923-844990919dde" (UID: "3b89f1c9-b82d-4bf9-9923-844990919dde"). InnerVolumeSpecName "kube-api-access-g9mlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:12.054526 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.054458 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4609d443-3e1a-4d52-92e5-939f0f90fce7-kube-api-access-fb2bg" pod "4609d443-3e1a-4d52-92e5-939f0f90fce7" (UID: "4609d443-3e1a-4d52-92e5-939f0f90fce7"). InnerVolumeSpecName "kube-api-access-fb2bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:12.090378 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.090327 4925 generic.go:356] "Generic (PLEG): container finished" podID="4609d443-3e1a-4d52-92e5-939f0f90fce7" containerID="d7981097195b342a37b9e857cfb7711ca60f33b99debe24e59b4703d51b830fa" exitCode=0 Oct 30 03:56:12.090547 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.090527 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" event={"ID":"4609d443-3e1a-4d52-92e5-939f0f90fce7","Type":"ContainerDied","Data":"d7981097195b342a37b9e857cfb7711ca60f33b99debe24e59b4703d51b830fa"} Oct 30 03:56:12.090620 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.090580 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" event={"ID":"4609d443-3e1a-4d52-92e5-939f0f90fce7","Type":"ContainerDied","Data":"56db9d1e5d8296030c0813797551e8913d72be259464eb6aeffc068743a5d72a"} Oct 30 03:56:12.090620 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.090609 4925 scope.go:122] "RemoveContainer" containerID="d7981097195b342a37b9e857cfb7711ca60f33b99debe24e59b4703d51b830fa" Oct 30 03:56:12.090778 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.090743 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw" Oct 30 03:56:12.099472 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.095017 4925 generic.go:356] "Generic (PLEG): container finished" podID="3b89f1c9-b82d-4bf9-9923-844990919dde" containerID="2877365544e74e090c756323c86e81d65fe7285498fcfd01dd55d2c7693a8955" exitCode=0 Oct 30 03:56:12.099472 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.095058 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" event={"ID":"3b89f1c9-b82d-4bf9-9923-844990919dde","Type":"ContainerDied","Data":"2877365544e74e090c756323c86e81d65fe7285498fcfd01dd55d2c7693a8955"} Oct 30 03:56:12.099472 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.095085 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" Oct 30 03:56:12.099472 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.095105 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7" event={"ID":"3b89f1c9-b82d-4bf9-9923-844990919dde","Type":"ContainerDied","Data":"7337a10cf238290c1af071dbdcd5313db181836d75e01a6b54ee9cf6d877e201"} Oct 30 03:56:12.106692 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.106560 4925 scope.go:122] "RemoveContainer" containerID="d7981097195b342a37b9e857cfb7711ca60f33b99debe24e59b4703d51b830fa" Oct 30 03:56:12.107409 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:56:12.107245 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7981097195b342a37b9e857cfb7711ca60f33b99debe24e59b4703d51b830fa\": container with ID starting with d7981097195b342a37b9e857cfb7711ca60f33b99debe24e59b4703d51b830fa not found: ID does not exist" containerID="d7981097195b342a37b9e857cfb7711ca60f33b99debe24e59b4703d51b830fa" Oct 30 03:56:12.107409 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.107282 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7981097195b342a37b9e857cfb7711ca60f33b99debe24e59b4703d51b830fa"} err="failed to get container status \"d7981097195b342a37b9e857cfb7711ca60f33b99debe24e59b4703d51b830fa\": rpc error: code = NotFound desc = could not find container \"d7981097195b342a37b9e857cfb7711ca60f33b99debe24e59b4703d51b830fa\": container with ID starting with d7981097195b342a37b9e857cfb7711ca60f33b99debe24e59b4703d51b830fa not found: ID does not exist" Oct 30 03:56:12.107409 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.107305 4925 scope.go:122] "RemoveContainer" containerID="2877365544e74e090c756323c86e81d65fe7285498fcfd01dd55d2c7693a8955" Oct 30 03:56:12.116544 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.116522 4925 scope.go:122] "RemoveContainer" containerID="2877365544e74e090c756323c86e81d65fe7285498fcfd01dd55d2c7693a8955" Oct 30 03:56:12.116869 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:56:12.116839 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2877365544e74e090c756323c86e81d65fe7285498fcfd01dd55d2c7693a8955\": container with ID starting with 2877365544e74e090c756323c86e81d65fe7285498fcfd01dd55d2c7693a8955 not found: ID does not exist" containerID="2877365544e74e090c756323c86e81d65fe7285498fcfd01dd55d2c7693a8955" Oct 30 03:56:12.116945 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.116879 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2877365544e74e090c756323c86e81d65fe7285498fcfd01dd55d2c7693a8955"} err="failed to get container status \"2877365544e74e090c756323c86e81d65fe7285498fcfd01dd55d2c7693a8955\": rpc error: code = NotFound desc = could not find container \"2877365544e74e090c756323c86e81d65fe7285498fcfd01dd55d2c7693a8955\": container with ID starting with 2877365544e74e090c756323c86e81d65fe7285498fcfd01dd55d2c7693a8955 not found: ID does not exist" Oct 30 03:56:12.152038 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.151959 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9mlc\" (UniqueName: \"kubernetes.io/projected/3b89f1c9-b82d-4bf9-9923-844990919dde-kube-api-access-g9mlc\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.152038 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.151989 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-prometheus-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.152038 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.152005 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4609d443-3e1a-4d52-92e5-939f0f90fce7-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.152038 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.152020 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4609d443-3e1a-4d52-92e5-939f0f90fce7-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.152446 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.152032 4925 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-kube-rbac-proxy-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.152446 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.152302 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb2bg\" (UniqueName: \"kubernetes.io/projected/4609d443-3e1a-4d52-92e5-939f0f90fce7-kube-api-access-fb2bg\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.152446 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.152339 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3b89f1c9-b82d-4bf9-9923-844990919dde-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.170002 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.169967 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw"] Oct 30 03:56:12.176960 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.176931 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-rl5pw"] Oct 30 03:56:12.221082 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.221024 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7"] Oct 30 03:56:12.222763 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.222724 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 03:56:12.222983 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.222963 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="2d166da7-36f8-45ab-bdc9-2eb4347094f2" containerName="alertmanager" containerID="cri-o://8dd05a7901c22fc9c847dd9d9779d924e9bb1e3e870923bf04a79d40dd3403ac" gracePeriod=120 Oct 30 03:56:12.223051 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.223029 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="2d166da7-36f8-45ab-bdc9-2eb4347094f2" containerName="config-reloader" containerID="cri-o://cd66980e9ac45eb8f8a14e29aa9bc83be7fc0e43d63487aa763cf43ab642914f" gracePeriod=120 Oct 30 03:56:12.227998 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.227968 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-clrg7"] Oct 30 03:56:12.238875 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.238852 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 03:56:12.239125 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.239099 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="6436e346-6d25-4373-98f0-7b2eaedaf457" containerName="thanos-sidecar" containerID="cri-o://ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4" gracePeriod=600 Oct 30 03:56:12.239251 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.239099 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="6436e346-6d25-4373-98f0-7b2eaedaf457" containerName="prometheus" containerID="cri-o://96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e" gracePeriod=600 Oct 30 03:56:12.239366 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.239105 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="6436e346-6d25-4373-98f0-7b2eaedaf457" containerName="config-reloader" containerID="cri-o://5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea" gracePeriod=600 Oct 30 03:56:12.570128 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.570104 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:56:12.656514 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.656488 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"6436e346-6d25-4373-98f0-7b2eaedaf457\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " Oct 30 03:56:12.656888 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.656526 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6436e346-6d25-4373-98f0-7b2eaedaf457-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6436e346-6d25-4373-98f0-7b2eaedaf457-config-out\") pod \"6436e346-6d25-4373-98f0-7b2eaedaf457\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " Oct 30 03:56:12.656888 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.656555 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"6436e346-6d25-4373-98f0-7b2eaedaf457\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " Oct 30 03:56:12.656888 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.656573 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-config\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-config\") pod \"6436e346-6d25-4373-98f0-7b2eaedaf457\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " Oct 30 03:56:12.656888 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.656778 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"6436e346-6d25-4373-98f0-7b2eaedaf457\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " Oct 30 03:56:12.656888 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.656843 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6436e346-6d25-4373-98f0-7b2eaedaf457-tls-assets\" (UniqueName: \"kubernetes.io/projected/6436e346-6d25-4373-98f0-7b2eaedaf457-tls-assets\") pod \"6436e346-6d25-4373-98f0-7b2eaedaf457\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " Oct 30 03:56:12.656888 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.656879 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config\") pod \"6436e346-6d25-4373-98f0-7b2eaedaf457\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " Oct 30 03:56:12.657237 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.656895 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-2" pod "6436e346-6d25-4373-98f0-7b2eaedaf457" (UID: "6436e346-6d25-4373-98f0-7b2eaedaf457"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:12.657237 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.656912 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-secret-prometheus-web-tls-ca\") pod \"6436e346-6d25-4373-98f0-7b2eaedaf457\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " Oct 30 03:56:12.657237 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.656966 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6436e346-6d25-4373-98f0-7b2eaedaf457-kube-api-access-wvjs9\" (UniqueName: \"kubernetes.io/projected/6436e346-6d25-4373-98f0-7b2eaedaf457-kube-api-access-wvjs9\") pod \"6436e346-6d25-4373-98f0-7b2eaedaf457\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " Oct 30 03:56:12.657237 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.657065 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"6436e346-6d25-4373-98f0-7b2eaedaf457\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " Oct 30 03:56:12.657237 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.657135 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"6436e346-6d25-4373-98f0-7b2eaedaf457\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " Oct 30 03:56:12.657237 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.657192 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"6436e346-6d25-4373-98f0-7b2eaedaf457\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " Oct 30 03:56:12.657237 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.657235 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-thanos-prometheus-http-client-file\") pod \"6436e346-6d25-4373-98f0-7b2eaedaf457\" (UID: \"6436e346-6d25-4373-98f0-7b2eaedaf457\") " Oct 30 03:56:12.657954 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.657804 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-1" pod "6436e346-6d25-4373-98f0-7b2eaedaf457" (UID: "6436e346-6d25-4373-98f0-7b2eaedaf457"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:12.658492 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.658455 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-0" pod "6436e346-6d25-4373-98f0-7b2eaedaf457" (UID: "6436e346-6d25-4373-98f0-7b2eaedaf457"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:12.659136 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.659110 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "6436e346-6d25-4373-98f0-7b2eaedaf457" (UID: "6436e346-6d25-4373-98f0-7b2eaedaf457"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:12.659238 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.659210 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-config" pod "6436e346-6d25-4373-98f0-7b2eaedaf457" (UID: "6436e346-6d25-4373-98f0-7b2eaedaf457"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:12.659402 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.659383 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.660005 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.659980 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6436e346-6d25-4373-98f0-7b2eaedaf457-tls-assets" pod "6436e346-6d25-4373-98f0-7b2eaedaf457" (UID: "6436e346-6d25-4373-98f0-7b2eaedaf457"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:12.660223 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.660199 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6436e346-6d25-4373-98f0-7b2eaedaf457-config-out" pod "6436e346-6d25-4373-98f0-7b2eaedaf457" (UID: "6436e346-6d25-4373-98f0-7b2eaedaf457"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:12.660666 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.660635 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-thanos-prometheus-http-client-file" pod "6436e346-6d25-4373-98f0-7b2eaedaf457" (UID: "6436e346-6d25-4373-98f0-7b2eaedaf457"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:12.661717 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.661689 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "6436e346-6d25-4373-98f0-7b2eaedaf457" (UID: "6436e346-6d25-4373-98f0-7b2eaedaf457"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:12.661900 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.661870 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6436e346-6d25-4373-98f0-7b2eaedaf457-kube-api-access-wvjs9" pod "6436e346-6d25-4373-98f0-7b2eaedaf457" (UID: "6436e346-6d25-4373-98f0-7b2eaedaf457"). InnerVolumeSpecName "kube-api-access-wvjs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:12.662208 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.662186 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-secret-prometheus-web-tls-ca" pod "6436e346-6d25-4373-98f0-7b2eaedaf457" (UID: "6436e346-6d25-4373-98f0-7b2eaedaf457"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:12.663448 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.663428 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4" pod "6436e346-6d25-4373-98f0-7b2eaedaf457" (UID: "6436e346-6d25-4373-98f0-7b2eaedaf457"). InnerVolumeSpecName "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 03:56:12.674247 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.674221 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config" pod "6436e346-6d25-4373-98f0-7b2eaedaf457" (UID: "6436e346-6d25-4373-98f0-7b2eaedaf457"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:12.760056 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.760029 4925 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6436e346-6d25-4373-98f0-7b2eaedaf457-config-out\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.760056 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.760053 4925 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.760203 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.760064 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.760203 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.760073 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.760203 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.760083 4925 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6436e346-6d25-4373-98f0-7b2eaedaf457-tls-assets\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.760203 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.760091 4925 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.760203 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.760099 4925 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-secret-prometheus-web-tls-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.760203 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.760108 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvjs9\" (UniqueName: \"kubernetes.io/projected/6436e346-6d25-4373-98f0-7b2eaedaf457-kube-api-access-wvjs9\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.760203 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.760137 4925 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-139-19.ec2.internal\" " Oct 30 03:56:12.760203 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.760152 4925 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.760203 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.760166 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6436e346-6d25-4373-98f0-7b2eaedaf457-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.760203 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.760181 4925 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6436e346-6d25-4373-98f0-7b2eaedaf457-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:12.771290 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.771273 4925 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4") on node "ip-10-0-139-19.ec2.internal" Oct 30 03:56:12.860525 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.860498 4925 reconciler_common.go:299] "Volume detached for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"csi-ec2ccb9dcc2629e5a59ea91641dbb6b15d80e87697977b7c41508ff0003b2f36\"" Oct 30 03:56:12.903359 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.903331 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:56:12.903553 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:12.903531 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="ebd20321-4299-431a-afd2-e56c1b110b63" containerName="otc-container" containerID="cri-o://58919e99e1d31ddfab14be9e41fc7fed78664bd3e23958a7a05061cd17298796" gracePeriod=30 Oct 30 03:56:13.103104 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.103063 4925 generic.go:356] "Generic (PLEG): container finished" podID="2d166da7-36f8-45ab-bdc9-2eb4347094f2" containerID="cd66980e9ac45eb8f8a14e29aa9bc83be7fc0e43d63487aa763cf43ab642914f" exitCode=0 Oct 30 03:56:13.103104 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.103100 4925 generic.go:356] "Generic (PLEG): container finished" podID="2d166da7-36f8-45ab-bdc9-2eb4347094f2" containerID="8dd05a7901c22fc9c847dd9d9779d924e9bb1e3e870923bf04a79d40dd3403ac" exitCode=0 Oct 30 03:56:13.103268 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.103124 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2d166da7-36f8-45ab-bdc9-2eb4347094f2","Type":"ContainerDied","Data":"cd66980e9ac45eb8f8a14e29aa9bc83be7fc0e43d63487aa763cf43ab642914f"} Oct 30 03:56:13.103268 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.103179 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2d166da7-36f8-45ab-bdc9-2eb4347094f2","Type":"ContainerDied","Data":"8dd05a7901c22fc9c847dd9d9779d924e9bb1e3e870923bf04a79d40dd3403ac"} Oct 30 03:56:13.105209 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.105183 4925 generic.go:356] "Generic (PLEG): container finished" podID="ebd20321-4299-431a-afd2-e56c1b110b63" containerID="58919e99e1d31ddfab14be9e41fc7fed78664bd3e23958a7a05061cd17298796" exitCode=0 Oct 30 03:56:13.105374 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.105288 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ebd20321-4299-431a-afd2-e56c1b110b63","Type":"ContainerDied","Data":"58919e99e1d31ddfab14be9e41fc7fed78664bd3e23958a7a05061cd17298796"} Oct 30 03:56:13.108840 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.108807 4925 generic.go:356] "Generic (PLEG): container finished" podID="6436e346-6d25-4373-98f0-7b2eaedaf457" containerID="ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4" exitCode=0 Oct 30 03:56:13.108840 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.108835 4925 generic.go:356] "Generic (PLEG): container finished" podID="6436e346-6d25-4373-98f0-7b2eaedaf457" containerID="5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea" exitCode=0 Oct 30 03:56:13.108970 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.108849 4925 generic.go:356] "Generic (PLEG): container finished" podID="6436e346-6d25-4373-98f0-7b2eaedaf457" containerID="96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e" exitCode=0 Oct 30 03:56:13.108970 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.108884 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"6436e346-6d25-4373-98f0-7b2eaedaf457","Type":"ContainerDied","Data":"ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4"} Oct 30 03:56:13.108970 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.108937 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"6436e346-6d25-4373-98f0-7b2eaedaf457","Type":"ContainerDied","Data":"5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea"} Oct 30 03:56:13.109062 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.108973 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"6436e346-6d25-4373-98f0-7b2eaedaf457","Type":"ContainerDied","Data":"96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e"} Oct 30 03:56:13.109062 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.109008 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"6436e346-6d25-4373-98f0-7b2eaedaf457","Type":"ContainerDied","Data":"6085eaa1052aa7a8695353df34f4a0184cc0011f5215dbac11d3f2fc1c69dbcb"} Oct 30 03:56:13.109062 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.109051 4925 scope.go:122] "RemoveContainer" containerID="ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4" Oct 30 03:56:13.109223 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.109207 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 03:56:13.130164 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.130136 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b89f1c9-b82d-4bf9-9923-844990919dde" path="/var/lib/kubelet/pods/3b89f1c9-b82d-4bf9-9923-844990919dde/volumes" Oct 30 03:56:13.130968 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.130940 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4609d443-3e1a-4d52-92e5-939f0f90fce7" path="/var/lib/kubelet/pods/4609d443-3e1a-4d52-92e5-939f0f90fce7/volumes" Oct 30 03:56:13.140206 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.140181 4925 scope.go:122] "RemoveContainer" containerID="5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea" Oct 30 03:56:13.153465 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.153437 4925 scope.go:122] "RemoveContainer" containerID="96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e" Oct 30 03:56:13.162966 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.162862 4925 scope.go:122] "RemoveContainer" containerID="d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11" Oct 30 03:56:13.165995 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.165975 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:56:13.171190 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.171164 4925 scope.go:122] "RemoveContainer" containerID="ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4" Oct 30 03:56:13.171461 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:56:13.171441 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4\": container with ID starting with ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4 not found: ID does not exist" containerID="ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4" Oct 30 03:56:13.171526 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.171470 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4"} err="failed to get container status \"ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4\": rpc error: code = NotFound desc = could not find container \"ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4\": container with ID starting with ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4 not found: ID does not exist" Oct 30 03:56:13.171526 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.171487 4925 scope.go:122] "RemoveContainer" containerID="5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea" Oct 30 03:56:13.171792 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:56:13.171742 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea\": container with ID starting with 5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea not found: ID does not exist" containerID="5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea" Oct 30 03:56:13.171902 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.171800 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea"} err="failed to get container status \"5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea\": rpc error: code = NotFound desc = could not find container \"5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea\": container with ID starting with 5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea not found: ID does not exist" Oct 30 03:56:13.171902 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.171816 4925 scope.go:122] "RemoveContainer" containerID="96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e" Oct 30 03:56:13.172138 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:56:13.172118 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e\": container with ID starting with 96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e not found: ID does not exist" containerID="96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e" Oct 30 03:56:13.172209 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.172154 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e"} err="failed to get container status \"96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e\": rpc error: code = NotFound desc = could not find container \"96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e\": container with ID starting with 96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e not found: ID does not exist" Oct 30 03:56:13.172209 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.172171 4925 scope.go:122] "RemoveContainer" containerID="d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11" Oct 30 03:56:13.172483 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:56:13.172458 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11\": container with ID starting with d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11 not found: ID does not exist" containerID="d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11" Oct 30 03:56:13.172591 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.172487 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11"} err="failed to get container status \"d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11\": rpc error: code = NotFound desc = could not find container \"d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11\": container with ID starting with d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11 not found: ID does not exist" Oct 30 03:56:13.172591 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.172505 4925 scope.go:122] "RemoveContainer" containerID="ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4" Oct 30 03:56:13.172867 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.172784 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4"} err="failed to get container status \"ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4\": rpc error: code = NotFound desc = could not find container \"ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4\": container with ID starting with ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4 not found: ID does not exist" Oct 30 03:56:13.172867 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.172809 4925 scope.go:122] "RemoveContainer" containerID="5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea" Oct 30 03:56:13.173287 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.173262 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea"} err="failed to get container status \"5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea\": rpc error: code = NotFound desc = could not find container \"5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea\": container with ID starting with 5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea not found: ID does not exist" Oct 30 03:56:13.173384 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.173287 4925 scope.go:122] "RemoveContainer" containerID="96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e" Oct 30 03:56:13.173566 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.173542 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e"} err="failed to get container status \"96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e\": rpc error: code = NotFound desc = could not find container \"96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e\": container with ID starting with 96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e not found: ID does not exist" Oct 30 03:56:13.173652 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.173567 4925 scope.go:122] "RemoveContainer" containerID="d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11" Oct 30 03:56:13.173875 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.173848 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11"} err="failed to get container status \"d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11\": rpc error: code = NotFound desc = could not find container \"d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11\": container with ID starting with d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11 not found: ID does not exist" Oct 30 03:56:13.173952 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.173875 4925 scope.go:122] "RemoveContainer" containerID="ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4" Oct 30 03:56:13.174124 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.174106 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4"} err="failed to get container status \"ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4\": rpc error: code = NotFound desc = could not find container \"ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4\": container with ID starting with ce13a2298be69f4593c624cc3911d8596343b63fd9da9239727cdc47b2c573e4 not found: ID does not exist" Oct 30 03:56:13.174192 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.174124 4925 scope.go:122] "RemoveContainer" containerID="5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea" Oct 30 03:56:13.174358 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.174343 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea"} err="failed to get container status \"5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea\": rpc error: code = NotFound desc = could not find container \"5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea\": container with ID starting with 5035b9b43b8d63c25101eb697d958d60983d8168f1136d0d51023483737a71ea not found: ID does not exist" Oct 30 03:56:13.174425 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.174358 4925 scope.go:122] "RemoveContainer" containerID="96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e" Oct 30 03:56:13.174584 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.174567 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e"} err="failed to get container status \"96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e\": rpc error: code = NotFound desc = could not find container \"96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e\": container with ID starting with 96d581a524d172ddbbecbdd3161a434f721840e501f5de8d2fbc0b07f588f56e not found: ID does not exist" Oct 30 03:56:13.174641 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.174584 4925 scope.go:122] "RemoveContainer" containerID="d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11" Oct 30 03:56:13.174735 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.174720 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 03:56:13.174818 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.174769 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11"} err="failed to get container status \"d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11\": rpc error: code = NotFound desc = could not find container \"d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11\": container with ID starting with d68a18dfda780858d1aa3ad9a51a8a60154c63ca590f46cb21740181f353ed11 not found: ID does not exist" Oct 30 03:56:13.187065 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.187045 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 03:56:13.200742 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.200722 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:56:13.263116 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.263029 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-tls-certs\") pod \"ebd20321-4299-431a-afd2-e56c1b110b63\" (UID: \"ebd20321-4299-431a-afd2-e56c1b110b63\") " Oct 30 03:56:13.263116 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.263084 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebd20321-4299-431a-afd2-e56c1b110b63-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ebd20321-4299-431a-afd2-e56c1b110b63-otc-internal\") pod \"ebd20321-4299-431a-afd2-e56c1b110b63\" (UID: \"ebd20321-4299-431a-afd2-e56c1b110b63\") " Oct 30 03:56:13.263332 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.263118 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-data-science-collector-ta-client-cert\") pod \"ebd20321-4299-431a-afd2-e56c1b110b63\" (UID: \"ebd20321-4299-431a-afd2-e56c1b110b63\") " Oct 30 03:56:13.263332 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.263184 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd20321-4299-431a-afd2-e56c1b110b63-kube-api-access-c6tqw\" (UniqueName: \"kubernetes.io/projected/ebd20321-4299-431a-afd2-e56c1b110b63-kube-api-access-c6tqw\") pod \"ebd20321-4299-431a-afd2-e56c1b110b63\" (UID: \"ebd20321-4299-431a-afd2-e56c1b110b63\") " Oct 30 03:56:13.263332 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.263225 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-monitoring-tls-certs\") pod \"ebd20321-4299-431a-afd2-e56c1b110b63\" (UID: \"ebd20321-4299-431a-afd2-e56c1b110b63\") " Oct 30 03:56:13.263530 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.263498 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd20321-4299-431a-afd2-e56c1b110b63-otc-internal" pod "ebd20321-4299-431a-afd2-e56c1b110b63" (UID: "ebd20321-4299-431a-afd2-e56c1b110b63"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:13.265621 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.265597 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-tls-certs" pod "ebd20321-4299-431a-afd2-e56c1b110b63" (UID: "ebd20321-4299-431a-afd2-e56c1b110b63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:13.265621 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.265606 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd20321-4299-431a-afd2-e56c1b110b63-kube-api-access-c6tqw" pod "ebd20321-4299-431a-afd2-e56c1b110b63" (UID: "ebd20321-4299-431a-afd2-e56c1b110b63"). InnerVolumeSpecName "kube-api-access-c6tqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:13.265772 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.265632 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-monitoring-tls-certs" pod "ebd20321-4299-431a-afd2-e56c1b110b63" (UID: "ebd20321-4299-431a-afd2-e56c1b110b63"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:13.265772 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.265657 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-data-science-collector-ta-client-cert" pod "ebd20321-4299-431a-afd2-e56c1b110b63" (UID: "ebd20321-4299-431a-afd2-e56c1b110b63"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:13.363851 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.363820 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-web-config\" (UniqueName: \"kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-web-config\") pod \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " Oct 30 03:56:13.364029 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.363859 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d166da7-36f8-45ab-bdc9-2eb4347094f2-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/2d166da7-36f8-45ab-bdc9-2eb4347094f2-alertmanager-data-science-monitoringstack-db\") pod \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " Oct 30 03:56:13.364029 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.363878 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d166da7-36f8-45ab-bdc9-2eb4347094f2-tls-assets\" (UniqueName: \"kubernetes.io/projected/2d166da7-36f8-45ab-bdc9-2eb4347094f2-tls-assets\") pod \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " Oct 30 03:56:13.364029 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.363900 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d166da7-36f8-45ab-bdc9-2eb4347094f2-config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d166da7-36f8-45ab-bdc9-2eb4347094f2-config-out\") pod \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " Oct 30 03:56:13.364029 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.363949 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d166da7-36f8-45ab-bdc9-2eb4347094f2-kube-api-access-g4pwl\" (UniqueName: \"kubernetes.io/projected/2d166da7-36f8-45ab-bdc9-2eb4347094f2-kube-api-access-g4pwl\") pod \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " Oct 30 03:56:13.364029 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.363974 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-config-volume\" (UniqueName: \"kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-config-volume\") pod \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " Oct 30 03:56:13.364029 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.364024 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-cluster-tls-config\") pod \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\" (UID: \"2d166da7-36f8-45ab-bdc9-2eb4347094f2\") " Oct 30 03:56:13.364351 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.364244 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d166da7-36f8-45ab-bdc9-2eb4347094f2-alertmanager-data-science-monitoringstack-db" pod "2d166da7-36f8-45ab-bdc9-2eb4347094f2" (UID: "2d166da7-36f8-45ab-bdc9-2eb4347094f2"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:13.364351 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.364322 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6tqw\" (UniqueName: \"kubernetes.io/projected/ebd20321-4299-431a-afd2-e56c1b110b63-kube-api-access-c6tqw\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:13.364351 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.364342 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:13.364679 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.364355 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:13.364679 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.364607 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ebd20321-4299-431a-afd2-e56c1b110b63-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:13.364679 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.364622 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ebd20321-4299-431a-afd2-e56c1b110b63-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:13.366210 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.366184 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d166da7-36f8-45ab-bdc9-2eb4347094f2-config-out" pod "2d166da7-36f8-45ab-bdc9-2eb4347094f2" (UID: "2d166da7-36f8-45ab-bdc9-2eb4347094f2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:13.366392 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.366355 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d166da7-36f8-45ab-bdc9-2eb4347094f2-tls-assets" pod "2d166da7-36f8-45ab-bdc9-2eb4347094f2" (UID: "2d166da7-36f8-45ab-bdc9-2eb4347094f2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:13.366489 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.366446 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d166da7-36f8-45ab-bdc9-2eb4347094f2-kube-api-access-g4pwl" pod "2d166da7-36f8-45ab-bdc9-2eb4347094f2" (UID: "2d166da7-36f8-45ab-bdc9-2eb4347094f2"). InnerVolumeSpecName "kube-api-access-g4pwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:13.367087 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.367061 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-config-volume" pod "2d166da7-36f8-45ab-bdc9-2eb4347094f2" (UID: "2d166da7-36f8-45ab-bdc9-2eb4347094f2"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:13.370264 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.370242 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-cluster-tls-config" pod "2d166da7-36f8-45ab-bdc9-2eb4347094f2" (UID: "2d166da7-36f8-45ab-bdc9-2eb4347094f2"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:13.376809 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.376789 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-web-config" pod "2d166da7-36f8-45ab-bdc9-2eb4347094f2" (UID: "2d166da7-36f8-45ab-bdc9-2eb4347094f2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:13.465033 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.465006 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4pwl\" (UniqueName: \"kubernetes.io/projected/2d166da7-36f8-45ab-bdc9-2eb4347094f2-kube-api-access-g4pwl\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:13.465033 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.465031 4925 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-config-volume\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:13.465235 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.465043 4925 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-cluster-tls-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:13.465235 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.465072 4925 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d166da7-36f8-45ab-bdc9-2eb4347094f2-web-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:13.465235 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.465082 4925 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/2d166da7-36f8-45ab-bdc9-2eb4347094f2-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:13.465235 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.465091 4925 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d166da7-36f8-45ab-bdc9-2eb4347094f2-tls-assets\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:13.465235 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:13.465099 4925 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d166da7-36f8-45ab-bdc9-2eb4347094f2-config-out\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:56:14.115846 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:14.115814 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2d166da7-36f8-45ab-bdc9-2eb4347094f2","Type":"ContainerDied","Data":"5daaa2fbfc181e866b82de5df35fb4260f72c709bc42ddbfe05a02644b107ba7"} Oct 30 03:56:14.116245 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:14.115872 4925 scope.go:122] "RemoveContainer" containerID="cd66980e9ac45eb8f8a14e29aa9bc83be7fc0e43d63487aa763cf43ab642914f" Oct 30 03:56:14.116245 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:14.115831 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 03:56:14.117247 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:14.117228 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ebd20321-4299-431a-afd2-e56c1b110b63","Type":"ContainerDied","Data":"a88fe651e511d08c7588db8691f274e24c8037b1e242cb522bba24f4e38808a9"} Oct 30 03:56:14.117322 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:14.117299 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:56:14.126336 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:14.126319 4925 scope.go:122] "RemoveContainer" containerID="8dd05a7901c22fc9c847dd9d9779d924e9bb1e3e870923bf04a79d40dd3403ac" Oct 30 03:56:14.134909 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:14.134887 4925 scope.go:122] "RemoveContainer" containerID="73b1de764e0b3602bc7e247249fc507ebac8a4f274148a9e8e40876dcfd2c694" Oct 30 03:56:14.145543 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:14.145523 4925 scope.go:122] "RemoveContainer" containerID="58919e99e1d31ddfab14be9e41fc7fed78664bd3e23958a7a05061cd17298796" Oct 30 03:56:14.168735 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:14.168713 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:56:14.174578 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:14.174558 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:56:14.210534 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:14.210511 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 03:56:14.214809 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:14.214791 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 03:56:15.119575 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:15.119541 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d166da7-36f8-45ab-bdc9-2eb4347094f2" path="/var/lib/kubelet/pods/2d166da7-36f8-45ab-bdc9-2eb4347094f2/volumes" Oct 30 03:56:15.120114 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:15.120098 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6436e346-6d25-4373-98f0-7b2eaedaf457" path="/var/lib/kubelet/pods/6436e346-6d25-4373-98f0-7b2eaedaf457/volumes" Oct 30 03:56:15.120852 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:15.120837 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebd20321-4299-431a-afd2-e56c1b110b63" path="/var/lib/kubelet/pods/ebd20321-4299-431a-afd2-e56c1b110b63/volumes" Oct 30 03:56:18.138798 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.138768 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-bpzvl"] Oct 30 03:56:18.139197 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139087 4925 cpu_manager.go:414] "containerMap: removing container" podUID="2d166da7-36f8-45ab-bdc9-2eb4347094f2" containerName="alertmanager" Oct 30 03:56:18.139197 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139098 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d166da7-36f8-45ab-bdc9-2eb4347094f2" containerName="alertmanager" Oct 30 03:56:18.139197 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139114 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6436e346-6d25-4373-98f0-7b2eaedaf457" containerName="config-reloader" Oct 30 03:56:18.139197 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139119 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6436e346-6d25-4373-98f0-7b2eaedaf457" containerName="config-reloader" Oct 30 03:56:18.139197 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139126 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6436e346-6d25-4373-98f0-7b2eaedaf457" containerName="thanos-sidecar" Oct 30 03:56:18.139197 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139132 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6436e346-6d25-4373-98f0-7b2eaedaf457" containerName="thanos-sidecar" Oct 30 03:56:18.139197 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139150 4925 cpu_manager.go:414] "containerMap: removing container" podUID="4609d443-3e1a-4d52-92e5-939f0f90fce7" containerName="korrel8r" Oct 30 03:56:18.139197 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139155 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4609d443-3e1a-4d52-92e5-939f0f90fce7" containerName="korrel8r" Oct 30 03:56:18.139197 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139162 4925 cpu_manager.go:414] "containerMap: removing container" podUID="3b89f1c9-b82d-4bf9-9923-844990919dde" containerName="kube-rbac-proxy" Oct 30 03:56:18.139197 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139167 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b89f1c9-b82d-4bf9-9923-844990919dde" containerName="kube-rbac-proxy" Oct 30 03:56:18.139197 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139179 4925 cpu_manager.go:414] "containerMap: removing container" podUID="2d166da7-36f8-45ab-bdc9-2eb4347094f2" containerName="init-config-reloader" Oct 30 03:56:18.139197 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139183 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d166da7-36f8-45ab-bdc9-2eb4347094f2" containerName="init-config-reloader" Oct 30 03:56:18.139197 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139193 4925 cpu_manager.go:414] "containerMap: removing container" podUID="2d166da7-36f8-45ab-bdc9-2eb4347094f2" containerName="config-reloader" Oct 30 03:56:18.139197 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139199 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d166da7-36f8-45ab-bdc9-2eb4347094f2" containerName="config-reloader" Oct 30 03:56:18.139639 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139207 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6436e346-6d25-4373-98f0-7b2eaedaf457" containerName="init-config-reloader" Oct 30 03:56:18.139639 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139212 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6436e346-6d25-4373-98f0-7b2eaedaf457" containerName="init-config-reloader" Oct 30 03:56:18.139639 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139226 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6436e346-6d25-4373-98f0-7b2eaedaf457" containerName="prometheus" Oct 30 03:56:18.139639 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139232 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6436e346-6d25-4373-98f0-7b2eaedaf457" containerName="prometheus" Oct 30 03:56:18.139639 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139239 4925 cpu_manager.go:414] "containerMap: removing container" podUID="ebd20321-4299-431a-afd2-e56c1b110b63" containerName="otc-container" Oct 30 03:56:18.139639 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139243 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebd20321-4299-431a-afd2-e56c1b110b63" containerName="otc-container" Oct 30 03:56:18.139639 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139292 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6436e346-6d25-4373-98f0-7b2eaedaf457" containerName="prometheus" Oct 30 03:56:18.139639 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139303 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebd20321-4299-431a-afd2-e56c1b110b63" containerName="otc-container" Oct 30 03:56:18.139639 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139313 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d166da7-36f8-45ab-bdc9-2eb4347094f2" containerName="alertmanager" Oct 30 03:56:18.139639 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139318 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6436e346-6d25-4373-98f0-7b2eaedaf457" containerName="config-reloader" Oct 30 03:56:18.139639 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139323 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6436e346-6d25-4373-98f0-7b2eaedaf457" containerName="thanos-sidecar" Oct 30 03:56:18.139639 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139328 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4609d443-3e1a-4d52-92e5-939f0f90fce7" containerName="korrel8r" Oct 30 03:56:18.139639 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139332 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b89f1c9-b82d-4bf9-9923-844990919dde" containerName="kube-rbac-proxy" Oct 30 03:56:18.139639 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.139339 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d166da7-36f8-45ab-bdc9-2eb4347094f2" containerName="config-reloader" Oct 30 03:56:18.149582 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.149555 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" Oct 30 03:56:18.163314 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.163275 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-bpzvl"] Oct 30 03:56:18.165227 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.165198 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 03:56:18.168190 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:56:18.168160 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:56:18.173628 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.173605 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 03:56:18.177405 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.177384 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 03:56:18.177498 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.177442 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-pbw4x\"" Oct 30 03:56:18.177498 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.177457 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 03:56:18.306162 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.306124 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-config\") pod \"korrel8r-6644769df5-bpzvl\" (UID: \"2ba0f69e-1f25-44be-b313-bcc9e14ac3c3\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" Oct 30 03:56:18.306383 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.306179 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5kv6\" (UniqueName: \"kubernetes.io/projected/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-kube-api-access-d5kv6\") pod \"korrel8r-6644769df5-bpzvl\" (UID: \"2ba0f69e-1f25-44be-b313-bcc9e14ac3c3\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" Oct 30 03:56:18.306383 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.306272 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-tls\") pod \"korrel8r-6644769df5-bpzvl\" (UID: \"2ba0f69e-1f25-44be-b313-bcc9e14ac3c3\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" Oct 30 03:56:18.407271 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.407172 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-config\") pod \"korrel8r-6644769df5-bpzvl\" (UID: \"2ba0f69e-1f25-44be-b313-bcc9e14ac3c3\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" Oct 30 03:56:18.407271 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.407238 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5kv6\" (UniqueName: \"kubernetes.io/projected/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-kube-api-access-d5kv6\") pod \"korrel8r-6644769df5-bpzvl\" (UID: \"2ba0f69e-1f25-44be-b313-bcc9e14ac3c3\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" Oct 30 03:56:18.407500 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.407350 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-tls\") pod \"korrel8r-6644769df5-bpzvl\" (UID: \"2ba0f69e-1f25-44be-b313-bcc9e14ac3c3\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" Oct 30 03:56:18.407963 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.407938 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-config\") pod \"korrel8r-6644769df5-bpzvl\" (UID: \"2ba0f69e-1f25-44be-b313-bcc9e14ac3c3\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" Oct 30 03:56:18.409726 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.409699 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-tls\") pod \"korrel8r-6644769df5-bpzvl\" (UID: \"2ba0f69e-1f25-44be-b313-bcc9e14ac3c3\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" Oct 30 03:56:18.431234 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.431205 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5kv6\" (UniqueName: \"kubernetes.io/projected/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-kube-api-access-d5kv6\") pod \"korrel8r-6644769df5-bpzvl\" (UID: \"2ba0f69e-1f25-44be-b313-bcc9e14ac3c3\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" Oct 30 03:56:18.465921 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.465886 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" Oct 30 03:56:18.766451 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.766426 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-bpzvl"] Oct 30 03:56:18.768788 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:56:18.768746 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba0f69e_1f25_44be_b313_bcc9e14ac3c3.slice/crio-05e9bb8740e4a336dee32d49293e2dec6b2d45c577f66d79a639ba0161440d00 WatchSource:0}: Error finding container 05e9bb8740e4a336dee32d49293e2dec6b2d45c577f66d79a639ba0161440d00: Status 404 returned error can't find the container with id 05e9bb8740e4a336dee32d49293e2dec6b2d45c577f66d79a639ba0161440d00 Oct 30 03:56:18.889779 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.889722 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:56:18.893781 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.893743 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:56:18.921734 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.921703 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-ffv7w\"" Oct 30 03:56:18.921895 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.921807 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-645168e1\"" Oct 30 03:56:18.921895 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.921819 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:56:18.923362 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:18.923341 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:56:19.012778 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:19.012716 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:56:19.012778 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:19.012768 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:56:19.012985 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:19.012900 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmw9j\" (UniqueName: \"kubernetes.io/projected/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-kube-api-access-dmw9j\") pod \"data-science-collector-collector-0\" (UID: \"cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:56:19.113912 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:19.113874 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmw9j\" (UniqueName: \"kubernetes.io/projected/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-kube-api-access-dmw9j\") pod \"data-science-collector-collector-0\" (UID: \"cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:56:19.114129 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:19.113927 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:56:19.114129 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:19.113959 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:56:19.114801 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:19.114777 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:56:19.116952 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:19.116927 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:56:19.137531 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:19.137501 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmw9j\" (UniqueName: \"kubernetes.io/projected/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-kube-api-access-dmw9j\") pod \"data-science-collector-collector-0\" (UID: \"cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:56:19.149292 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:19.149265 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" event={"ID":"2ba0f69e-1f25-44be-b313-bcc9e14ac3c3","Type":"ContainerStarted","Data":"2a385fcce0ebbc5e802044cbb3de4cb68e2f51760c46f99c995b803c49213ef7"} Oct 30 03:56:19.149639 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:19.149313 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" event={"ID":"2ba0f69e-1f25-44be-b313-bcc9e14ac3c3","Type":"ContainerStarted","Data":"05e9bb8740e4a336dee32d49293e2dec6b2d45c577f66d79a639ba0161440d00"} Oct 30 03:56:19.149639 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:19.149392 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" Oct 30 03:56:19.193952 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:19.193905 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" podStartSLOduration=1.193893282 podStartE2EDuration="1.193893282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:19.192566728 +0000 UTC m=+4540.205844831" watchObservedRunningTime="2026-10-30 03:56:19.193893282 +0000 UTC m=+4540.207171385" Oct 30 03:56:19.207966 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:19.207940 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:56:19.509738 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:19.509711 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:56:19.511634 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:56:19.511606 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbaeb11d_a4ac_4b5c_bd14_05e4cb1a284c.slice/crio-f955123dac8ea3282402d10460363f3af2843316ce0116adac593cff8bcc71ce WatchSource:0}: Error finding container f955123dac8ea3282402d10460363f3af2843316ce0116adac593cff8bcc71ce: Status 404 returned error can't find the container with id f955123dac8ea3282402d10460363f3af2843316ce0116adac593cff8bcc71ce Oct 30 03:56:19.513893 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:19.513866 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:56:19.513995 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:19.513928 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:56:20.155844 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:20.155810 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c","Type":"ContainerStarted","Data":"33aad113295d750f5502b7d19eb98323f8d69eed20e4409c11c968b2665edc9b"} Oct 30 03:56:20.156267 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:20.155864 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c","Type":"ContainerStarted","Data":"f955123dac8ea3282402d10460363f3af2843316ce0116adac593cff8bcc71ce"} Oct 30 03:56:20.199572 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:20.199523 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.199511946 podStartE2EDuration="2.199511946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:20.198139024 +0000 UTC m=+4541.211417120" watchObservedRunningTime="2026-10-30 03:56:20.199511946 +0000 UTC m=+4541.212790099" Oct 30 03:56:30.158362 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:56:30.158292 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" Oct 30 03:56:48.161659 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:56:48.161616 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:57:13.972993 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:13.972961 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-bpzvl"] Oct 30 03:57:13.973510 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:13.973171 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" podUID="2ba0f69e-1f25-44be-b313-bcc9e14ac3c3" containerName="korrel8r" containerID="cri-o://2a385fcce0ebbc5e802044cbb3de4cb68e2f51760c46f99c995b803c49213ef7" gracePeriod=30 Oct 30 03:57:14.218568 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.218542 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" Oct 30 03:57:14.335892 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.335861 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-tls\" (UniqueName: \"kubernetes.io/secret/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-tls\") pod \"2ba0f69e-1f25-44be-b313-bcc9e14ac3c3\" (UID: \"2ba0f69e-1f25-44be-b313-bcc9e14ac3c3\") " Oct 30 03:57:14.336051 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.335932 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-kube-api-access-d5kv6\" (UniqueName: \"kubernetes.io/projected/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-kube-api-access-d5kv6\") pod \"2ba0f69e-1f25-44be-b313-bcc9e14ac3c3\" (UID: \"2ba0f69e-1f25-44be-b313-bcc9e14ac3c3\") " Oct 30 03:57:14.336051 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.336015 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-config\" (UniqueName: \"kubernetes.io/configmap/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-config\") pod \"2ba0f69e-1f25-44be-b313-bcc9e14ac3c3\" (UID: \"2ba0f69e-1f25-44be-b313-bcc9e14ac3c3\") " Oct 30 03:57:14.336385 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.336357 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-config" pod "2ba0f69e-1f25-44be-b313-bcc9e14ac3c3" (UID: "2ba0f69e-1f25-44be-b313-bcc9e14ac3c3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:57:14.337824 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.337800 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-tls" pod "2ba0f69e-1f25-44be-b313-bcc9e14ac3c3" (UID: "2ba0f69e-1f25-44be-b313-bcc9e14ac3c3"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:57:14.337989 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.337969 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-kube-api-access-d5kv6" pod "2ba0f69e-1f25-44be-b313-bcc9e14ac3c3" (UID: "2ba0f69e-1f25-44be-b313-bcc9e14ac3c3"). InnerVolumeSpecName "kube-api-access-d5kv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:57:14.393792 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.386441 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:57:14.393792 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.386664 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c" containerName="otc-container" containerID="cri-o://33aad113295d750f5502b7d19eb98323f8d69eed20e4409c11c968b2665edc9b" gracePeriod=30 Oct 30 03:57:14.422477 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.422441 4925 generic.go:356] "Generic (PLEG): container finished" podID="2ba0f69e-1f25-44be-b313-bcc9e14ac3c3" containerID="2a385fcce0ebbc5e802044cbb3de4cb68e2f51760c46f99c995b803c49213ef7" exitCode=0 Oct 30 03:57:14.422647 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.422495 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" Oct 30 03:57:14.422647 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.422519 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" event={"ID":"2ba0f69e-1f25-44be-b313-bcc9e14ac3c3","Type":"ContainerDied","Data":"2a385fcce0ebbc5e802044cbb3de4cb68e2f51760c46f99c995b803c49213ef7"} Oct 30 03:57:14.422647 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.422599 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-bpzvl" event={"ID":"2ba0f69e-1f25-44be-b313-bcc9e14ac3c3","Type":"ContainerDied","Data":"05e9bb8740e4a336dee32d49293e2dec6b2d45c577f66d79a639ba0161440d00"} Oct 30 03:57:14.422647 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.422626 4925 scope.go:122] "RemoveContainer" containerID="2a385fcce0ebbc5e802044cbb3de4cb68e2f51760c46f99c995b803c49213ef7" Oct 30 03:57:14.437073 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.437050 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:57:14.437073 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.437073 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:57:14.437232 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.437084 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5kv6\" (UniqueName: \"kubernetes.io/projected/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3-kube-api-access-d5kv6\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:57:14.501391 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.501355 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-bpzvl"] Oct 30 03:57:14.502443 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.502424 4925 scope.go:122] "RemoveContainer" containerID="2a385fcce0ebbc5e802044cbb3de4cb68e2f51760c46f99c995b803c49213ef7" Oct 30 03:57:14.502838 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:57:14.502816 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a385fcce0ebbc5e802044cbb3de4cb68e2f51760c46f99c995b803c49213ef7\": container with ID starting with 2a385fcce0ebbc5e802044cbb3de4cb68e2f51760c46f99c995b803c49213ef7 not found: ID does not exist" containerID="2a385fcce0ebbc5e802044cbb3de4cb68e2f51760c46f99c995b803c49213ef7" Oct 30 03:57:14.502934 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.502845 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a385fcce0ebbc5e802044cbb3de4cb68e2f51760c46f99c995b803c49213ef7"} err="failed to get container status \"2a385fcce0ebbc5e802044cbb3de4cb68e2f51760c46f99c995b803c49213ef7\": rpc error: code = NotFound desc = could not find container \"2a385fcce0ebbc5e802044cbb3de4cb68e2f51760c46f99c995b803c49213ef7\": container with ID starting with 2a385fcce0ebbc5e802044cbb3de4cb68e2f51760c46f99c995b803c49213ef7 not found: ID does not exist" Oct 30 03:57:14.503583 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.503563 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-bpzvl"] Oct 30 03:57:14.643177 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.643153 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:14.740513 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.740479 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-monitoring-tls-certs\") pod \"cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c\" (UID: \"cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c\") " Oct 30 03:57:14.740691 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.740529 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-otc-internal\" (UniqueName: \"kubernetes.io/configmap/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-otc-internal\") pod \"cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c\" (UID: \"cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c\") " Oct 30 03:57:14.740691 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.740548 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-kube-api-access-dmw9j\" (UniqueName: \"kubernetes.io/projected/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-kube-api-access-dmw9j\") pod \"cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c\" (UID: \"cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c\") " Oct 30 03:57:14.740906 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.740882 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-otc-internal" pod "cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c" (UID: "cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:57:14.742619 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.742593 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-monitoring-tls-certs" pod "cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c" (UID: "cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:57:14.742712 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.742697 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-kube-api-access-dmw9j" pod "cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c" (UID: "cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c"). InnerVolumeSpecName "kube-api-access-dmw9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:57:14.842122 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.842084 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:57:14.842122 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.842120 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:57:14.842122 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:14.842130 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmw9j\" (UniqueName: \"kubernetes.io/projected/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c-kube-api-access-dmw9j\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:57:15.119490 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:15.119458 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ba0f69e-1f25-44be-b313-bcc9e14ac3c3" path="/var/lib/kubelet/pods/2ba0f69e-1f25-44be-b313-bcc9e14ac3c3/volumes" Oct 30 03:57:15.429860 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:15.429824 4925 generic.go:356] "Generic (PLEG): container finished" podID="cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c" containerID="33aad113295d750f5502b7d19eb98323f8d69eed20e4409c11c968b2665edc9b" exitCode=0 Oct 30 03:57:15.430036 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:15.429867 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:15.430036 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:15.429910 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c","Type":"ContainerDied","Data":"33aad113295d750f5502b7d19eb98323f8d69eed20e4409c11c968b2665edc9b"} Oct 30 03:57:15.430036 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:15.429957 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c","Type":"ContainerDied","Data":"f955123dac8ea3282402d10460363f3af2843316ce0116adac593cff8bcc71ce"} Oct 30 03:57:15.430036 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:15.429981 4925 scope.go:122] "RemoveContainer" containerID="33aad113295d750f5502b7d19eb98323f8d69eed20e4409c11c968b2665edc9b" Oct 30 03:57:15.438487 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:15.438468 4925 scope.go:122] "RemoveContainer" containerID="33aad113295d750f5502b7d19eb98323f8d69eed20e4409c11c968b2665edc9b" Oct 30 03:57:15.438759 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:57:15.438721 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33aad113295d750f5502b7d19eb98323f8d69eed20e4409c11c968b2665edc9b\": container with ID starting with 33aad113295d750f5502b7d19eb98323f8d69eed20e4409c11c968b2665edc9b not found: ID does not exist" containerID="33aad113295d750f5502b7d19eb98323f8d69eed20e4409c11c968b2665edc9b" Oct 30 03:57:15.438815 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:15.438788 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33aad113295d750f5502b7d19eb98323f8d69eed20e4409c11c968b2665edc9b"} err="failed to get container status \"33aad113295d750f5502b7d19eb98323f8d69eed20e4409c11c968b2665edc9b\": rpc error: code = NotFound desc = could not find container \"33aad113295d750f5502b7d19eb98323f8d69eed20e4409c11c968b2665edc9b\": container with ID starting with 33aad113295d750f5502b7d19eb98323f8d69eed20e4409c11c968b2665edc9b not found: ID does not exist" Oct 30 03:57:15.481223 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:15.481191 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:57:15.493710 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:15.493681 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:57:17.118922 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.118886 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c" path="/var/lib/kubelet/pods/cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c/volumes" Oct 30 03:57:17.361950 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.361919 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-x4zdz"] Oct 30 03:57:17.362251 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.362239 4925 cpu_manager.go:414] "containerMap: removing container" podUID="cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c" containerName="otc-container" Oct 30 03:57:17.362297 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.362251 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c" containerName="otc-container" Oct 30 03:57:17.362297 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.362272 4925 cpu_manager.go:414] "containerMap: removing container" podUID="2ba0f69e-1f25-44be-b313-bcc9e14ac3c3" containerName="korrel8r" Oct 30 03:57:17.362297 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.362278 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba0f69e-1f25-44be-b313-bcc9e14ac3c3" containerName="korrel8r" Oct 30 03:57:17.362392 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.362327 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbaeb11d-a4ac-4b5c-bd14-05e4cb1a284c" containerName="otc-container" Oct 30 03:57:17.362392 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.362338 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba0f69e-1f25-44be-b313-bcc9e14ac3c3" containerName="korrel8r" Oct 30 03:57:17.366510 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.366489 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" Oct 30 03:57:17.398240 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.398204 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 03:57:17.398240 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.398209 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 03:57:17.398405 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.398281 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 03:57:17.398405 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.398288 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 03:57:17.398581 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.398562 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-xd5vq\"" Oct 30 03:57:17.399792 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.399773 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-x4zdz"] Oct 30 03:57:17.466997 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.466964 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f87a7847-84fe-4313-8f58-d6248b90cd61-config\") pod \"korrel8r-6644769df5-x4zdz\" (UID: \"f87a7847-84fe-4313-8f58-d6248b90cd61\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" Oct 30 03:57:17.467125 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.467072 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cphd\" (UniqueName: \"kubernetes.io/projected/f87a7847-84fe-4313-8f58-d6248b90cd61-kube-api-access-8cphd\") pod \"korrel8r-6644769df5-x4zdz\" (UID: \"f87a7847-84fe-4313-8f58-d6248b90cd61\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" Oct 30 03:57:17.467125 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.467102 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f87a7847-84fe-4313-8f58-d6248b90cd61-tls\") pod \"korrel8r-6644769df5-x4zdz\" (UID: \"f87a7847-84fe-4313-8f58-d6248b90cd61\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" Oct 30 03:57:17.567582 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.567548 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cphd\" (UniqueName: \"kubernetes.io/projected/f87a7847-84fe-4313-8f58-d6248b90cd61-kube-api-access-8cphd\") pod \"korrel8r-6644769df5-x4zdz\" (UID: \"f87a7847-84fe-4313-8f58-d6248b90cd61\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" Oct 30 03:57:17.567582 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.567581 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f87a7847-84fe-4313-8f58-d6248b90cd61-tls\") pod \"korrel8r-6644769df5-x4zdz\" (UID: \"f87a7847-84fe-4313-8f58-d6248b90cd61\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" Oct 30 03:57:17.567737 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.567686 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f87a7847-84fe-4313-8f58-d6248b90cd61-config\") pod \"korrel8r-6644769df5-x4zdz\" (UID: \"f87a7847-84fe-4313-8f58-d6248b90cd61\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" Oct 30 03:57:17.568420 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.568399 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f87a7847-84fe-4313-8f58-d6248b90cd61-config\") pod \"korrel8r-6644769df5-x4zdz\" (UID: \"f87a7847-84fe-4313-8f58-d6248b90cd61\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" Oct 30 03:57:17.570147 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.570125 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f87a7847-84fe-4313-8f58-d6248b90cd61-tls\") pod \"korrel8r-6644769df5-x4zdz\" (UID: \"f87a7847-84fe-4313-8f58-d6248b90cd61\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" Oct 30 03:57:17.591081 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.591055 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cphd\" (UniqueName: \"kubernetes.io/projected/f87a7847-84fe-4313-8f58-d6248b90cd61-kube-api-access-8cphd\") pod \"korrel8r-6644769df5-x4zdz\" (UID: \"f87a7847-84fe-4313-8f58-d6248b90cd61\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" Oct 30 03:57:17.679093 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.679014 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" Oct 30 03:57:17.980648 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.980586 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-x4zdz"] Oct 30 03:57:17.984644 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:57:17.984584 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87a7847_84fe_4313_8f58_d6248b90cd61.slice/crio-14209525c862c40f3e9856af0003cbfbb3b6d0038135ea97d97574c31608882f WatchSource:0}: Error finding container 14209525c862c40f3e9856af0003cbfbb3b6d0038135ea97d97574c31608882f: Status 404 returned error can't find the container with id 14209525c862c40f3e9856af0003cbfbb3b6d0038135ea97d97574c31608882f Oct 30 03:57:17.989858 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.989834 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:57:17.994527 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:17.994500 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:18.016259 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:18.015976 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-645168e1\"" Oct 30 03:57:18.016259 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:18.016002 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:57:18.016673 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:18.016654 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:57:18.073327 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:18.073299 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2e2d41a8-f51c-490b-a1fe-58dde1095ba3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:18.073480 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:18.073367 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv9lx\" (UniqueName: \"kubernetes.io/projected/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-kube-api-access-hv9lx\") pod \"data-science-collector-collector-0\" (UID: \"2e2d41a8-f51c-490b-a1fe-58dde1095ba3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:18.073480 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:18.073428 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"2e2d41a8-f51c-490b-a1fe-58dde1095ba3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:18.173633 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:57:18.173594 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:57:18.174043 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:18.173863 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2e2d41a8-f51c-490b-a1fe-58dde1095ba3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:18.174043 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:18.173940 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv9lx\" (UniqueName: \"kubernetes.io/projected/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-kube-api-access-hv9lx\") pod \"data-science-collector-collector-0\" (UID: \"2e2d41a8-f51c-490b-a1fe-58dde1095ba3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:18.174043 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:18.173974 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"2e2d41a8-f51c-490b-a1fe-58dde1095ba3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:18.174618 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:18.174584 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"2e2d41a8-f51c-490b-a1fe-58dde1095ba3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:18.176373 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:18.176352 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2e2d41a8-f51c-490b-a1fe-58dde1095ba3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:18.197593 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:18.197568 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv9lx\" (UniqueName: \"kubernetes.io/projected/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-kube-api-access-hv9lx\") pod \"data-science-collector-collector-0\" (UID: \"2e2d41a8-f51c-490b-a1fe-58dde1095ba3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:18.310298 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:18.310223 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:18.448381 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:18.448345 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" event={"ID":"f87a7847-84fe-4313-8f58-d6248b90cd61","Type":"ContainerStarted","Data":"33df3e894a67d6a113a307683e471c3f88d35fe2718ad50aaec1df025bec5215"} Oct 30 03:57:18.448553 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:18.448404 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" event={"ID":"f87a7847-84fe-4313-8f58-d6248b90cd61","Type":"ContainerStarted","Data":"14209525c862c40f3e9856af0003cbfbb3b6d0038135ea97d97574c31608882f"} Oct 30 03:57:18.448553 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:18.448483 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" Oct 30 03:57:18.490327 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:18.490280 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" podStartSLOduration=1.490267081 podStartE2EDuration="1.490267081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:18.489321984 +0000 UTC m=+4599.502600088" watchObservedRunningTime="2026-10-30 03:57:18.490267081 +0000 UTC m=+4599.503545177" Oct 30 03:57:18.609326 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:18.609300 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:57:18.611292 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:57:18.611263 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2d41a8_f51c_490b_a1fe_58dde1095ba3.slice/crio-39bb02817f5cefa113d2bdd18b15c2db19c00199f6e991b00f5a804258620ddd WatchSource:0}: Error finding container 39bb02817f5cefa113d2bdd18b15c2db19c00199f6e991b00f5a804258620ddd: Status 404 returned error can't find the container with id 39bb02817f5cefa113d2bdd18b15c2db19c00199f6e991b00f5a804258620ddd Oct 30 03:57:18.613486 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:18.613452 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:57:18.613587 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:18.613517 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:57:19.454537 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:19.454491 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"2e2d41a8-f51c-490b-a1fe-58dde1095ba3","Type":"ContainerStarted","Data":"b5c88ea1b476faf0c0cf59600c0cceef647bb4bd3214933a82e2655ec5e75658"} Oct 30 03:57:19.455129 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:19.454561 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"2e2d41a8-f51c-490b-a1fe-58dde1095ba3","Type":"ContainerStarted","Data":"39bb02817f5cefa113d2bdd18b15c2db19c00199f6e991b00f5a804258620ddd"} Oct 30 03:57:19.497562 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:19.497506 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.49749096 podStartE2EDuration="2.49749096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:19.497333986 +0000 UTC m=+4600.510612089" watchObservedRunningTime="2026-10-30 03:57:19.49749096 +0000 UTC m=+4600.510769065" Oct 30 03:57:29.456496 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:29.456467 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" Oct 30 03:57:33.023389 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.023349 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-x4zdz"] Oct 30 03:57:33.023830 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.023543 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" podUID="f87a7847-84fe-4313-8f58-d6248b90cd61" containerName="korrel8r" containerID="cri-o://33df3e894a67d6a113a307683e471c3f88d35fe2718ad50aaec1df025bec5215" gracePeriod=30 Oct 30 03:57:33.274537 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.274467 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" Oct 30 03:57:33.303049 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.303011 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f87a7847-84fe-4313-8f58-d6248b90cd61-config\" (UniqueName: \"kubernetes.io/configmap/f87a7847-84fe-4313-8f58-d6248b90cd61-config\") pod \"f87a7847-84fe-4313-8f58-d6248b90cd61\" (UID: \"f87a7847-84fe-4313-8f58-d6248b90cd61\") " Oct 30 03:57:33.303237 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.303081 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f87a7847-84fe-4313-8f58-d6248b90cd61-tls\" (UniqueName: \"kubernetes.io/secret/f87a7847-84fe-4313-8f58-d6248b90cd61-tls\") pod \"f87a7847-84fe-4313-8f58-d6248b90cd61\" (UID: \"f87a7847-84fe-4313-8f58-d6248b90cd61\") " Oct 30 03:57:33.303237 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.303132 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f87a7847-84fe-4313-8f58-d6248b90cd61-kube-api-access-8cphd\" (UniqueName: \"kubernetes.io/projected/f87a7847-84fe-4313-8f58-d6248b90cd61-kube-api-access-8cphd\") pod \"f87a7847-84fe-4313-8f58-d6248b90cd61\" (UID: \"f87a7847-84fe-4313-8f58-d6248b90cd61\") " Oct 30 03:57:33.303532 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.303499 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87a7847-84fe-4313-8f58-d6248b90cd61-config" pod "f87a7847-84fe-4313-8f58-d6248b90cd61" (UID: "f87a7847-84fe-4313-8f58-d6248b90cd61"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:57:33.305278 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.305249 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f87a7847-84fe-4313-8f58-d6248b90cd61-kube-api-access-8cphd" pod "f87a7847-84fe-4313-8f58-d6248b90cd61" (UID: "f87a7847-84fe-4313-8f58-d6248b90cd61"). InnerVolumeSpecName "kube-api-access-8cphd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:57:33.305373 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.305324 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f87a7847-84fe-4313-8f58-d6248b90cd61-tls" pod "f87a7847-84fe-4313-8f58-d6248b90cd61" (UID: "f87a7847-84fe-4313-8f58-d6248b90cd61"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:57:33.403873 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.403841 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f87a7847-84fe-4313-8f58-d6248b90cd61-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:57:33.403873 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.403867 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f87a7847-84fe-4313-8f58-d6248b90cd61-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:57:33.403873 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.403876 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cphd\" (UniqueName: \"kubernetes.io/projected/f87a7847-84fe-4313-8f58-d6248b90cd61-kube-api-access-8cphd\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:57:33.457739 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.457697 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:57:33.457923 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.457907 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="2e2d41a8-f51c-490b-a1fe-58dde1095ba3" containerName="otc-container" containerID="cri-o://b5c88ea1b476faf0c0cf59600c0cceef647bb4bd3214933a82e2655ec5e75658" gracePeriod=30 Oct 30 03:57:33.523687 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.523654 4925 generic.go:356] "Generic (PLEG): container finished" podID="f87a7847-84fe-4313-8f58-d6248b90cd61" containerID="33df3e894a67d6a113a307683e471c3f88d35fe2718ad50aaec1df025bec5215" exitCode=0 Oct 30 03:57:33.523907 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.523698 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" Oct 30 03:57:33.523907 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.523736 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" event={"ID":"f87a7847-84fe-4313-8f58-d6248b90cd61","Type":"ContainerDied","Data":"33df3e894a67d6a113a307683e471c3f88d35fe2718ad50aaec1df025bec5215"} Oct 30 03:57:33.523907 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.523823 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-x4zdz" event={"ID":"f87a7847-84fe-4313-8f58-d6248b90cd61","Type":"ContainerDied","Data":"14209525c862c40f3e9856af0003cbfbb3b6d0038135ea97d97574c31608882f"} Oct 30 03:57:33.523907 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.523854 4925 scope.go:122] "RemoveContainer" containerID="33df3e894a67d6a113a307683e471c3f88d35fe2718ad50aaec1df025bec5215" Oct 30 03:57:33.569568 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.569542 4925 scope.go:122] "RemoveContainer" containerID="33df3e894a67d6a113a307683e471c3f88d35fe2718ad50aaec1df025bec5215" Oct 30 03:57:33.569962 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:57:33.569936 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33df3e894a67d6a113a307683e471c3f88d35fe2718ad50aaec1df025bec5215\": container with ID starting with 33df3e894a67d6a113a307683e471c3f88d35fe2718ad50aaec1df025bec5215 not found: ID does not exist" containerID="33df3e894a67d6a113a307683e471c3f88d35fe2718ad50aaec1df025bec5215" Oct 30 03:57:33.570074 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.569968 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33df3e894a67d6a113a307683e471c3f88d35fe2718ad50aaec1df025bec5215"} err="failed to get container status \"33df3e894a67d6a113a307683e471c3f88d35fe2718ad50aaec1df025bec5215\": rpc error: code = NotFound desc = could not find container \"33df3e894a67d6a113a307683e471c3f88d35fe2718ad50aaec1df025bec5215\": container with ID starting with 33df3e894a67d6a113a307683e471c3f88d35fe2718ad50aaec1df025bec5215 not found: ID does not exist" Oct 30 03:57:33.575042 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.575013 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-x4zdz"] Oct 30 03:57:33.578993 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.578972 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-x4zdz"] Oct 30 03:57:33.711379 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.711352 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:33.806524 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.806432 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-kube-api-access-hv9lx\" (UniqueName: \"kubernetes.io/projected/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-kube-api-access-hv9lx\") pod \"2e2d41a8-f51c-490b-a1fe-58dde1095ba3\" (UID: \"2e2d41a8-f51c-490b-a1fe-58dde1095ba3\") " Oct 30 03:57:33.806524 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.806498 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-otc-internal\") pod \"2e2d41a8-f51c-490b-a1fe-58dde1095ba3\" (UID: \"2e2d41a8-f51c-490b-a1fe-58dde1095ba3\") " Oct 30 03:57:33.806742 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.806535 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-monitoring-tls-certs\") pod \"2e2d41a8-f51c-490b-a1fe-58dde1095ba3\" (UID: \"2e2d41a8-f51c-490b-a1fe-58dde1095ba3\") " Oct 30 03:57:33.806994 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.806888 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-otc-internal" pod "2e2d41a8-f51c-490b-a1fe-58dde1095ba3" (UID: "2e2d41a8-f51c-490b-a1fe-58dde1095ba3"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:57:33.808672 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.808641 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-monitoring-tls-certs" pod "2e2d41a8-f51c-490b-a1fe-58dde1095ba3" (UID: "2e2d41a8-f51c-490b-a1fe-58dde1095ba3"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:57:33.808799 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.808707 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-kube-api-access-hv9lx" pod "2e2d41a8-f51c-490b-a1fe-58dde1095ba3" (UID: "2e2d41a8-f51c-490b-a1fe-58dde1095ba3"). InnerVolumeSpecName "kube-api-access-hv9lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:57:33.907434 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.907400 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:57:33.907434 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.907430 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:57:33.907647 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:33.907445 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv9lx\" (UniqueName: \"kubernetes.io/projected/2e2d41a8-f51c-490b-a1fe-58dde1095ba3-kube-api-access-hv9lx\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:57:34.530782 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:34.530736 4925 generic.go:356] "Generic (PLEG): container finished" podID="2e2d41a8-f51c-490b-a1fe-58dde1095ba3" containerID="b5c88ea1b476faf0c0cf59600c0cceef647bb4bd3214933a82e2655ec5e75658" exitCode=0 Oct 30 03:57:34.531272 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:34.530795 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"2e2d41a8-f51c-490b-a1fe-58dde1095ba3","Type":"ContainerDied","Data":"b5c88ea1b476faf0c0cf59600c0cceef647bb4bd3214933a82e2655ec5e75658"} Oct 30 03:57:34.531272 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:34.530816 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:34.531272 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:34.530865 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"2e2d41a8-f51c-490b-a1fe-58dde1095ba3","Type":"ContainerDied","Data":"39bb02817f5cefa113d2bdd18b15c2db19c00199f6e991b00f5a804258620ddd"} Oct 30 03:57:34.531272 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:34.530899 4925 scope.go:122] "RemoveContainer" containerID="b5c88ea1b476faf0c0cf59600c0cceef647bb4bd3214933a82e2655ec5e75658" Oct 30 03:57:34.539553 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:34.539534 4925 scope.go:122] "RemoveContainer" containerID="b5c88ea1b476faf0c0cf59600c0cceef647bb4bd3214933a82e2655ec5e75658" Oct 30 03:57:34.539826 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:57:34.539807 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5c88ea1b476faf0c0cf59600c0cceef647bb4bd3214933a82e2655ec5e75658\": container with ID starting with b5c88ea1b476faf0c0cf59600c0cceef647bb4bd3214933a82e2655ec5e75658 not found: ID does not exist" containerID="b5c88ea1b476faf0c0cf59600c0cceef647bb4bd3214933a82e2655ec5e75658" Oct 30 03:57:34.539906 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:34.539838 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c88ea1b476faf0c0cf59600c0cceef647bb4bd3214933a82e2655ec5e75658"} err="failed to get container status \"b5c88ea1b476faf0c0cf59600c0cceef647bb4bd3214933a82e2655ec5e75658\": rpc error: code = NotFound desc = could not find container \"b5c88ea1b476faf0c0cf59600c0cceef647bb4bd3214933a82e2655ec5e75658\": container with ID starting with b5c88ea1b476faf0c0cf59600c0cceef647bb4bd3214933a82e2655ec5e75658 not found: ID does not exist" Oct 30 03:57:34.586912 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:34.586882 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:57:34.599573 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:34.599549 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:57:35.119440 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:35.119407 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e2d41a8-f51c-490b-a1fe-58dde1095ba3" path="/var/lib/kubelet/pods/2e2d41a8-f51c-490b-a1fe-58dde1095ba3/volumes" Oct 30 03:57:35.119976 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:35.119951 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f87a7847-84fe-4313-8f58-d6248b90cd61" path="/var/lib/kubelet/pods/f87a7847-84fe-4313-8f58-d6248b90cd61/volumes" Oct 30 03:57:36.206319 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.206285 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-r8rxc"] Oct 30 03:57:36.206705 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.206627 4925 cpu_manager.go:414] "containerMap: removing container" podUID="2e2d41a8-f51c-490b-a1fe-58dde1095ba3" containerName="otc-container" Oct 30 03:57:36.206705 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.206638 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2d41a8-f51c-490b-a1fe-58dde1095ba3" containerName="otc-container" Oct 30 03:57:36.206705 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.206649 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f87a7847-84fe-4313-8f58-d6248b90cd61" containerName="korrel8r" Oct 30 03:57:36.206705 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.206655 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f87a7847-84fe-4313-8f58-d6248b90cd61" containerName="korrel8r" Oct 30 03:57:36.206850 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.206713 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2d41a8-f51c-490b-a1fe-58dde1095ba3" containerName="otc-container" Oct 30 03:57:36.206850 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.206722 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f87a7847-84fe-4313-8f58-d6248b90cd61" containerName="korrel8r" Oct 30 03:57:36.210883 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.210865 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" Oct 30 03:57:36.226968 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.226907 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/99a96eae-d48d-434b-b114-e3fa51d65966-config\") pod \"korrel8r-6644769df5-r8rxc\" (UID: \"99a96eae-d48d-434b-b114-e3fa51d65966\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" Oct 30 03:57:36.227096 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.227033 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/99a96eae-d48d-434b-b114-e3fa51d65966-tls\") pod \"korrel8r-6644769df5-r8rxc\" (UID: \"99a96eae-d48d-434b-b114-e3fa51d65966\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" Oct 30 03:57:36.227173 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.227144 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j929h\" (UniqueName: \"kubernetes.io/projected/99a96eae-d48d-434b-b114-e3fa51d65966-kube-api-access-j929h\") pod \"korrel8r-6644769df5-r8rxc\" (UID: \"99a96eae-d48d-434b-b114-e3fa51d65966\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" Oct 30 03:57:36.227805 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.227742 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 03:57:36.229898 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.229878 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-r8rxc"] Oct 30 03:57:36.240449 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.240420 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-2fmwt\"" Oct 30 03:57:36.240574 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.240453 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 03:57:36.240574 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.240469 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 03:57:36.240574 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.240540 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 03:57:36.328288 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.328241 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/99a96eae-d48d-434b-b114-e3fa51d65966-config\") pod \"korrel8r-6644769df5-r8rxc\" (UID: \"99a96eae-d48d-434b-b114-e3fa51d65966\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" Oct 30 03:57:36.328288 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.328293 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/99a96eae-d48d-434b-b114-e3fa51d65966-tls\") pod \"korrel8r-6644769df5-r8rxc\" (UID: \"99a96eae-d48d-434b-b114-e3fa51d65966\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" Oct 30 03:57:36.328539 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.328349 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j929h\" (UniqueName: \"kubernetes.io/projected/99a96eae-d48d-434b-b114-e3fa51d65966-kube-api-access-j929h\") pod \"korrel8r-6644769df5-r8rxc\" (UID: \"99a96eae-d48d-434b-b114-e3fa51d65966\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" Oct 30 03:57:36.328539 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:57:36.328467 4925 secret.go:189] Couldn't get secret redhat-ods-monitoring/korrel8r-tls: secret "korrel8r-tls" not found Oct 30 03:57:36.328646 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:57:36.328544 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a96eae-d48d-434b-b114-e3fa51d65966-tls podName:99a96eae-d48d-434b-b114-e3fa51d65966 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:36.828525954 +0000 UTC m=+4617.841804035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/99a96eae-d48d-434b-b114-e3fa51d65966-tls") pod "korrel8r-6644769df5-r8rxc" (UID: "99a96eae-d48d-434b-b114-e3fa51d65966") : secret "korrel8r-tls" not found Oct 30 03:57:36.329060 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.329042 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/99a96eae-d48d-434b-b114-e3fa51d65966-config\") pod \"korrel8r-6644769df5-r8rxc\" (UID: \"99a96eae-d48d-434b-b114-e3fa51d65966\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" Oct 30 03:57:36.352625 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.352601 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j929h\" (UniqueName: \"kubernetes.io/projected/99a96eae-d48d-434b-b114-e3fa51d65966-kube-api-access-j929h\") pod \"korrel8r-6644769df5-r8rxc\" (UID: \"99a96eae-d48d-434b-b114-e3fa51d65966\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" Oct 30 03:57:36.832254 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.832209 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/99a96eae-d48d-434b-b114-e3fa51d65966-tls\") pod \"korrel8r-6644769df5-r8rxc\" (UID: \"99a96eae-d48d-434b-b114-e3fa51d65966\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" Oct 30 03:57:36.837520 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.837482 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/99a96eae-d48d-434b-b114-e3fa51d65966-tls\") pod \"korrel8r-6644769df5-r8rxc\" (UID: \"99a96eae-d48d-434b-b114-e3fa51d65966\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" Oct 30 03:57:36.867160 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.867124 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:57:36.871335 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.871310 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:36.890486 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.890452 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:57:36.895073 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.895039 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-frlgf\"" Oct 30 03:57:36.901362 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.901336 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:57:36.901362 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.901351 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-645168e1\"" Oct 30 03:57:36.932840 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.932806 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6411d342-767f-4fea-855b-d8eba7e591f6-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6411d342-767f-4fea-855b-d8eba7e591f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:36.933020 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.932998 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zxpk\" (UniqueName: \"kubernetes.io/projected/6411d342-767f-4fea-855b-d8eba7e591f6-kube-api-access-5zxpk\") pod \"data-science-collector-collector-0\" (UID: \"6411d342-767f-4fea-855b-d8eba7e591f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:36.933110 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:36.933090 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6411d342-767f-4fea-855b-d8eba7e591f6-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6411d342-767f-4fea-855b-d8eba7e591f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:37.034171 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:37.034123 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6411d342-767f-4fea-855b-d8eba7e591f6-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6411d342-767f-4fea-855b-d8eba7e591f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:37.034328 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:37.034241 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zxpk\" (UniqueName: \"kubernetes.io/projected/6411d342-767f-4fea-855b-d8eba7e591f6-kube-api-access-5zxpk\") pod \"data-science-collector-collector-0\" (UID: \"6411d342-767f-4fea-855b-d8eba7e591f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:37.034328 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:37.034283 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6411d342-767f-4fea-855b-d8eba7e591f6-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6411d342-767f-4fea-855b-d8eba7e591f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:37.035068 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:37.035041 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6411d342-767f-4fea-855b-d8eba7e591f6-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6411d342-767f-4fea-855b-d8eba7e591f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:37.036534 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:37.036507 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6411d342-767f-4fea-855b-d8eba7e591f6-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6411d342-767f-4fea-855b-d8eba7e591f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:37.058184 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:37.058156 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zxpk\" (UniqueName: \"kubernetes.io/projected/6411d342-767f-4fea-855b-d8eba7e591f6-kube-api-access-5zxpk\") pod \"data-science-collector-collector-0\" (UID: \"6411d342-767f-4fea-855b-d8eba7e591f6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:37.124981 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:37.124905 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" Oct 30 03:57:37.185132 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:37.185090 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:57:37.426411 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:37.426385 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-r8rxc"] Oct 30 03:57:37.428552 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:57:37.428525 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a96eae_d48d_434b_b114_e3fa51d65966.slice/crio-b92a799600acd525ca345dc7cb160de184aab1218d99c7596354ba17fd0e3202 WatchSource:0}: Error finding container b92a799600acd525ca345dc7cb160de184aab1218d99c7596354ba17fd0e3202: Status 404 returned error can't find the container with id b92a799600acd525ca345dc7cb160de184aab1218d99c7596354ba17fd0e3202 Oct 30 03:57:37.491510 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:37.491443 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:57:37.494191 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:57:37.494163 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6411d342_767f_4fea_855b_d8eba7e591f6.slice/crio-22346050db73ae4e84291fc9d03105491dc1cd69d5ef3ab5d1eefd44d9c56275 WatchSource:0}: Error finding container 22346050db73ae4e84291fc9d03105491dc1cd69d5ef3ab5d1eefd44d9c56275: Status 404 returned error can't find the container with id 22346050db73ae4e84291fc9d03105491dc1cd69d5ef3ab5d1eefd44d9c56275 Oct 30 03:57:37.496353 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:37.496320 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:57:37.496425 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:37.496392 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:57:37.551782 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:37.551086 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" event={"ID":"99a96eae-d48d-434b-b114-e3fa51d65966","Type":"ContainerStarted","Data":"3994143b9d77da06296761f70ce2387ed7f599e2d6703b0fb731305f1408fa22"} Oct 30 03:57:37.551782 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:37.551145 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" event={"ID":"99a96eae-d48d-434b-b114-e3fa51d65966","Type":"ContainerStarted","Data":"b92a799600acd525ca345dc7cb160de184aab1218d99c7596354ba17fd0e3202"} Oct 30 03:57:37.551782 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:37.551171 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" Oct 30 03:57:37.552903 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:37.552882 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6411d342-767f-4fea-855b-d8eba7e591f6","Type":"ContainerStarted","Data":"22346050db73ae4e84291fc9d03105491dc1cd69d5ef3ab5d1eefd44d9c56275"} Oct 30 03:57:37.592928 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:37.592869 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" podStartSLOduration=1.5928556230000002 podStartE2EDuration="1.592855623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:37.592520217 +0000 UTC m=+4618.605798302" watchObservedRunningTime="2026-10-30 03:57:37.592855623 +0000 UTC m=+4618.606133725" Oct 30 03:57:38.560010 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:38.559963 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6411d342-767f-4fea-855b-d8eba7e591f6","Type":"ContainerStarted","Data":"adf87bbdc454a6532c95c870d18bf710ef8286a5dab3ff6e2d3f79e0e3328ff5"} Oct 30 03:57:38.603380 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:38.603324 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.603310351 podStartE2EDuration="2.603310351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:38.601809684 +0000 UTC m=+4619.615087788" watchObservedRunningTime="2026-10-30 03:57:38.603310351 +0000 UTC m=+4619.616588455" Oct 30 03:57:48.161230 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:57:48.161193 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:57:48.562117 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:57:48.562036 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" Oct 30 03:58:18.161602 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:58:18.161563 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:58:23.989174 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:23.989136 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-r8rxc"] Oct 30 03:58:23.989669 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:23.989338 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" podUID="99a96eae-d48d-434b-b114-e3fa51d65966" containerName="korrel8r" containerID="cri-o://3994143b9d77da06296761f70ce2387ed7f599e2d6703b0fb731305f1408fa22" gracePeriod=30 Oct 30 03:58:24.236357 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.236330 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" Oct 30 03:58:24.397796 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.397763 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:58:24.398030 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.397988 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="6411d342-767f-4fea-855b-d8eba7e591f6" containerName="otc-container" containerID="cri-o://adf87bbdc454a6532c95c870d18bf710ef8286a5dab3ff6e2d3f79e0e3328ff5" gracePeriod=30 Oct 30 03:58:24.414666 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.414636 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99a96eae-d48d-434b-b114-e3fa51d65966-tls\" (UniqueName: \"kubernetes.io/secret/99a96eae-d48d-434b-b114-e3fa51d65966-tls\") pod \"99a96eae-d48d-434b-b114-e3fa51d65966\" (UID: \"99a96eae-d48d-434b-b114-e3fa51d65966\") " Oct 30 03:58:24.414880 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.414704 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a96eae-d48d-434b-b114-e3fa51d65966-kube-api-access-j929h\" (UniqueName: \"kubernetes.io/projected/99a96eae-d48d-434b-b114-e3fa51d65966-kube-api-access-j929h\") pod \"99a96eae-d48d-434b-b114-e3fa51d65966\" (UID: \"99a96eae-d48d-434b-b114-e3fa51d65966\") " Oct 30 03:58:24.414880 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.414799 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99a96eae-d48d-434b-b114-e3fa51d65966-config\" (UniqueName: \"kubernetes.io/configmap/99a96eae-d48d-434b-b114-e3fa51d65966-config\") pod \"99a96eae-d48d-434b-b114-e3fa51d65966\" (UID: \"99a96eae-d48d-434b-b114-e3fa51d65966\") " Oct 30 03:58:24.415991 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.415435 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a96eae-d48d-434b-b114-e3fa51d65966-config" pod "99a96eae-d48d-434b-b114-e3fa51d65966" (UID: "99a96eae-d48d-434b-b114-e3fa51d65966"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:58:24.418728 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.418692 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99a96eae-d48d-434b-b114-e3fa51d65966-tls" pod "99a96eae-d48d-434b-b114-e3fa51d65966" (UID: "99a96eae-d48d-434b-b114-e3fa51d65966"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:58:24.420275 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.420246 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a96eae-d48d-434b-b114-e3fa51d65966-kube-api-access-j929h" pod "99a96eae-d48d-434b-b114-e3fa51d65966" (UID: "99a96eae-d48d-434b-b114-e3fa51d65966"). InnerVolumeSpecName "kube-api-access-j929h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:58:24.515641 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.515623 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/99a96eae-d48d-434b-b114-e3fa51d65966-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:58:24.515700 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.515646 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j929h\" (UniqueName: \"kubernetes.io/projected/99a96eae-d48d-434b-b114-e3fa51d65966-kube-api-access-j929h\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:58:24.515700 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.515657 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/99a96eae-d48d-434b-b114-e3fa51d65966-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:58:24.638343 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.638317 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:58:24.773633 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.773558 4925 generic.go:356] "Generic (PLEG): container finished" podID="99a96eae-d48d-434b-b114-e3fa51d65966" containerID="3994143b9d77da06296761f70ce2387ed7f599e2d6703b0fb731305f1408fa22" exitCode=0 Oct 30 03:58:24.773633 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.773600 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" Oct 30 03:58:24.773870 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.773632 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" event={"ID":"99a96eae-d48d-434b-b114-e3fa51d65966","Type":"ContainerDied","Data":"3994143b9d77da06296761f70ce2387ed7f599e2d6703b0fb731305f1408fa22"} Oct 30 03:58:24.773870 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.773684 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-r8rxc" event={"ID":"99a96eae-d48d-434b-b114-e3fa51d65966","Type":"ContainerDied","Data":"b92a799600acd525ca345dc7cb160de184aab1218d99c7596354ba17fd0e3202"} Oct 30 03:58:24.773870 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.773712 4925 scope.go:122] "RemoveContainer" containerID="3994143b9d77da06296761f70ce2387ed7f599e2d6703b0fb731305f1408fa22" Oct 30 03:58:24.775070 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.775050 4925 generic.go:356] "Generic (PLEG): container finished" podID="6411d342-767f-4fea-855b-d8eba7e591f6" containerID="adf87bbdc454a6532c95c870d18bf710ef8286a5dab3ff6e2d3f79e0e3328ff5" exitCode=0 Oct 30 03:58:24.775178 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.775079 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6411d342-767f-4fea-855b-d8eba7e591f6","Type":"ContainerDied","Data":"adf87bbdc454a6532c95c870d18bf710ef8286a5dab3ff6e2d3f79e0e3328ff5"} Oct 30 03:58:24.775178 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.775134 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6411d342-767f-4fea-855b-d8eba7e591f6","Type":"ContainerDied","Data":"22346050db73ae4e84291fc9d03105491dc1cd69d5ef3ab5d1eefd44d9c56275"} Oct 30 03:58:24.775178 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.775093 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:58:24.782319 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.782296 4925 scope.go:122] "RemoveContainer" containerID="3994143b9d77da06296761f70ce2387ed7f599e2d6703b0fb731305f1408fa22" Oct 30 03:58:24.782559 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:58:24.782541 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3994143b9d77da06296761f70ce2387ed7f599e2d6703b0fb731305f1408fa22\": container with ID starting with 3994143b9d77da06296761f70ce2387ed7f599e2d6703b0fb731305f1408fa22 not found: ID does not exist" containerID="3994143b9d77da06296761f70ce2387ed7f599e2d6703b0fb731305f1408fa22" Oct 30 03:58:24.782642 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.782571 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3994143b9d77da06296761f70ce2387ed7f599e2d6703b0fb731305f1408fa22"} err="failed to get container status \"3994143b9d77da06296761f70ce2387ed7f599e2d6703b0fb731305f1408fa22\": rpc error: code = NotFound desc = could not find container \"3994143b9d77da06296761f70ce2387ed7f599e2d6703b0fb731305f1408fa22\": container with ID starting with 3994143b9d77da06296761f70ce2387ed7f599e2d6703b0fb731305f1408fa22 not found: ID does not exist" Oct 30 03:58:24.782642 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.782593 4925 scope.go:122] "RemoveContainer" containerID="adf87bbdc454a6532c95c870d18bf710ef8286a5dab3ff6e2d3f79e0e3328ff5" Oct 30 03:58:24.790092 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.790076 4925 scope.go:122] "RemoveContainer" containerID="adf87bbdc454a6532c95c870d18bf710ef8286a5dab3ff6e2d3f79e0e3328ff5" Oct 30 03:58:24.790350 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:58:24.790332 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adf87bbdc454a6532c95c870d18bf710ef8286a5dab3ff6e2d3f79e0e3328ff5\": container with ID starting with adf87bbdc454a6532c95c870d18bf710ef8286a5dab3ff6e2d3f79e0e3328ff5 not found: ID does not exist" containerID="adf87bbdc454a6532c95c870d18bf710ef8286a5dab3ff6e2d3f79e0e3328ff5" Oct 30 03:58:24.790392 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.790357 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf87bbdc454a6532c95c870d18bf710ef8286a5dab3ff6e2d3f79e0e3328ff5"} err="failed to get container status \"adf87bbdc454a6532c95c870d18bf710ef8286a5dab3ff6e2d3f79e0e3328ff5\": rpc error: code = NotFound desc = could not find container \"adf87bbdc454a6532c95c870d18bf710ef8286a5dab3ff6e2d3f79e0e3328ff5\": container with ID starting with adf87bbdc454a6532c95c870d18bf710ef8286a5dab3ff6e2d3f79e0e3328ff5 not found: ID does not exist" Oct 30 03:58:24.818710 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.818692 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6411d342-767f-4fea-855b-d8eba7e591f6-otc-internal\" (UniqueName: \"kubernetes.io/configmap/6411d342-767f-4fea-855b-d8eba7e591f6-otc-internal\") pod \"6411d342-767f-4fea-855b-d8eba7e591f6\" (UID: \"6411d342-767f-4fea-855b-d8eba7e591f6\") " Oct 30 03:58:24.818843 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.818731 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6411d342-767f-4fea-855b-d8eba7e591f6-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6411d342-767f-4fea-855b-d8eba7e591f6-monitoring-tls-certs\") pod \"6411d342-767f-4fea-855b-d8eba7e591f6\" (UID: \"6411d342-767f-4fea-855b-d8eba7e591f6\") " Oct 30 03:58:24.818843 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.818785 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6411d342-767f-4fea-855b-d8eba7e591f6-kube-api-access-5zxpk\" (UniqueName: \"kubernetes.io/projected/6411d342-767f-4fea-855b-d8eba7e591f6-kube-api-access-5zxpk\") pod \"6411d342-767f-4fea-855b-d8eba7e591f6\" (UID: \"6411d342-767f-4fea-855b-d8eba7e591f6\") " Oct 30 03:58:24.819077 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.819055 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6411d342-767f-4fea-855b-d8eba7e591f6-otc-internal" pod "6411d342-767f-4fea-855b-d8eba7e591f6" (UID: "6411d342-767f-4fea-855b-d8eba7e591f6"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:58:24.820660 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.820636 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6411d342-767f-4fea-855b-d8eba7e591f6-kube-api-access-5zxpk" pod "6411d342-767f-4fea-855b-d8eba7e591f6" (UID: "6411d342-767f-4fea-855b-d8eba7e591f6"). InnerVolumeSpecName "kube-api-access-5zxpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:58:24.820737 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.820668 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6411d342-767f-4fea-855b-d8eba7e591f6-monitoring-tls-certs" pod "6411d342-767f-4fea-855b-d8eba7e591f6" (UID: "6411d342-767f-4fea-855b-d8eba7e591f6"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:58:24.826025 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.826006 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-r8rxc"] Oct 30 03:58:24.830454 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.830434 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-r8rxc"] Oct 30 03:58:24.920259 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.920239 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6411d342-767f-4fea-855b-d8eba7e591f6-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:58:24.920259 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.920257 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6411d342-767f-4fea-855b-d8eba7e591f6-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:58:24.920375 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:24.920266 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zxpk\" (UniqueName: \"kubernetes.io/projected/6411d342-767f-4fea-855b-d8eba7e591f6-kube-api-access-5zxpk\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 03:58:25.119043 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:25.119011 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a96eae-d48d-434b-b114-e3fa51d65966" path="/var/lib/kubelet/pods/99a96eae-d48d-434b-b114-e3fa51d65966/volumes" Oct 30 03:58:25.124797 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:25.124776 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:58:25.129055 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:25.129032 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:58:27.120088 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:27.120044 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6411d342-767f-4fea-855b-d8eba7e591f6" path="/var/lib/kubelet/pods/6411d342-767f-4fea-855b-d8eba7e591f6/volumes" Oct 30 03:58:48.160502 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:58:48.160459 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:58:55.679016 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.678981 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk"] Oct 30 03:58:55.679496 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.679330 4925 cpu_manager.go:414] "containerMap: removing container" podUID="99a96eae-d48d-434b-b114-e3fa51d65966" containerName="korrel8r" Oct 30 03:58:55.679496 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.679342 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a96eae-d48d-434b-b114-e3fa51d65966" containerName="korrel8r" Oct 30 03:58:55.679496 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.679365 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6411d342-767f-4fea-855b-d8eba7e591f6" containerName="otc-container" Oct 30 03:58:55.679496 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.679370 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6411d342-767f-4fea-855b-d8eba7e591f6" containerName="otc-container" Oct 30 03:58:55.679496 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.679424 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a96eae-d48d-434b-b114-e3fa51d65966" containerName="korrel8r" Oct 30 03:58:55.679496 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.679436 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6411d342-767f-4fea-855b-d8eba7e591f6" containerName="otc-container" Oct 30 03:58:55.682157 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.682139 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" Oct 30 03:58:55.712297 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.712274 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-57qx5\"" Oct 30 03:58:55.712447 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.712300 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 03:58:55.712447 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.712310 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 03:58:55.712447 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.712362 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 03:58:55.714393 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.714371 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk"] Oct 30 03:58:55.714579 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.714566 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 03:58:55.746341 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.746318 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5762b60-b2a0-47e8-8b52-93336610564e-config\") pod \"korrel8r-5d45699dc4-7khpk\" (UID: \"d5762b60-b2a0-47e8-8b52-93336610564e\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" Oct 30 03:58:55.746475 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.746357 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57sgl\" (UniqueName: \"kubernetes.io/projected/d5762b60-b2a0-47e8-8b52-93336610564e-kube-api-access-57sgl\") pod \"korrel8r-5d45699dc4-7khpk\" (UID: \"d5762b60-b2a0-47e8-8b52-93336610564e\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" Oct 30 03:58:55.746475 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.746458 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d5762b60-b2a0-47e8-8b52-93336610564e-tls\") pod \"korrel8r-5d45699dc4-7khpk\" (UID: \"d5762b60-b2a0-47e8-8b52-93336610564e\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" Oct 30 03:58:55.847292 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.847262 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5762b60-b2a0-47e8-8b52-93336610564e-config\") pod \"korrel8r-5d45699dc4-7khpk\" (UID: \"d5762b60-b2a0-47e8-8b52-93336610564e\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" Oct 30 03:58:55.847409 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.847307 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57sgl\" (UniqueName: \"kubernetes.io/projected/d5762b60-b2a0-47e8-8b52-93336610564e-kube-api-access-57sgl\") pod \"korrel8r-5d45699dc4-7khpk\" (UID: \"d5762b60-b2a0-47e8-8b52-93336610564e\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" Oct 30 03:58:55.847409 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.847375 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d5762b60-b2a0-47e8-8b52-93336610564e-tls\") pod \"korrel8r-5d45699dc4-7khpk\" (UID: \"d5762b60-b2a0-47e8-8b52-93336610564e\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" Oct 30 03:58:55.848102 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.848078 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5762b60-b2a0-47e8-8b52-93336610564e-config\") pod \"korrel8r-5d45699dc4-7khpk\" (UID: \"d5762b60-b2a0-47e8-8b52-93336610564e\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" Oct 30 03:58:55.849708 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.849685 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d5762b60-b2a0-47e8-8b52-93336610564e-tls\") pod \"korrel8r-5d45699dc4-7khpk\" (UID: \"d5762b60-b2a0-47e8-8b52-93336610564e\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" Oct 30 03:58:55.871093 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.871062 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57sgl\" (UniqueName: \"kubernetes.io/projected/d5762b60-b2a0-47e8-8b52-93336610564e-kube-api-access-57sgl\") pod \"korrel8r-5d45699dc4-7khpk\" (UID: \"d5762b60-b2a0-47e8-8b52-93336610564e\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" Oct 30 03:58:55.994098 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:55.994034 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" Oct 30 03:58:56.314377 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.314349 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk"] Oct 30 03:58:56.353439 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.353409 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:58:56.358557 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.358537 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:58:56.374142 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.374109 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8l25s\"" Oct 30 03:58:56.376867 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.376821 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 03:58:56.386728 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.386707 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-90b17333\"" Oct 30 03:58:56.388558 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.388535 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:58:56.462283 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.462255 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e5091261-7125-499f-9e8b-e9a04e4c6ea0-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"e5091261-7125-499f-9e8b-e9a04e4c6ea0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:58:56.462283 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.462288 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5091261-7125-499f-9e8b-e9a04e4c6ea0-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e5091261-7125-499f-9e8b-e9a04e4c6ea0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:58:56.462466 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.462399 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcrkg\" (UniqueName: \"kubernetes.io/projected/e5091261-7125-499f-9e8b-e9a04e4c6ea0-kube-api-access-gcrkg\") pod \"data-science-collector-collector-0\" (UID: \"e5091261-7125-499f-9e8b-e9a04e4c6ea0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:58:56.563837 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.563745 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e5091261-7125-499f-9e8b-e9a04e4c6ea0-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"e5091261-7125-499f-9e8b-e9a04e4c6ea0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:58:56.563837 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.563792 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5091261-7125-499f-9e8b-e9a04e4c6ea0-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e5091261-7125-499f-9e8b-e9a04e4c6ea0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:58:56.564030 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.563851 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcrkg\" (UniqueName: \"kubernetes.io/projected/e5091261-7125-499f-9e8b-e9a04e4c6ea0-kube-api-access-gcrkg\") pod \"data-science-collector-collector-0\" (UID: \"e5091261-7125-499f-9e8b-e9a04e4c6ea0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:58:56.564530 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.564501 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e5091261-7125-499f-9e8b-e9a04e4c6ea0-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"e5091261-7125-499f-9e8b-e9a04e4c6ea0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:58:56.566162 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.566135 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5091261-7125-499f-9e8b-e9a04e4c6ea0-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"e5091261-7125-499f-9e8b-e9a04e4c6ea0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:58:56.587565 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.587537 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcrkg\" (UniqueName: \"kubernetes.io/projected/e5091261-7125-499f-9e8b-e9a04e4c6ea0-kube-api-access-gcrkg\") pod \"data-science-collector-collector-0\" (UID: \"e5091261-7125-499f-9e8b-e9a04e4c6ea0\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:58:56.688525 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.688496 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 03:58:56.923897 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.923865 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" event={"ID":"d5762b60-b2a0-47e8-8b52-93336610564e","Type":"ContainerStarted","Data":"df38ec7ff0643c45d1c4d5f6f6490ce964b91681086cf36183c8ffc31bc4d27e"} Oct 30 03:58:56.924052 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.923913 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" event={"ID":"d5762b60-b2a0-47e8-8b52-93336610564e","Type":"ContainerStarted","Data":"d825888edd15cfb5dbb3846ca3821fe72291a5505817e700d411f2ffe377b95c"} Oct 30 03:58:56.924052 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.923999 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" Oct 30 03:58:56.964586 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.964541 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" podStartSLOduration=1.964528724 podStartE2EDuration="1.964528724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:58:56.963985921 +0000 UTC m=+4697.977264023" watchObservedRunningTime="2026-10-30 03:58:56.964528724 +0000 UTC m=+4697.977806827" Oct 30 03:58:56.985921 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.985898 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 03:58:56.987952 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:58:56.987926 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5091261_7125_499f_9e8b_e9a04e4c6ea0.slice/crio-fe209dd7f5aeeb4fd456c65af06e7150b03f3d2c0b1c1c21d5e753568b25fec7 WatchSource:0}: Error finding container fe209dd7f5aeeb4fd456c65af06e7150b03f3d2c0b1c1c21d5e753568b25fec7: Status 404 returned error can't find the container with id fe209dd7f5aeeb4fd456c65af06e7150b03f3d2c0b1c1c21d5e753568b25fec7 Oct 30 03:58:56.990188 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.990154 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:58:56.990287 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:56.990223 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 03:58:57.930839 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:57.930800 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e5091261-7125-499f-9e8b-e9a04e4c6ea0","Type":"ContainerStarted","Data":"3d4c40ea352c8880bf62d66874aeca180d9ab90f0047f618901921316665ead1"} Oct 30 03:58:57.931243 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:57.930861 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e5091261-7125-499f-9e8b-e9a04e4c6ea0","Type":"ContainerStarted","Data":"fe209dd7f5aeeb4fd456c65af06e7150b03f3d2c0b1c1c21d5e753568b25fec7"} Oct 30 03:58:57.973932 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:58:57.973879 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.973865354 podStartE2EDuration="1.973865354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:58:57.972986208 +0000 UTC m=+4698.986264316" watchObservedRunningTime="2026-10-30 03:58:57.973865354 +0000 UTC m=+4698.987143457" Oct 30 03:59:00.630199 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.630164 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6"] Oct 30 03:59:00.633808 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.633792 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.649409 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.649386 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\"" Oct 30 03:59:00.649506 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.649470 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-cabundle\"" Oct 30 03:59:00.649745 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.649727 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 30 03:59:00.662669 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.662647 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 30 03:59:00.663571 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.662646 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-mtls\"" Oct 30 03:59:00.663699 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.662790 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-dockercfg-25rxl\"" Oct 30 03:59:00.663699 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.662915 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-tls\"" Oct 30 03:59:00.665991 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.665968 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6"] Oct 30 03:59:00.697963 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.697935 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-serving-certs\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.698075 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.697979 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtsnh\" (UniqueName: \"kubernetes.io/projected/f43c74ef-ec4b-433f-809f-8345121068d0-kube-api-access-rtsnh\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.698075 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.698000 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-rbac\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.698147 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.698090 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-cabundle\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.698147 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.698116 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.698147 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.698141 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.698242 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.698174 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-tenant\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.799525 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.799489 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-tenant\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.799686 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.799540 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-serving-certs\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.799686 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.799593 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtsnh\" (UniqueName: \"kubernetes.io/projected/f43c74ef-ec4b-433f-809f-8345121068d0-kube-api-access-rtsnh\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.799686 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.799621 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-rbac\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.799686 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.799657 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-cabundle\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.799943 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.799682 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.799943 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.799716 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.800953 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.800906 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.801386 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.801366 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-rbac\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.801481 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.801464 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-cabundle\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.802423 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.802403 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-tenant\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.802506 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.802431 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-serving-certs\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.802506 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.802498 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.823456 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.823436 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtsnh\" (UniqueName: \"kubernetes.io/projected/f43c74ef-ec4b-433f-809f-8345121068d0-kube-api-access-rtsnh\") pod \"tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:00.948155 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:00.948090 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:01.276471 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:01.276444 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6"] Oct 30 03:59:01.278073 ip-10-0-139-19 kubenswrapper[4925]: W1030 03:59:01.278042 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43c74ef_ec4b_433f_809f_8345121068d0.slice/crio-7f9d74b09d95f6f0021f0bc2259d1b8a619b7968eadd7091ad8f3c3ecfca0f2b WatchSource:0}: Error finding container 7f9d74b09d95f6f0021f0bc2259d1b8a619b7968eadd7091ad8f3c3ecfca0f2b: Status 404 returned error can't find the container with id 7f9d74b09d95f6f0021f0bc2259d1b8a619b7968eadd7091ad8f3c3ecfca0f2b Oct 30 03:59:01.280086 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:01.280068 4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:59:01.950812 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:01.950775 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" event={"ID":"f43c74ef-ec4b-433f-809f-8345121068d0","Type":"ContainerStarted","Data":"7f9d74b09d95f6f0021f0bc2259d1b8a619b7968eadd7091ad8f3c3ecfca0f2b"} Oct 30 03:59:03.964796 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:03.964737 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" event={"ID":"f43c74ef-ec4b-433f-809f-8345121068d0","Type":"ContainerStarted","Data":"d2d14dce9a4473ea7eca234afad56d677ebe58698809c5f9b495be810ca1d4dd"} Oct 30 03:59:05.976112 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:05.976075 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" event={"ID":"f43c74ef-ec4b-433f-809f-8345121068d0","Type":"ContainerStarted","Data":"4e63057d6eca5d7db98616feaa677c4c1808e9f956af9082ede301bb90503402"} Oct 30 03:59:05.976539 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:05.976138 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:05.976539 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:05.976237 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:05.977261 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:05.977235 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" podUID="f43c74ef-ec4b-433f-809f-8345121068d0" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.89:8083/ready\": dial tcp 10.132.0.89:8083: connect: connection refused" Oct 30 03:59:06.029105 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:06.029080 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:06.047691 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:06.047648 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" podStartSLOduration=2.325153172 podStartE2EDuration="6.047635561s" totalImagesPullingTime="3.722482389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:59:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:59:06.033603051 +0000 UTC m=+4707.046881154" watchObservedRunningTime="2026-10-30 03:59:06.047635561 +0000 UTC m=+4707.060913664" Oct 30 03:59:06.982718 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:06.982693 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 03:59:07.933499 ip-10-0-139-19 kubenswrapper[4925]: I1030 03:59:07.933472 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" Oct 30 03:59:18.161082 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:59:18.161036 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 03:59:48.161053 ip-10-0-139-19 kubenswrapper[4925]: E1030 03:59:48.161016 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:00:06.931666 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:06.931624 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:00:06.932841 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:06.931890 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="e5091261-7125-499f-9e8b-e9a04e4c6ea0" containerName="otc-container" containerID="cri-o://3d4c40ea352c8880bf62d66874aeca180d9ab90f0047f618901921316665ead1" gracePeriod=30 Oct 30 04:00:07.173678 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.173654 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:07.222308 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.222235 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5091261-7125-499f-9e8b-e9a04e4c6ea0-kube-api-access-gcrkg\" (UniqueName: \"kubernetes.io/projected/e5091261-7125-499f-9e8b-e9a04e4c6ea0-kube-api-access-gcrkg\") pod \"e5091261-7125-499f-9e8b-e9a04e4c6ea0\" (UID: \"e5091261-7125-499f-9e8b-e9a04e4c6ea0\") " Oct 30 04:00:07.222308 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.222290 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5091261-7125-499f-9e8b-e9a04e4c6ea0-otc-internal\" (UniqueName: \"kubernetes.io/configmap/e5091261-7125-499f-9e8b-e9a04e4c6ea0-otc-internal\") pod \"e5091261-7125-499f-9e8b-e9a04e4c6ea0\" (UID: \"e5091261-7125-499f-9e8b-e9a04e4c6ea0\") " Oct 30 04:00:07.222506 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.222315 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5091261-7125-499f-9e8b-e9a04e4c6ea0-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5091261-7125-499f-9e8b-e9a04e4c6ea0-monitoring-tls-certs\") pod \"e5091261-7125-499f-9e8b-e9a04e4c6ea0\" (UID: \"e5091261-7125-499f-9e8b-e9a04e4c6ea0\") " Oct 30 04:00:07.222689 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.222657 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5091261-7125-499f-9e8b-e9a04e4c6ea0-otc-internal" pod "e5091261-7125-499f-9e8b-e9a04e4c6ea0" (UID: "e5091261-7125-499f-9e8b-e9a04e4c6ea0"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:07.224483 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.224460 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5091261-7125-499f-9e8b-e9a04e4c6ea0-monitoring-tls-certs" pod "e5091261-7125-499f-9e8b-e9a04e4c6ea0" (UID: "e5091261-7125-499f-9e8b-e9a04e4c6ea0"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:00:07.224628 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.224598 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5091261-7125-499f-9e8b-e9a04e4c6ea0-kube-api-access-gcrkg" pod "e5091261-7125-499f-9e8b-e9a04e4c6ea0" (UID: "e5091261-7125-499f-9e8b-e9a04e4c6ea0"). InnerVolumeSpecName "kube-api-access-gcrkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:07.271392 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.271359 4925 generic.go:356] "Generic (PLEG): container finished" podID="e5091261-7125-499f-9e8b-e9a04e4c6ea0" containerID="3d4c40ea352c8880bf62d66874aeca180d9ab90f0047f618901921316665ead1" exitCode=0 Oct 30 04:00:07.271501 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.271402 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e5091261-7125-499f-9e8b-e9a04e4c6ea0","Type":"ContainerDied","Data":"3d4c40ea352c8880bf62d66874aeca180d9ab90f0047f618901921316665ead1"} Oct 30 04:00:07.271501 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.271450 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"e5091261-7125-499f-9e8b-e9a04e4c6ea0","Type":"ContainerDied","Data":"fe209dd7f5aeeb4fd456c65af06e7150b03f3d2c0b1c1c21d5e753568b25fec7"} Oct 30 04:00:07.271501 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.271473 4925 scope.go:122] "RemoveContainer" containerID="3d4c40ea352c8880bf62d66874aeca180d9ab90f0047f618901921316665ead1" Oct 30 04:00:07.271620 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.271406 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:07.281438 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.281417 4925 scope.go:122] "RemoveContainer" containerID="3d4c40ea352c8880bf62d66874aeca180d9ab90f0047f618901921316665ead1" Oct 30 04:00:07.281695 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:00:07.281676 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d4c40ea352c8880bf62d66874aeca180d9ab90f0047f618901921316665ead1\": container with ID starting with 3d4c40ea352c8880bf62d66874aeca180d9ab90f0047f618901921316665ead1 not found: ID does not exist" containerID="3d4c40ea352c8880bf62d66874aeca180d9ab90f0047f618901921316665ead1" Oct 30 04:00:07.281743 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.281704 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d4c40ea352c8880bf62d66874aeca180d9ab90f0047f618901921316665ead1"} err="failed to get container status \"3d4c40ea352c8880bf62d66874aeca180d9ab90f0047f618901921316665ead1\": rpc error: code = NotFound desc = could not find container \"3d4c40ea352c8880bf62d66874aeca180d9ab90f0047f618901921316665ead1\": container with ID starting with 3d4c40ea352c8880bf62d66874aeca180d9ab90f0047f618901921316665ead1 not found: ID does not exist" Oct 30 04:00:07.322786 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.322743 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e5091261-7125-499f-9e8b-e9a04e4c6ea0-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:00:07.322887 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.322788 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5091261-7125-499f-9e8b-e9a04e4c6ea0-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:00:07.322887 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.322798 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcrkg\" (UniqueName: \"kubernetes.io/projected/e5091261-7125-499f-9e8b-e9a04e4c6ea0-kube-api-access-gcrkg\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:00:07.324110 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.324092 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:00:07.328202 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.328182 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:00:07.364295 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.364273 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:00:07.364610 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.364599 4925 cpu_manager.go:414] "containerMap: removing container" podUID="e5091261-7125-499f-9e8b-e9a04e4c6ea0" containerName="otc-container" Oct 30 04:00:07.364653 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.364612 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5091261-7125-499f-9e8b-e9a04e4c6ea0" containerName="otc-container" Oct 30 04:00:07.364694 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.364685 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5091261-7125-499f-9e8b-e9a04e4c6ea0" containerName="otc-container" Oct 30 04:00:07.368913 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.368898 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:07.384339 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.384317 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:00:07.384521 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.384508 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-90b17333\"" Oct 30 04:00:07.384565 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.384533 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8l25s\"" Oct 30 04:00:07.389550 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.389528 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:00:07.423813 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.423784 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gshgz\" (UniqueName: \"kubernetes.io/projected/fe8cbfcc-d492-4704-afba-c9831dc6f365-kube-api-access-gshgz\") pod \"data-science-collector-collector-0\" (UID: \"fe8cbfcc-d492-4704-afba-c9831dc6f365\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:07.423920 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.423841 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fe8cbfcc-d492-4704-afba-c9831dc6f365-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"fe8cbfcc-d492-4704-afba-c9831dc6f365\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:07.423963 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.423912 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8cbfcc-d492-4704-afba-c9831dc6f365-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fe8cbfcc-d492-4704-afba-c9831dc6f365\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:07.525064 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.525009 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8cbfcc-d492-4704-afba-c9831dc6f365-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fe8cbfcc-d492-4704-afba-c9831dc6f365\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:07.525161 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.525063 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gshgz\" (UniqueName: \"kubernetes.io/projected/fe8cbfcc-d492-4704-afba-c9831dc6f365-kube-api-access-gshgz\") pod \"data-science-collector-collector-0\" (UID: \"fe8cbfcc-d492-4704-afba-c9831dc6f365\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:07.525161 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.525127 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fe8cbfcc-d492-4704-afba-c9831dc6f365-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"fe8cbfcc-d492-4704-afba-c9831dc6f365\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:07.525831 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.525804 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fe8cbfcc-d492-4704-afba-c9831dc6f365-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"fe8cbfcc-d492-4704-afba-c9831dc6f365\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:07.527393 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.527375 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8cbfcc-d492-4704-afba-c9831dc6f365-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fe8cbfcc-d492-4704-afba-c9831dc6f365\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:07.548897 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.548879 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gshgz\" (UniqueName: \"kubernetes.io/projected/fe8cbfcc-d492-4704-afba-c9831dc6f365-kube-api-access-gshgz\") pod \"data-science-collector-collector-0\" (UID: \"fe8cbfcc-d492-4704-afba-c9831dc6f365\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:07.681394 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.681367 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:07.978891 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.978868 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:00:07.981257 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:00:07.981223 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8cbfcc_d492_4704_afba_c9831dc6f365.slice/crio-bdb154058553acd642effcf30acb540d03d981c7f613183da588a352ece3cf36 WatchSource:0}: Error finding container bdb154058553acd642effcf30acb540d03d981c7f613183da588a352ece3cf36: Status 404 returned error can't find the container with id bdb154058553acd642effcf30acb540d03d981c7f613183da588a352ece3cf36 Oct 30 04:00:07.983863 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.983833 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:00:07.983971 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:07.983899 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:00:08.142947 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:08.142911 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk"] Oct 30 04:00:08.143570 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:08.143523 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" podUID="d5762b60-b2a0-47e8-8b52-93336610564e" containerName="korrel8r" containerID="cri-o://df38ec7ff0643c45d1c4d5f6f6490ce964b91681086cf36183c8ffc31bc4d27e" gracePeriod=30 Oct 30 04:00:08.279789 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:08.279742 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"fe8cbfcc-d492-4704-afba-c9831dc6f365","Type":"ContainerStarted","Data":"9768512a3ba8963a8399e756a8e16e55b4128df39f2603e196ca996f0ffd52b9"} Oct 30 04:00:08.279946 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:08.279838 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"fe8cbfcc-d492-4704-afba-c9831dc6f365","Type":"ContainerStarted","Data":"bdb154058553acd642effcf30acb540d03d981c7f613183da588a352ece3cf36"} Oct 30 04:00:08.281466 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:08.281436 4925 generic.go:356] "Generic (PLEG): container finished" podID="d5762b60-b2a0-47e8-8b52-93336610564e" containerID="df38ec7ff0643c45d1c4d5f6f6490ce964b91681086cf36183c8ffc31bc4d27e" exitCode=0 Oct 30 04:00:08.281574 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:08.281505 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" event={"ID":"d5762b60-b2a0-47e8-8b52-93336610564e","Type":"ContainerDied","Data":"df38ec7ff0643c45d1c4d5f6f6490ce964b91681086cf36183c8ffc31bc4d27e"} Oct 30 04:00:08.323852 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:08.323807 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.323789017 podStartE2EDuration="1.323789017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:08.321523694 +0000 UTC m=+4769.334801797" watchObservedRunningTime="2026-10-30 04:00:08.323789017 +0000 UTC m=+4769.337067133" Oct 30 04:00:08.396550 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:08.396528 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" Oct 30 04:00:08.433488 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:08.433455 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5762b60-b2a0-47e8-8b52-93336610564e-tls\" (UniqueName: \"kubernetes.io/secret/d5762b60-b2a0-47e8-8b52-93336610564e-tls\") pod \"d5762b60-b2a0-47e8-8b52-93336610564e\" (UID: \"d5762b60-b2a0-47e8-8b52-93336610564e\") " Oct 30 04:00:08.433597 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:08.433503 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5762b60-b2a0-47e8-8b52-93336610564e-config\" (UniqueName: \"kubernetes.io/configmap/d5762b60-b2a0-47e8-8b52-93336610564e-config\") pod \"d5762b60-b2a0-47e8-8b52-93336610564e\" (UID: \"d5762b60-b2a0-47e8-8b52-93336610564e\") " Oct 30 04:00:08.433597 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:08.433568 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5762b60-b2a0-47e8-8b52-93336610564e-kube-api-access-57sgl\" (UniqueName: \"kubernetes.io/projected/d5762b60-b2a0-47e8-8b52-93336610564e-kube-api-access-57sgl\") pod \"d5762b60-b2a0-47e8-8b52-93336610564e\" (UID: \"d5762b60-b2a0-47e8-8b52-93336610564e\") " Oct 30 04:00:08.433886 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:08.433865 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5762b60-b2a0-47e8-8b52-93336610564e-config" pod "d5762b60-b2a0-47e8-8b52-93336610564e" (UID: "d5762b60-b2a0-47e8-8b52-93336610564e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:08.435492 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:08.435470 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5762b60-b2a0-47e8-8b52-93336610564e-kube-api-access-57sgl" pod "d5762b60-b2a0-47e8-8b52-93336610564e" (UID: "d5762b60-b2a0-47e8-8b52-93336610564e"). InnerVolumeSpecName "kube-api-access-57sgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:08.435556 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:08.435535 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5762b60-b2a0-47e8-8b52-93336610564e-tls" pod "d5762b60-b2a0-47e8-8b52-93336610564e" (UID: "d5762b60-b2a0-47e8-8b52-93336610564e"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:00:08.534470 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:08.534387 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d5762b60-b2a0-47e8-8b52-93336610564e-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:00:08.534470 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:08.534420 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5762b60-b2a0-47e8-8b52-93336610564e-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:00:08.534470 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:08.534430 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57sgl\" (UniqueName: \"kubernetes.io/projected/d5762b60-b2a0-47e8-8b52-93336610564e-kube-api-access-57sgl\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:00:08.542363 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:08.542330 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:00:09.121841 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:09.121809 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5091261-7125-499f-9e8b-e9a04e4c6ea0" path="/var/lib/kubelet/pods/e5091261-7125-499f-9e8b-e9a04e4c6ea0/volumes" Oct 30 04:00:09.178185 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:09.178152 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6"] Oct 30 04:00:09.178405 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:09.178378 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" podUID="f43c74ef-ec4b-433f-809f-8345121068d0" containerName="tempo-gateway" containerID="cri-o://d2d14dce9a4473ea7eca234afad56d677ebe58698809c5f9b495be810ca1d4dd" gracePeriod=30 Oct 30 04:00:09.178497 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:09.178422 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" podUID="f43c74ef-ec4b-433f-809f-8345121068d0" containerName="tempo-gateway-opa" containerID="cri-o://4e63057d6eca5d7db98616feaa677c4c1808e9f956af9082ede301bb90503402" gracePeriod=30 Oct 30 04:00:09.288803 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:09.288701 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" Oct 30 04:00:09.288965 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:09.288801 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk" event={"ID":"d5762b60-b2a0-47e8-8b52-93336610564e","Type":"ContainerDied","Data":"d825888edd15cfb5dbb3846ca3821fe72291a5505817e700d411f2ffe377b95c"} Oct 30 04:00:09.288965 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:09.288852 4925 scope.go:122] "RemoveContainer" containerID="df38ec7ff0643c45d1c4d5f6f6490ce964b91681086cf36183c8ffc31bc4d27e" Oct 30 04:00:09.289378 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:09.289353 4925 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/data-science-collector-collector-0" secret="" err="secret \"data-science-collector-collector-dockercfg-8l25s\" not found" Oct 30 04:00:09.341381 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:00:09.341351 4925 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-monitoring-tls: secret "data-science-collector-monitoring-tls" not found Oct 30 04:00:09.341381 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:00:09.341372 4925 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-collector-90b17333: configmap "data-science-collector-collector-90b17333" not found Oct 30 04:00:09.341611 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:00:09.341433 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe8cbfcc-d492-4704-afba-c9831dc6f365-monitoring-tls-certs podName:fe8cbfcc-d492-4704-afba-c9831dc6f365 nodeName:}" failed. No retries permitted until 2026-10-30 04:00:09.841413591 +0000 UTC m=+4770.854691693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-tls-certs" (UniqueName: "kubernetes.io/secret/fe8cbfcc-d492-4704-afba-c9831dc6f365-monitoring-tls-certs") pod "data-science-collector-collector-0" (UID: "fe8cbfcc-d492-4704-afba-c9831dc6f365") : secret "data-science-collector-monitoring-tls" not found Oct 30 04:00:09.341611 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:00:09.341452 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fe8cbfcc-d492-4704-afba-c9831dc6f365-otc-internal podName:fe8cbfcc-d492-4704-afba-c9831dc6f365 nodeName:}" failed. No retries permitted until 2026-10-30 04:00:09.841444228 +0000 UTC m=+4770.854722308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/fe8cbfcc-d492-4704-afba-c9831dc6f365-otc-internal") pod "data-science-collector-collector-0" (UID: "fe8cbfcc-d492-4704-afba-c9831dc6f365") : configmap "data-science-collector-collector-90b17333" not found Oct 30 04:00:09.346110 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:09.346086 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk"] Oct 30 04:00:09.352163 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:09.352140 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-7khpk"] Oct 30 04:00:09.845201 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:00:09.845168 4925 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-collector-90b17333: configmap "data-science-collector-collector-90b17333" not found Oct 30 04:00:09.845201 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:00:09.845191 4925 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-monitoring-tls: secret "data-science-collector-monitoring-tls" not found Oct 30 04:00:09.845457 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:00:09.845244 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fe8cbfcc-d492-4704-afba-c9831dc6f365-otc-internal podName:fe8cbfcc-d492-4704-afba-c9831dc6f365 nodeName:}" failed. No retries permitted until 2026-10-30 04:00:10.845229199 +0000 UTC m=+4771.858507284 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/fe8cbfcc-d492-4704-afba-c9831dc6f365-otc-internal") pod "data-science-collector-collector-0" (UID: "fe8cbfcc-d492-4704-afba-c9831dc6f365") : configmap "data-science-collector-collector-90b17333" not found Oct 30 04:00:09.845457 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:00:09.845258 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe8cbfcc-d492-4704-afba-c9831dc6f365-monitoring-tls-certs podName:fe8cbfcc-d492-4704-afba-c9831dc6f365 nodeName:}" failed. No retries permitted until 2026-10-30 04:00:10.845252749 +0000 UTC m=+4771.858530835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-tls-certs" (UniqueName: "kubernetes.io/secret/fe8cbfcc-d492-4704-afba-c9831dc6f365-monitoring-tls-certs") pod "data-science-collector-collector-0" (UID: "fe8cbfcc-d492-4704-afba-c9831dc6f365") : secret "data-science-collector-monitoring-tls" not found Oct 30 04:00:10.295241 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.295212 4925 generic.go:356] "Generic (PLEG): container finished" podID="f43c74ef-ec4b-433f-809f-8345121068d0" containerID="4e63057d6eca5d7db98616feaa677c4c1808e9f956af9082ede301bb90503402" exitCode=0 Oct 30 04:00:10.295241 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.295241 4925 generic.go:356] "Generic (PLEG): container finished" podID="f43c74ef-ec4b-433f-809f-8345121068d0" containerID="d2d14dce9a4473ea7eca234afad56d677ebe58698809c5f9b495be810ca1d4dd" exitCode=0 Oct 30 04:00:10.295636 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.295279 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" event={"ID":"f43c74ef-ec4b-433f-809f-8345121068d0","Type":"ContainerDied","Data":"4e63057d6eca5d7db98616feaa677c4c1808e9f956af9082ede301bb90503402"} Oct 30 04:00:10.295636 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.295337 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" event={"ID":"f43c74ef-ec4b-433f-809f-8345121068d0","Type":"ContainerDied","Data":"d2d14dce9a4473ea7eca234afad56d677ebe58698809c5f9b495be810ca1d4dd"} Oct 30 04:00:10.296499 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.296432 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="fe8cbfcc-d492-4704-afba-c9831dc6f365" containerName="otc-container" containerID="cri-o://9768512a3ba8963a8399e756a8e16e55b4128df39f2603e196ca996f0ffd52b9" gracePeriod=30 Oct 30 04:00:10.440214 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.440188 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 04:00:10.540067 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.540043 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:10.551300 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.551274 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-tempo-data-science-tempostack-ca-bundle\") pod \"f43c74ef-ec4b-433f-809f-8345121068d0\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " Oct 30 04:00:10.551401 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.551329 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43c74ef-ec4b-433f-809f-8345121068d0-kube-api-access-rtsnh\" (UniqueName: \"kubernetes.io/projected/f43c74ef-ec4b-433f-809f-8345121068d0-kube-api-access-rtsnh\") pod \"f43c74ef-ec4b-433f-809f-8345121068d0\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " Oct 30 04:00:10.551401 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.551362 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-cabundle\" (UniqueName: \"kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-cabundle\") pod \"f43c74ef-ec4b-433f-809f-8345121068d0\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " Oct 30 04:00:10.551516 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.551402 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-serving-certs\" (UniqueName: \"kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-serving-certs\") pod \"f43c74ef-ec4b-433f-809f-8345121068d0\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " Oct 30 04:00:10.551516 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.551452 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-tenant\" (UniqueName: \"kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-tenant\") pod \"f43c74ef-ec4b-433f-809f-8345121068d0\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " Oct 30 04:00:10.551629 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.551517 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-rbac\" (UniqueName: \"kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-rbac\") pod \"f43c74ef-ec4b-433f-809f-8345121068d0\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " Oct 30 04:00:10.551629 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.551546 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-tempo-data-science-tempostack-gateway-mtls\") pod \"f43c74ef-ec4b-433f-809f-8345121068d0\" (UID: \"f43c74ef-ec4b-433f-809f-8345121068d0\") " Oct 30 04:00:10.551736 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.551612 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-tempo-data-science-tempostack-ca-bundle" pod "f43c74ef-ec4b-433f-809f-8345121068d0" (UID: "f43c74ef-ec4b-433f-809f-8345121068d0"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:10.551943 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.551920 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-cabundle" pod "f43c74ef-ec4b-433f-809f-8345121068d0" (UID: "f43c74ef-ec4b-433f-809f-8345121068d0"). InnerVolumeSpecName "cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:10.552012 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.551947 4925 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:00:10.552071 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.552033 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-rbac" pod "f43c74ef-ec4b-433f-809f-8345121068d0" (UID: "f43c74ef-ec4b-433f-809f-8345121068d0"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:10.553598 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.553573 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-serving-certs" pod "f43c74ef-ec4b-433f-809f-8345121068d0" (UID: "f43c74ef-ec4b-433f-809f-8345121068d0"). InnerVolumeSpecName "serving-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:00:10.553598 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.553592 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-tempo-data-science-tempostack-gateway-mtls" pod "f43c74ef-ec4b-433f-809f-8345121068d0" (UID: "f43c74ef-ec4b-433f-809f-8345121068d0"). InnerVolumeSpecName "tempo-data-science-tempostack-gateway-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:00:10.553835 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.553619 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43c74ef-ec4b-433f-809f-8345121068d0-kube-api-access-rtsnh" pod "f43c74ef-ec4b-433f-809f-8345121068d0" (UID: "f43c74ef-ec4b-433f-809f-8345121068d0"). InnerVolumeSpecName "kube-api-access-rtsnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:10.557577 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.557558 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-tenant" pod "f43c74ef-ec4b-433f-809f-8345121068d0" (UID: "f43c74ef-ec4b-433f-809f-8345121068d0"). InnerVolumeSpecName "tenant". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:00:10.652482 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.652454 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe8cbfcc-d492-4704-afba-c9831dc6f365-kube-api-access-gshgz\" (UniqueName: \"kubernetes.io/projected/fe8cbfcc-d492-4704-afba-c9831dc6f365-kube-api-access-gshgz\") pod \"fe8cbfcc-d492-4704-afba-c9831dc6f365\" (UID: \"fe8cbfcc-d492-4704-afba-c9831dc6f365\") " Oct 30 04:00:10.652599 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.652495 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe8cbfcc-d492-4704-afba-c9831dc6f365-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8cbfcc-d492-4704-afba-c9831dc6f365-monitoring-tls-certs\") pod \"fe8cbfcc-d492-4704-afba-c9831dc6f365\" (UID: \"fe8cbfcc-d492-4704-afba-c9831dc6f365\") " Oct 30 04:00:10.652699 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.652681 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe8cbfcc-d492-4704-afba-c9831dc6f365-otc-internal\" (UniqueName: \"kubernetes.io/configmap/fe8cbfcc-d492-4704-afba-c9831dc6f365-otc-internal\") pod \"fe8cbfcc-d492-4704-afba-c9831dc6f365\" (UID: \"fe8cbfcc-d492-4704-afba-c9831dc6f365\") " Oct 30 04:00:10.652923 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.652907 4925 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-tempo-data-science-tempostack-gateway-mtls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:00:10.653013 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.652924 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtsnh\" (UniqueName: \"kubernetes.io/projected/f43c74ef-ec4b-433f-809f-8345121068d0-kube-api-access-rtsnh\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:00:10.653013 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.652935 4925 reconciler_common.go:299] "Volume detached for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-cabundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:00:10.653013 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.652944 4925 reconciler_common.go:299] "Volume detached for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-serving-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:00:10.653013 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.652955 4925 reconciler_common.go:299] "Volume detached for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/f43c74ef-ec4b-433f-809f-8345121068d0-tenant\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:00:10.653013 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.652966 4925 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f43c74ef-ec4b-433f-809f-8345121068d0-rbac\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:00:10.653227 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.653063 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe8cbfcc-d492-4704-afba-c9831dc6f365-otc-internal" pod "fe8cbfcc-d492-4704-afba-c9831dc6f365" (UID: "fe8cbfcc-d492-4704-afba-c9831dc6f365"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:10.654393 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.654372 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8cbfcc-d492-4704-afba-c9831dc6f365-kube-api-access-gshgz" pod "fe8cbfcc-d492-4704-afba-c9831dc6f365" (UID: "fe8cbfcc-d492-4704-afba-c9831dc6f365"). InnerVolumeSpecName "kube-api-access-gshgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:10.654498 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.654404 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe8cbfcc-d492-4704-afba-c9831dc6f365-monitoring-tls-certs" pod "fe8cbfcc-d492-4704-afba-c9831dc6f365" (UID: "fe8cbfcc-d492-4704-afba-c9831dc6f365"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:00:10.753785 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.753765 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fe8cbfcc-d492-4704-afba-c9831dc6f365-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:00:10.753785 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.753785 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gshgz\" (UniqueName: \"kubernetes.io/projected/fe8cbfcc-d492-4704-afba-c9831dc6f365-kube-api-access-gshgz\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:00:10.753920 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:10.753795 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8cbfcc-d492-4704-afba-c9831dc6f365-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:00:11.119971 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:11.119938 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5762b60-b2a0-47e8-8b52-93336610564e" path="/var/lib/kubelet/pods/d5762b60-b2a0-47e8-8b52-93336610564e/volumes" Oct 30 04:00:11.302379 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:11.302343 4925 generic.go:356] "Generic (PLEG): container finished" podID="fe8cbfcc-d492-4704-afba-c9831dc6f365" containerID="9768512a3ba8963a8399e756a8e16e55b4128df39f2603e196ca996f0ffd52b9" exitCode=0 Oct 30 04:00:11.302846 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:11.302393 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:11.302846 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:11.302425 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"fe8cbfcc-d492-4704-afba-c9831dc6f365","Type":"ContainerDied","Data":"9768512a3ba8963a8399e756a8e16e55b4128df39f2603e196ca996f0ffd52b9"} Oct 30 04:00:11.302846 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:11.302477 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"fe8cbfcc-d492-4704-afba-c9831dc6f365","Type":"ContainerDied","Data":"bdb154058553acd642effcf30acb540d03d981c7f613183da588a352ece3cf36"} Oct 30 04:00:11.302846 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:11.302504 4925 scope.go:122] "RemoveContainer" containerID="9768512a3ba8963a8399e756a8e16e55b4128df39f2603e196ca996f0ffd52b9" Oct 30 04:00:11.304405 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:11.304382 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" event={"ID":"f43c74ef-ec4b-433f-809f-8345121068d0","Type":"ContainerDied","Data":"7f9d74b09d95f6f0021f0bc2259d1b8a619b7968eadd7091ad8f3c3ecfca0f2b"} Oct 30 04:00:11.304405 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:11.304404 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6" Oct 30 04:00:11.311569 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:11.311551 4925 scope.go:122] "RemoveContainer" containerID="9768512a3ba8963a8399e756a8e16e55b4128df39f2603e196ca996f0ffd52b9" Oct 30 04:00:11.311859 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:00:11.311837 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9768512a3ba8963a8399e756a8e16e55b4128df39f2603e196ca996f0ffd52b9\": container with ID starting with 9768512a3ba8963a8399e756a8e16e55b4128df39f2603e196ca996f0ffd52b9 not found: ID does not exist" containerID="9768512a3ba8963a8399e756a8e16e55b4128df39f2603e196ca996f0ffd52b9" Oct 30 04:00:11.311970 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:11.311868 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9768512a3ba8963a8399e756a8e16e55b4128df39f2603e196ca996f0ffd52b9"} err="failed to get container status \"9768512a3ba8963a8399e756a8e16e55b4128df39f2603e196ca996f0ffd52b9\": rpc error: code = NotFound desc = could not find container \"9768512a3ba8963a8399e756a8e16e55b4128df39f2603e196ca996f0ffd52b9\": container with ID starting with 9768512a3ba8963a8399e756a8e16e55b4128df39f2603e196ca996f0ffd52b9 not found: ID does not exist" Oct 30 04:00:11.311970 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:11.311890 4925 scope.go:122] "RemoveContainer" containerID="4e63057d6eca5d7db98616feaa677c4c1808e9f956af9082ede301bb90503402" Oct 30 04:00:11.319677 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:11.319662 4925 scope.go:122] "RemoveContainer" containerID="d2d14dce9a4473ea7eca234afad56d677ebe58698809c5f9b495be810ca1d4dd" Oct 30 04:00:11.354904 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:11.354885 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6"] Oct 30 04:00:11.368115 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:11.368095 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6f76fdcc67-6drg6"] Oct 30 04:00:11.402710 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:11.402663 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:00:11.408976 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:11.408953 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:00:13.119087 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:13.119056 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f43c74ef-ec4b-433f-809f-8345121068d0" path="/var/lib/kubelet/pods/f43c74ef-ec4b-433f-809f-8345121068d0/volumes" Oct 30 04:00:13.119605 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:13.119589 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe8cbfcc-d492-4704-afba-c9831dc6f365" path="/var/lib/kubelet/pods/fe8cbfcc-d492-4704-afba-c9831dc6f365/volumes" Oct 30 04:00:18.161190 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:00:18.161154 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:00:39.613101 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:39.612997 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 04:00:39.621684 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:39.621652 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 04:00:39.627294 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:39.627276 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 04:00:39.635263 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:39.635246 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 04:00:48.161995 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:00:48.161952 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:00:53.804903 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.804829 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n"] Oct 30 04:00:53.805282 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.805154 4925 cpu_manager.go:414] "containerMap: removing container" podUID="d5762b60-b2a0-47e8-8b52-93336610564e" containerName="korrel8r" Oct 30 04:00:53.805282 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.805167 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5762b60-b2a0-47e8-8b52-93336610564e" containerName="korrel8r" Oct 30 04:00:53.805282 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.805190 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f43c74ef-ec4b-433f-809f-8345121068d0" containerName="tempo-gateway-opa" Oct 30 04:00:53.805282 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.805195 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43c74ef-ec4b-433f-809f-8345121068d0" containerName="tempo-gateway-opa" Oct 30 04:00:53.805282 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.805202 4925 cpu_manager.go:414] "containerMap: removing container" podUID="fe8cbfcc-d492-4704-afba-c9831dc6f365" containerName="otc-container" Oct 30 04:00:53.805282 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.805208 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe8cbfcc-d492-4704-afba-c9831dc6f365" containerName="otc-container" Oct 30 04:00:53.805282 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.805220 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f43c74ef-ec4b-433f-809f-8345121068d0" containerName="tempo-gateway" Oct 30 04:00:53.805282 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.805225 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43c74ef-ec4b-433f-809f-8345121068d0" containerName="tempo-gateway" Oct 30 04:00:53.805282 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.805280 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5762b60-b2a0-47e8-8b52-93336610564e" containerName="korrel8r" Oct 30 04:00:53.805282 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.805289 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43c74ef-ec4b-433f-809f-8345121068d0" containerName="tempo-gateway-opa" Oct 30 04:00:53.805644 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.805295 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe8cbfcc-d492-4704-afba-c9831dc6f365" containerName="otc-container" Oct 30 04:00:53.805644 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.805301 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43c74ef-ec4b-433f-809f-8345121068d0" containerName="tempo-gateway" Oct 30 04:00:53.808300 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.808281 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" Oct 30 04:00:53.827031 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.827005 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 04:00:53.829450 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.829427 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n"] Oct 30 04:00:53.839210 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.839186 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 04:00:53.839302 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.839247 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-p4jhb\"" Oct 30 04:00:53.839302 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.839257 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 04:00:53.839410 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.839395 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 04:00:53.880169 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.880144 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/052f928b-5f7e-409d-b037-ed9af947a8fe-tls\") pod \"korrel8r-5d45699dc4-5np8n\" (UID: \"052f928b-5f7e-409d-b037-ed9af947a8fe\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" Oct 30 04:00:53.880300 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.880193 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-854p2\" (UniqueName: \"kubernetes.io/projected/052f928b-5f7e-409d-b037-ed9af947a8fe-kube-api-access-854p2\") pod \"korrel8r-5d45699dc4-5np8n\" (UID: \"052f928b-5f7e-409d-b037-ed9af947a8fe\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" Oct 30 04:00:53.880374 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.880329 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/052f928b-5f7e-409d-b037-ed9af947a8fe-config\") pod \"korrel8r-5d45699dc4-5np8n\" (UID: \"052f928b-5f7e-409d-b037-ed9af947a8fe\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" Oct 30 04:00:53.980987 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.980958 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-854p2\" (UniqueName: \"kubernetes.io/projected/052f928b-5f7e-409d-b037-ed9af947a8fe-kube-api-access-854p2\") pod \"korrel8r-5d45699dc4-5np8n\" (UID: \"052f928b-5f7e-409d-b037-ed9af947a8fe\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" Oct 30 04:00:53.981158 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.981049 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/052f928b-5f7e-409d-b037-ed9af947a8fe-config\") pod \"korrel8r-5d45699dc4-5np8n\" (UID: \"052f928b-5f7e-409d-b037-ed9af947a8fe\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" Oct 30 04:00:53.981158 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.981088 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/052f928b-5f7e-409d-b037-ed9af947a8fe-tls\") pod \"korrel8r-5d45699dc4-5np8n\" (UID: \"052f928b-5f7e-409d-b037-ed9af947a8fe\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" Oct 30 04:00:53.981728 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.981706 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/052f928b-5f7e-409d-b037-ed9af947a8fe-config\") pod \"korrel8r-5d45699dc4-5np8n\" (UID: \"052f928b-5f7e-409d-b037-ed9af947a8fe\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" Oct 30 04:00:53.983646 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:53.983619 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/052f928b-5f7e-409d-b037-ed9af947a8fe-tls\") pod \"korrel8r-5d45699dc4-5np8n\" (UID: \"052f928b-5f7e-409d-b037-ed9af947a8fe\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" Oct 30 04:00:54.004607 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.004584 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-854p2\" (UniqueName: \"kubernetes.io/projected/052f928b-5f7e-409d-b037-ed9af947a8fe-kube-api-access-854p2\") pod \"korrel8r-5d45699dc4-5np8n\" (UID: \"052f928b-5f7e-409d-b037-ed9af947a8fe\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" Oct 30 04:00:54.121186 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.121158 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" Oct 30 04:00:54.420429 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.420406 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n"] Oct 30 04:00:54.422059 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:00:54.422030 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052f928b_5f7e_409d_b037_ed9af947a8fe.slice/crio-1ba020e8037b8e26edcab9fdc1f3f15238baaacef9f2d6417f5e1a42b89f5cc1 WatchSource:0}: Error finding container 1ba020e8037b8e26edcab9fdc1f3f15238baaacef9f2d6417f5e1a42b89f5cc1: Status 404 returned error can't find the container with id 1ba020e8037b8e26edcab9fdc1f3f15238baaacef9f2d6417f5e1a42b89f5cc1 Oct 30 04:00:54.454369 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.454340 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:00:54.459685 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.459573 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:54.479966 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.479942 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:00:54.480308 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.480040 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-90b17333\"" Oct 30 04:00:54.480650 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.480628 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:00:54.485339 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.485313 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7gvd\" (UniqueName: \"kubernetes.io/projected/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-kube-api-access-q7gvd\") pod \"data-science-collector-collector-0\" (UID: \"f76a3ac9-2cfa-493c-8a85-7c1a161419ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:54.485447 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.485358 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f76a3ac9-2cfa-493c-8a85-7c1a161419ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:54.485447 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.485388 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"f76a3ac9-2cfa-493c-8a85-7c1a161419ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:54.517995 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.517959 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" event={"ID":"052f928b-5f7e-409d-b037-ed9af947a8fe","Type":"ContainerStarted","Data":"c40b111ad49bb06e73b900ffa25d036457ee57ea7e0fa0cc9d2e32be698944d6"} Oct 30 04:00:54.518131 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.518009 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" event={"ID":"052f928b-5f7e-409d-b037-ed9af947a8fe","Type":"ContainerStarted","Data":"1ba020e8037b8e26edcab9fdc1f3f15238baaacef9f2d6417f5e1a42b89f5cc1"} Oct 30 04:00:54.518131 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.518093 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" Oct 30 04:00:54.561258 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.561201 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" podStartSLOduration=1.561184055 podStartE2EDuration="1.561184055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:54.559366935 +0000 UTC m=+4815.572645040" watchObservedRunningTime="2026-10-30 04:00:54.561184055 +0000 UTC m=+4815.574462161" Oct 30 04:00:54.585688 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.585660 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7gvd\" (UniqueName: \"kubernetes.io/projected/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-kube-api-access-q7gvd\") pod \"data-science-collector-collector-0\" (UID: \"f76a3ac9-2cfa-493c-8a85-7c1a161419ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:54.585862 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.585697 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f76a3ac9-2cfa-493c-8a85-7c1a161419ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:54.585862 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.585714 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"f76a3ac9-2cfa-493c-8a85-7c1a161419ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:54.586398 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.586377 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"f76a3ac9-2cfa-493c-8a85-7c1a161419ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:54.588201 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.588182 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f76a3ac9-2cfa-493c-8a85-7c1a161419ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:54.611986 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.611965 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7gvd\" (UniqueName: \"kubernetes.io/projected/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-kube-api-access-q7gvd\") pod \"data-science-collector-collector-0\" (UID: \"f76a3ac9-2cfa-493c-8a85-7c1a161419ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:54.778690 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:54.778605 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:00:55.078327 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:55.078299 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:00:55.080473 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:00:55.080442 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76a3ac9_2cfa_493c_8a85_7c1a161419ef.slice/crio-085f4ed9459e67f1127aebf59cb7f7320317891ae86ce5e5f2fdc01acdc60fdc WatchSource:0}: Error finding container 085f4ed9459e67f1127aebf59cb7f7320317891ae86ce5e5f2fdc01acdc60fdc: Status 404 returned error can't find the container with id 085f4ed9459e67f1127aebf59cb7f7320317891ae86ce5e5f2fdc01acdc60fdc Oct 30 04:00:55.082719 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:55.082686 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:00:55.082823 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:55.082779 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:00:55.526025 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:55.525930 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"f76a3ac9-2cfa-493c-8a85-7c1a161419ef","Type":"ContainerStarted","Data":"1a2f55183bec0bf4f24fb3377ad2a74a398ad8ad0bd05dbae00217aab255ef83"} Oct 30 04:00:55.526025 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:55.525989 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"f76a3ac9-2cfa-493c-8a85-7c1a161419ef","Type":"ContainerStarted","Data":"085f4ed9459e67f1127aebf59cb7f7320317891ae86ce5e5f2fdc01acdc60fdc"} Oct 30 04:00:55.580093 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:55.580020 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.580002779 podStartE2EDuration="1.580002779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:55.567313348 +0000 UTC m=+4816.580591493" watchObservedRunningTime="2026-10-30 04:00:55.580002779 +0000 UTC m=+4816.593280882" Oct 30 04:00:58.338996 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.338960 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp"] Oct 30 04:00:58.342456 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.342436 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:00:58.370852 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.370825 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\"" Oct 30 04:00:58.370852 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.370845 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-distributor-mtls\"" Oct 30 04:00:58.371103 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.371072 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-dockercfg-hk87b\"" Oct 30 04:00:58.371245 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.371166 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack\"" Oct 30 04:00:58.372764 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.372729 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp"] Oct 30 04:00:58.415618 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.415586 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp\" (UID: \"2437ead6-6d92-4324-9cca-a355c73a6646\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:00:58.415839 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.415637 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp\" (UID: \"2437ead6-6d92-4324-9cca-a355c73a6646\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:00:58.415839 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.415742 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp\" (UID: \"2437ead6-6d92-4324-9cca-a355c73a6646\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:00:58.415839 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.415831 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp\" (UID: \"2437ead6-6d92-4324-9cca-a355c73a6646\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:00:58.415956 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.415864 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmcdr\" (UniqueName: \"kubernetes.io/projected/2437ead6-6d92-4324-9cca-a355c73a6646-kube-api-access-qmcdr\") pod \"tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp\" (UID: \"2437ead6-6d92-4324-9cca-a355c73a6646\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:00:58.426605 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.426578 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4"] Oct 30 04:00:58.430048 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.430029 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:00:58.454780 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.454733 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4"] Oct 30 04:00:58.456643 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.456620 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx"] Oct 30 04:00:58.458493 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.458463 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"gcs-secret\"" Oct 30 04:00:58.458719 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.458699 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-query-frontend-mtls\"" Oct 30 04:00:58.460208 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.460193 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:00:58.483114 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.483090 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx"] Oct 30 04:00:58.491052 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.491029 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq"] Oct 30 04:00:58.494410 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.494391 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:00:58.515490 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.515465 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq"] Oct 30 04:00:58.516242 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.516222 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:00:58.516346 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.516258 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:00:58.516346 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.516292 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp\" (UID: \"2437ead6-6d92-4324-9cca-a355c73a6646\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:00:58.516458 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.516349 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:00:58.516528 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.516483 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kmd4\" (UniqueName: \"kubernetes.io/projected/d0480bc1-0f02-40a0-9ff9-dd717975b349-kube-api-access-5kmd4\") pod \"tempo-data-science-tempostack-querier-7c5b99b89b-n42qx\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:00:58.516587 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.516561 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-7c5b99b89b-n42qx\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:00:58.516644 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.516600 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:00:58.516644 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.516635 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmcdr\" (UniqueName: \"kubernetes.io/projected/2437ead6-6d92-4324-9cca-a355c73a6646-kube-api-access-qmcdr\") pod \"tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp\" (UID: \"2437ead6-6d92-4324-9cca-a355c73a6646\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:00:58.516764 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.516660 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6v4n\" (UniqueName: \"kubernetes.io/projected/d52093ca-d97a-4e66-ac82-3a52d050525f-kube-api-access-l6v4n\") pod \"tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:00:58.516764 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.516679 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp\" (UID: \"2437ead6-6d92-4324-9cca-a355c73a6646\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:00:58.516842 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.516770 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp\" (UID: \"2437ead6-6d92-4324-9cca-a355c73a6646\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:00:58.516880 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.516839 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp\" (UID: \"2437ead6-6d92-4324-9cca-a355c73a6646\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:00:58.516880 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.516872 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:00:58.516946 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.516913 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-7c5b99b89b-n42qx\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:00:58.516981 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.516949 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:00:58.517022 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.516976 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-conf\") pod \"tempo-data-science-tempostack-querier-7c5b99b89b-n42qx\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:00:58.517022 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.517009 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/d0480bc1-0f02-40a0-9ff9-dd717975b349-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-7c5b99b89b-n42qx\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:00:58.517099 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.517033 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:00:58.517099 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.517057 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wggpw\" (UniqueName: \"kubernetes.io/projected/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-kube-api-access-wggpw\") pod \"tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:00:58.517662 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.517638 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:00:58.517794 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.517683 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:00:58.517794 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.517713 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/d52093ca-d97a-4e66-ac82-3a52d050525f-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:00:58.517794 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.517767 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp\" (UID: \"2437ead6-6d92-4324-9cca-a355c73a6646\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:00:58.517948 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.517801 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp\" (UID: \"2437ead6-6d92-4324-9cca-a355c73a6646\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:00:58.517948 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.517830 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-7c5b99b89b-n42qx\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:00:58.518483 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.518459 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp\" (UID: \"2437ead6-6d92-4324-9cca-a355c73a6646\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:00:58.519699 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.519673 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp\" (UID: \"2437ead6-6d92-4324-9cca-a355c73a6646\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:00:58.540072 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.540051 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmcdr\" (UniqueName: \"kubernetes.io/projected/2437ead6-6d92-4324-9cca-a355c73a6646-kube-api-access-qmcdr\") pod \"tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp\" (UID: \"2437ead6-6d92-4324-9cca-a355c73a6646\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:00:58.558911 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.558887 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-querier-mtls\"" Oct 30 04:00:58.559023 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.558909 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-compactor-mtls\"" Oct 30 04:00:58.560713 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.560691 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4"] Oct 30 04:00:58.565408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.565390 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.583342 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.583318 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 30 04:00:58.583486 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.583470 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-cabundle\"" Oct 30 04:00:58.585588 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.585559 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4"] Oct 30 04:00:58.619076 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.619013 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-rbac\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.619076 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.619046 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.619214 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.619112 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-7c5b99b89b-n42qx\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:00:58.619214 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.619141 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:00:58.619214 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.619158 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-conf\") pod \"tempo-data-science-tempostack-querier-7c5b99b89b-n42qx\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:00:58.619214 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.619175 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/d0480bc1-0f02-40a0-9ff9-dd717975b349-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-7c5b99b89b-n42qx\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:00:58.619214 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.619192 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:00:58.619214 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.619209 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wggpw\" (UniqueName: \"kubernetes.io/projected/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-kube-api-access-wggpw\") pod \"tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:00:58.619517 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.619233 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:00:58.619517 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.619261 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:00:58.619517 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.619290 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.619517 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.619316 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghfwd\" (UniqueName: \"kubernetes.io/projected/d4fec03a-ad3e-4bfc-9910-41485267ff19-kube-api-access-ghfwd\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.619735 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.619561 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/d52093ca-d97a-4e66-ac82-3a52d050525f-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:00:58.619735 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.619600 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:00:58.620208 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.620077 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-7c5b99b89b-n42qx\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:00:58.620208 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.620110 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:00:58.620208 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.619605 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-7c5b99b89b-n42qx\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:00:58.620208 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.620144 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-7c5b99b89b-n42qx\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:00:58.620208 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.620166 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-tenant\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.620208 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.620198 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:00:58.620592 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.620253 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:00:58.620592 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.620283 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:00:58.620592 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.620310 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kmd4\" (UniqueName: \"kubernetes.io/projected/d0480bc1-0f02-40a0-9ff9-dd717975b349-kube-api-access-5kmd4\") pod \"tempo-data-science-tempostack-querier-7c5b99b89b-n42qx\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:00:58.620592 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.620351 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-7c5b99b89b-n42qx\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:00:58.620592 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.620366 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-conf\") pod \"tempo-data-science-tempostack-querier-7c5b99b89b-n42qx\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:00:58.620592 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.620371 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:00:58.620592 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.620396 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:00:58.620592 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.620443 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6v4n\" (UniqueName: \"kubernetes.io/projected/d52093ca-d97a-4e66-ac82-3a52d050525f-kube-api-access-l6v4n\") pod \"tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:00:58.620592 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.620475 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-serving-certs\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.620592 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.620571 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:00:58.621239 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.620600 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-cabundle\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.621239 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.621047 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:00:58.621239 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.621203 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:00:58.621511 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.621486 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:00:58.622413 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.622385 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/d52093ca-d97a-4e66-ac82-3a52d050525f-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:00:58.622413 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.622397 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:00:58.622613 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.622590 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/d0480bc1-0f02-40a0-9ff9-dd717975b349-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-7c5b99b89b-n42qx\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:00:58.622964 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.622931 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:00:58.623351 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.623332 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:00:58.623919 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.623893 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-7c5b99b89b-n42qx\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:00:58.642563 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.642540 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wggpw\" (UniqueName: \"kubernetes.io/projected/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-kube-api-access-wggpw\") pod \"tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:00:58.643558 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.643538 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6v4n\" (UniqueName: \"kubernetes.io/projected/d52093ca-d97a-4e66-ac82-3a52d050525f-kube-api-access-l6v4n\") pod \"tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:00:58.644103 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.644082 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kmd4\" (UniqueName: \"kubernetes.io/projected/d0480bc1-0f02-40a0-9ff9-dd717975b349-kube-api-access-5kmd4\") pod \"tempo-data-science-tempostack-querier-7c5b99b89b-n42qx\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:00:58.654781 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.654759 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:00:58.659956 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.659935 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 30 04:00:58.660050 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.659967 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-dockercfg-kctfp\"" Oct 30 04:00:58.660050 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.659995 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-tls\"" Oct 30 04:00:58.660050 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.660026 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-mtls\"" Oct 30 04:00:58.721767 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.721722 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-serving-certs\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.721897 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.721836 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-cabundle\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.722022 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.722004 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-rbac\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.722090 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.722039 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.722130 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.722091 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.722130 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.722118 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghfwd\" (UniqueName: \"kubernetes.io/projected/d4fec03a-ad3e-4bfc-9910-41485267ff19-kube-api-access-ghfwd\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.722197 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.722155 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-tenant\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.722684 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.722658 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-cabundle\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.723032 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.723004 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.723250 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.723231 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-rbac\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.724847 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.724778 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-serving-certs\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.725171 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.725133 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.725934 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.725903 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-tenant\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.742803 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.742783 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:00:58.746966 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.746858 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghfwd\" (UniqueName: \"kubernetes.io/projected/d4fec03a-ad3e-4bfc-9910-41485267ff19-kube-api-access-ghfwd\") pod \"tempo-data-science-tempostack-gateway-746f7f6687-rsdl4\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.773072 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.773044 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:00:58.808408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.808378 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:00:58.881316 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.881050 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:00:58.968960 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:58.968927 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp"] Oct 30 04:00:58.969446 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:00:58.969416 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2437ead6_6d92_4324_9cca_a355c73a6646.slice/crio-c9408d72eb03cb3494eac70d839b89f482c2bedbbaa00a08f6976555a39d2016 WatchSource:0}: Error finding container c9408d72eb03cb3494eac70d839b89f482c2bedbbaa00a08f6976555a39d2016: Status 404 returned error can't find the container with id c9408d72eb03cb3494eac70d839b89f482c2bedbbaa00a08f6976555a39d2016 Oct 30 04:00:59.052214 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:59.052184 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4"] Oct 30 04:00:59.053721 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:00:59.053696 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52093ca_d97a_4e66_ac82_3a52d050525f.slice/crio-a2f9693605afb6c11e99faa4af866307a433c70536fa649864c9235ec1fe15dd WatchSource:0}: Error finding container a2f9693605afb6c11e99faa4af866307a433c70536fa649864c9235ec1fe15dd: Status 404 returned error can't find the container with id a2f9693605afb6c11e99faa4af866307a433c70536fa649864c9235ec1fe15dd Oct 30 04:00:59.079623 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:59.079602 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx"] Oct 30 04:00:59.081301 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:00:59.081276 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0480bc1_0f02_40a0_9ff9_dd717975b349.slice/crio-ea7e233140c655bba6f5c3b69c93f55929a170a99bdc9e8f3160b335fb5daefe WatchSource:0}: Error finding container ea7e233140c655bba6f5c3b69c93f55929a170a99bdc9e8f3160b335fb5daefe: Status 404 returned error can't find the container with id ea7e233140c655bba6f5c3b69c93f55929a170a99bdc9e8f3160b335fb5daefe Oct 30 04:00:59.123451 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:59.123429 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq"] Oct 30 04:00:59.124982 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:00:59.124952 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68f2b05_a8fc_4c2f_bf74_b4c0f11e76ed.slice/crio-05eb3686933a4067bbaa8898ec8f442002dfa250b33732d0f85914ec6cd79943 WatchSource:0}: Error finding container 05eb3686933a4067bbaa8898ec8f442002dfa250b33732d0f85914ec6cd79943: Status 404 returned error can't find the container with id 05eb3686933a4067bbaa8898ec8f442002dfa250b33732d0f85914ec6cd79943 Oct 30 04:00:59.190370 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:59.190348 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4"] Oct 30 04:00:59.192066 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:00:59.192042 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fec03a_ad3e_4bfc_9910_41485267ff19.slice/crio-d36beecbca7f6145b5113de58d6027179a6eb04b977964efc60b00a91f95fca0 WatchSource:0}: Error finding container d36beecbca7f6145b5113de58d6027179a6eb04b977964efc60b00a91f95fca0: Status 404 returned error can't find the container with id d36beecbca7f6145b5113de58d6027179a6eb04b977964efc60b00a91f95fca0 Oct 30 04:00:59.550153 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:59.550115 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" event={"ID":"d52093ca-d97a-4e66-ac82-3a52d050525f","Type":"ContainerStarted","Data":"a2f9693605afb6c11e99faa4af866307a433c70536fa649864c9235ec1fe15dd"} Oct 30 04:00:59.551503 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:59.551476 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" event={"ID":"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed","Type":"ContainerStarted","Data":"05eb3686933a4067bbaa8898ec8f442002dfa250b33732d0f85914ec6cd79943"} Oct 30 04:00:59.552951 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:59.552933 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" event={"ID":"d4fec03a-ad3e-4bfc-9910-41485267ff19","Type":"ContainerStarted","Data":"02f6972dc85b69a4abb45211ab0187e6dcf4e0011025ce0e0c7e0e7ecb5ee96e"} Oct 30 04:00:59.553022 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:59.552978 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" event={"ID":"d4fec03a-ad3e-4bfc-9910-41485267ff19","Type":"ContainerStarted","Data":"d36beecbca7f6145b5113de58d6027179a6eb04b977964efc60b00a91f95fca0"} Oct 30 04:00:59.554174 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:59.554151 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" event={"ID":"2437ead6-6d92-4324-9cca-a355c73a6646","Type":"ContainerStarted","Data":"c9408d72eb03cb3494eac70d839b89f482c2bedbbaa00a08f6976555a39d2016"} Oct 30 04:00:59.555398 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:00:59.555378 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" event={"ID":"d0480bc1-0f02-40a0-9ff9-dd717975b349","Type":"ContainerStarted","Data":"ea7e233140c655bba6f5c3b69c93f55929a170a99bdc9e8f3160b335fb5daefe"} Oct 30 04:01:00.566560 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:00.566519 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" event={"ID":"d4fec03a-ad3e-4bfc-9910-41485267ff19","Type":"ContainerStarted","Data":"a5660ca25a190b08c34e24c02f2f465cb1e00302472395006c77a8c57556f54d"} Oct 30 04:01:00.567990 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:00.567661 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:01:00.567990 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:00.567712 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:01:00.570104 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:00.569844 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" podUID="d4fec03a-ad3e-4bfc-9910-41485267ff19" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.97:8083/ready\": dial tcp 10.132.0.97:8083: connect: connection refused" Oct 30 04:01:00.641383 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:00.641321 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" podStartSLOduration=2.641301527 podStartE2EDuration="2.641301527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:00.626327032 +0000 UTC m=+4821.639605138" watchObservedRunningTime="2026-10-30 04:01:00.641301527 +0000 UTC m=+4821.654579631" Oct 30 04:01:01.568412 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:01.568349 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" podUID="d4fec03a-ad3e-4bfc-9910-41485267ff19" containerName="tempo-gateway" probeResult="failure" output="Get \"https://10.132.0.97:8081/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 04:01:01.575175 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:01.575082 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" podUID="d4fec03a-ad3e-4bfc-9910-41485267ff19" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.97:8083/ready\": dial tcp 10.132.0.97:8083: connect: connection refused" Oct 30 04:01:01.830401 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:01.830132 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:01:02.581781 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:02.581739 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" event={"ID":"2437ead6-6d92-4324-9cca-a355c73a6646","Type":"ContainerStarted","Data":"efaa7ceb7d5929deffe440f046904a07c52d2d8c05b05aa44c6f35da4d49c0db"} Oct 30 04:01:02.582187 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:02.581876 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:01:02.584271 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:02.584249 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" event={"ID":"d0480bc1-0f02-40a0-9ff9-dd717975b349","Type":"ContainerStarted","Data":"8db4e5eb58c261dc5722836620e56eee0229dfb44a0f76491b5898fc8a7a2209"} Oct 30 04:01:02.584468 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:02.584454 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:01:02.585851 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:02.585831 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:01:02.625231 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:02.625184 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" podStartSLOduration=1.13913941 podStartE2EDuration="4.625167619s" totalImagesPullingTime="3.486028209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:02.624296037 +0000 UTC m=+4823.637574142" watchObservedRunningTime="2026-10-30 04:01:02.625167619 +0000 UTC m=+4823.638445723" Oct 30 04:01:02.668291 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:02.668251 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" podStartSLOduration=1.430414401 podStartE2EDuration="4.66823845s" totalImagesPullingTime="3.237824049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:02.666799734 +0000 UTC m=+4823.680077836" watchObservedRunningTime="2026-10-30 04:01:02.66823845 +0000 UTC m=+4823.681516553" Oct 30 04:01:03.591694 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:03.591655 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" event={"ID":"d52093ca-d97a-4e66-ac82-3a52d050525f","Type":"ContainerStarted","Data":"aafc5b90548175086e4e71d442fa0fc8f9b812d6191d6d9ac2a52cb1b3193cb9"} Oct 30 04:01:03.592182 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:03.591796 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:01:03.593375 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:03.593342 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" event={"ID":"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed","Type":"ContainerStarted","Data":"de9fda749df78c14966905ecd10717c4ac9ff817243c02793ac554a4d0de4dee"} Oct 30 04:01:03.593885 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:03.593870 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:01:03.635705 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:03.635658 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" podStartSLOduration=2.243669771 podStartE2EDuration="5.635645901s" totalImagesPullingTime="3.39197613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:03.634690856 +0000 UTC m=+4824.647968959" watchObservedRunningTime="2026-10-30 04:01:03.635645901 +0000 UTC m=+4824.648923998" Oct 30 04:01:03.679343 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:03.679304 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" podStartSLOduration=2.225118608 podStartE2EDuration="5.679293592s" totalImagesPullingTime="3.454174984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:03.678938943 +0000 UTC m=+4824.692217046" watchObservedRunningTime="2026-10-30 04:01:03.679293592 +0000 UTC m=+4824.692571692" Oct 30 04:01:05.527895 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:05.527860 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" Oct 30 04:01:18.162467 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:01:18.162429 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:01:23.599349 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:23.599315 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:01:23.599787 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:23.599631 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:01:24.628537 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:24.628502 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:01:25.608952 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:25.608911 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" podUID="e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:01:35.608078 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:35.608039 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" podUID="e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:01:45.608269 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:45.608226 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" podUID="e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:01:48.164422 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:01:48.164382 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:01:55.607996 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:01:55.607957 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" podUID="e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:02:05.608239 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:05.608209 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:02:11.525049 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.525001 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:02:11.525655 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.525348 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="f76a3ac9-2cfa-493c-8a85-7c1a161419ef" containerName="otc-container" containerID="cri-o://1a2f55183bec0bf4f24fb3377ad2a74a398ad8ad0bd05dbae00217aab255ef83" gracePeriod=30 Oct 30 04:02:11.782823 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.782764 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:11.891434 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.891398 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-monitoring-tls-certs\") pod \"f76a3ac9-2cfa-493c-8a85-7c1a161419ef\" (UID: \"f76a3ac9-2cfa-493c-8a85-7c1a161419ef\") " Oct 30 04:02:11.891637 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.891473 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-kube-api-access-q7gvd\" (UniqueName: \"kubernetes.io/projected/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-kube-api-access-q7gvd\") pod \"f76a3ac9-2cfa-493c-8a85-7c1a161419ef\" (UID: \"f76a3ac9-2cfa-493c-8a85-7c1a161419ef\") " Oct 30 04:02:11.891637 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.891500 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-otc-internal\" (UniqueName: \"kubernetes.io/configmap/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-otc-internal\") pod \"f76a3ac9-2cfa-493c-8a85-7c1a161419ef\" (UID: \"f76a3ac9-2cfa-493c-8a85-7c1a161419ef\") " Oct 30 04:02:11.891961 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.891922 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-otc-internal" pod "f76a3ac9-2cfa-493c-8a85-7c1a161419ef" (UID: "f76a3ac9-2cfa-493c-8a85-7c1a161419ef"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:02:11.893672 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.893644 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-monitoring-tls-certs" pod "f76a3ac9-2cfa-493c-8a85-7c1a161419ef" (UID: "f76a3ac9-2cfa-493c-8a85-7c1a161419ef"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:02:11.893811 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.893693 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-kube-api-access-q7gvd" pod "f76a3ac9-2cfa-493c-8a85-7c1a161419ef" (UID: "f76a3ac9-2cfa-493c-8a85-7c1a161419ef"). InnerVolumeSpecName "kube-api-access-q7gvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:02:11.937024 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.936995 4925 generic.go:356] "Generic (PLEG): container finished" podID="f76a3ac9-2cfa-493c-8a85-7c1a161419ef" containerID="1a2f55183bec0bf4f24fb3377ad2a74a398ad8ad0bd05dbae00217aab255ef83" exitCode=0 Oct 30 04:02:11.937180 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.937050 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:11.937180 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.937084 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"f76a3ac9-2cfa-493c-8a85-7c1a161419ef","Type":"ContainerDied","Data":"1a2f55183bec0bf4f24fb3377ad2a74a398ad8ad0bd05dbae00217aab255ef83"} Oct 30 04:02:11.937180 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.937160 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"f76a3ac9-2cfa-493c-8a85-7c1a161419ef","Type":"ContainerDied","Data":"085f4ed9459e67f1127aebf59cb7f7320317891ae86ce5e5f2fdc01acdc60fdc"} Oct 30 04:02:11.937336 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.937186 4925 scope.go:122] "RemoveContainer" containerID="1a2f55183bec0bf4f24fb3377ad2a74a398ad8ad0bd05dbae00217aab255ef83" Oct 30 04:02:11.946730 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.946709 4925 scope.go:122] "RemoveContainer" containerID="1a2f55183bec0bf4f24fb3377ad2a74a398ad8ad0bd05dbae00217aab255ef83" Oct 30 04:02:11.947033 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:02:11.947010 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a2f55183bec0bf4f24fb3377ad2a74a398ad8ad0bd05dbae00217aab255ef83\": container with ID starting with 1a2f55183bec0bf4f24fb3377ad2a74a398ad8ad0bd05dbae00217aab255ef83 not found: ID does not exist" containerID="1a2f55183bec0bf4f24fb3377ad2a74a398ad8ad0bd05dbae00217aab255ef83" Oct 30 04:02:11.947085 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.947044 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a2f55183bec0bf4f24fb3377ad2a74a398ad8ad0bd05dbae00217aab255ef83"} err="failed to get container status \"1a2f55183bec0bf4f24fb3377ad2a74a398ad8ad0bd05dbae00217aab255ef83\": rpc error: code = NotFound desc = could not find container \"1a2f55183bec0bf4f24fb3377ad2a74a398ad8ad0bd05dbae00217aab255ef83\": container with ID starting with 1a2f55183bec0bf4f24fb3377ad2a74a398ad8ad0bd05dbae00217aab255ef83 not found: ID does not exist" Oct 30 04:02:11.989488 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.989455 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:02:11.992980 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.992957 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7gvd\" (UniqueName: \"kubernetes.io/projected/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-kube-api-access-q7gvd\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:11.992980 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.992979 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:11.993144 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.992989 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f76a3ac9-2cfa-493c-8a85-7c1a161419ef-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:11.995454 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:11.995435 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:02:12.033601 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.033546 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:02:12.033926 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.033913 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f76a3ac9-2cfa-493c-8a85-7c1a161419ef" containerName="otc-container" Oct 30 04:02:12.033985 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.033929 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f76a3ac9-2cfa-493c-8a85-7c1a161419ef" containerName="otc-container" Oct 30 04:02:12.034038 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.034027 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f76a3ac9-2cfa-493c-8a85-7c1a161419ef" containerName="otc-container" Oct 30 04:02:12.038371 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.038354 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:12.056474 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.056446 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:02:12.056933 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.056909 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-90b17333\"" Oct 30 04:02:12.057033 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.056915 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-vv788\"" Oct 30 04:02:12.058886 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.058863 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:02:12.194489 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.194455 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0ca9a92d-d5d1-4476-9211-0e27e9be58df-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"0ca9a92d-d5d1-4476-9211-0e27e9be58df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:12.194489 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.194492 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca9a92d-d5d1-4476-9211-0e27e9be58df-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0ca9a92d-d5d1-4476-9211-0e27e9be58df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:12.194714 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.194513 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxhjw\" (UniqueName: \"kubernetes.io/projected/0ca9a92d-d5d1-4476-9211-0e27e9be58df-kube-api-access-vxhjw\") pod \"data-science-collector-collector-0\" (UID: \"0ca9a92d-d5d1-4476-9211-0e27e9be58df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:12.296249 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.296151 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0ca9a92d-d5d1-4476-9211-0e27e9be58df-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"0ca9a92d-d5d1-4476-9211-0e27e9be58df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:12.296249 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.296196 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca9a92d-d5d1-4476-9211-0e27e9be58df-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0ca9a92d-d5d1-4476-9211-0e27e9be58df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:12.296249 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.296224 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxhjw\" (UniqueName: \"kubernetes.io/projected/0ca9a92d-d5d1-4476-9211-0e27e9be58df-kube-api-access-vxhjw\") pod \"data-science-collector-collector-0\" (UID: \"0ca9a92d-d5d1-4476-9211-0e27e9be58df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:12.296990 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.296963 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0ca9a92d-d5d1-4476-9211-0e27e9be58df-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"0ca9a92d-d5d1-4476-9211-0e27e9be58df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:12.298867 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.298848 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca9a92d-d5d1-4476-9211-0e27e9be58df-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0ca9a92d-d5d1-4476-9211-0e27e9be58df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:12.320991 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.320967 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxhjw\" (UniqueName: \"kubernetes.io/projected/0ca9a92d-d5d1-4476-9211-0e27e9be58df-kube-api-access-vxhjw\") pod \"data-science-collector-collector-0\" (UID: \"0ca9a92d-d5d1-4476-9211-0e27e9be58df\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:12.350513 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.350492 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:12.647728 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.647703 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:02:12.650023 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:02:12.649996 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca9a92d_d5d1_4476_9211_0e27e9be58df.slice/crio-85d5030b8c27b931da7318f056790f7d3f3d2a1c39103ee743fc5c2f9d00b319 WatchSource:0}: Error finding container 85d5030b8c27b931da7318f056790f7d3f3d2a1c39103ee743fc5c2f9d00b319: Status 404 returned error can't find the container with id 85d5030b8c27b931da7318f056790f7d3f3d2a1c39103ee743fc5c2f9d00b319 Oct 30 04:02:12.652102 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.652072 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:02:12.652173 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.652125 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:02:12.945157 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.945066 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"0ca9a92d-d5d1-4476-9211-0e27e9be58df","Type":"ContainerStarted","Data":"e99000f962990002572b3c70d6229a1eb59203e2c959b28763b106a723467091"} Oct 30 04:02:12.945157 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.945128 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"0ca9a92d-d5d1-4476-9211-0e27e9be58df","Type":"ContainerStarted","Data":"85d5030b8c27b931da7318f056790f7d3f3d2a1c39103ee743fc5c2f9d00b319"} Oct 30 04:02:12.988538 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:12.988484 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=0.988470772 podStartE2EDuration="988.470772ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:02:12.987174371 +0000 UTC m=+4894.000452484" watchObservedRunningTime="2026-10-30 04:02:12.988470772 +0000 UTC m=+4894.001748874" Oct 30 04:02:13.119892 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:13.119861 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f76a3ac9-2cfa-493c-8a85-7c1a161419ef" path="/var/lib/kubelet/pods/f76a3ac9-2cfa-493c-8a85-7c1a161419ef/volumes" Oct 30 04:02:18.161160 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:02:18.161116 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:02:42.474882 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:42.474849 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n"] Oct 30 04:02:42.475385 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:42.475039 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" podUID="052f928b-5f7e-409d-b037-ed9af947a8fe" containerName="korrel8r" containerID="cri-o://c40b111ad49bb06e73b900ffa25d036457ee57ea7e0fa0cc9d2e32be698944d6" gracePeriod=30 Oct 30 04:02:42.731294 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:42.731238 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" Oct 30 04:02:42.855582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:42.855550 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052f928b-5f7e-409d-b037-ed9af947a8fe-kube-api-access-854p2\" (UniqueName: \"kubernetes.io/projected/052f928b-5f7e-409d-b037-ed9af947a8fe-kube-api-access-854p2\") pod \"052f928b-5f7e-409d-b037-ed9af947a8fe\" (UID: \"052f928b-5f7e-409d-b037-ed9af947a8fe\") " Oct 30 04:02:42.855795 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:42.855625 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/052f928b-5f7e-409d-b037-ed9af947a8fe-config\" (UniqueName: \"kubernetes.io/configmap/052f928b-5f7e-409d-b037-ed9af947a8fe-config\") pod \"052f928b-5f7e-409d-b037-ed9af947a8fe\" (UID: \"052f928b-5f7e-409d-b037-ed9af947a8fe\") " Oct 30 04:02:42.855877 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:42.855814 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/052f928b-5f7e-409d-b037-ed9af947a8fe-tls\" (UniqueName: \"kubernetes.io/secret/052f928b-5f7e-409d-b037-ed9af947a8fe-tls\") pod \"052f928b-5f7e-409d-b037-ed9af947a8fe\" (UID: \"052f928b-5f7e-409d-b037-ed9af947a8fe\") " Oct 30 04:02:42.856339 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:42.856311 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052f928b-5f7e-409d-b037-ed9af947a8fe-config" pod "052f928b-5f7e-409d-b037-ed9af947a8fe" (UID: "052f928b-5f7e-409d-b037-ed9af947a8fe"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:02:42.859551 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:42.859525 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/052f928b-5f7e-409d-b037-ed9af947a8fe-tls" pod "052f928b-5f7e-409d-b037-ed9af947a8fe" (UID: "052f928b-5f7e-409d-b037-ed9af947a8fe"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:02:42.861317 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:42.861282 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052f928b-5f7e-409d-b037-ed9af947a8fe-kube-api-access-854p2" pod "052f928b-5f7e-409d-b037-ed9af947a8fe" (UID: "052f928b-5f7e-409d-b037-ed9af947a8fe"). InnerVolumeSpecName "kube-api-access-854p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:02:42.868347 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:42.868321 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:02:42.868573 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:42.868548 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="0ca9a92d-d5d1-4476-9211-0e27e9be58df" containerName="otc-container" containerID="cri-o://e99000f962990002572b3c70d6229a1eb59203e2c959b28763b106a723467091" gracePeriod=30 Oct 30 04:02:42.957278 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:42.957247 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/052f928b-5f7e-409d-b037-ed9af947a8fe-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:42.957278 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:42.957275 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/052f928b-5f7e-409d-b037-ed9af947a8fe-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:42.957453 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:42.957284 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-854p2\" (UniqueName: \"kubernetes.io/projected/052f928b-5f7e-409d-b037-ed9af947a8fe-kube-api-access-854p2\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:43.097928 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.097894 4925 generic.go:356] "Generic (PLEG): container finished" podID="0ca9a92d-d5d1-4476-9211-0e27e9be58df" containerID="e99000f962990002572b3c70d6229a1eb59203e2c959b28763b106a723467091" exitCode=0 Oct 30 04:02:43.098081 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.097962 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"0ca9a92d-d5d1-4476-9211-0e27e9be58df","Type":"ContainerDied","Data":"e99000f962990002572b3c70d6229a1eb59203e2c959b28763b106a723467091"} Oct 30 04:02:43.099336 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.099286 4925 generic.go:356] "Generic (PLEG): container finished" podID="052f928b-5f7e-409d-b037-ed9af947a8fe" containerID="c40b111ad49bb06e73b900ffa25d036457ee57ea7e0fa0cc9d2e32be698944d6" exitCode=0 Oct 30 04:02:43.099474 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.099374 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" event={"ID":"052f928b-5f7e-409d-b037-ed9af947a8fe","Type":"ContainerDied","Data":"c40b111ad49bb06e73b900ffa25d036457ee57ea7e0fa0cc9d2e32be698944d6"} Oct 30 04:02:43.099474 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.099394 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" Oct 30 04:02:43.099474 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.099429 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n" event={"ID":"052f928b-5f7e-409d-b037-ed9af947a8fe","Type":"ContainerDied","Data":"1ba020e8037b8e26edcab9fdc1f3f15238baaacef9f2d6417f5e1a42b89f5cc1"} Oct 30 04:02:43.099474 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.099460 4925 scope.go:122] "RemoveContainer" containerID="c40b111ad49bb06e73b900ffa25d036457ee57ea7e0fa0cc9d2e32be698944d6" Oct 30 04:02:43.108434 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.108412 4925 scope.go:122] "RemoveContainer" containerID="c40b111ad49bb06e73b900ffa25d036457ee57ea7e0fa0cc9d2e32be698944d6" Oct 30 04:02:43.108689 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:02:43.108672 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c40b111ad49bb06e73b900ffa25d036457ee57ea7e0fa0cc9d2e32be698944d6\": container with ID starting with c40b111ad49bb06e73b900ffa25d036457ee57ea7e0fa0cc9d2e32be698944d6 not found: ID does not exist" containerID="c40b111ad49bb06e73b900ffa25d036457ee57ea7e0fa0cc9d2e32be698944d6" Oct 30 04:02:43.108780 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.108697 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c40b111ad49bb06e73b900ffa25d036457ee57ea7e0fa0cc9d2e32be698944d6"} err="failed to get container status \"c40b111ad49bb06e73b900ffa25d036457ee57ea7e0fa0cc9d2e32be698944d6\": rpc error: code = NotFound desc = could not find container \"c40b111ad49bb06e73b900ffa25d036457ee57ea7e0fa0cc9d2e32be698944d6\": container with ID starting with c40b111ad49bb06e73b900ffa25d036457ee57ea7e0fa0cc9d2e32be698944d6 not found: ID does not exist" Oct 30 04:02:43.126577 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.126560 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:43.152696 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.152670 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n"] Oct 30 04:02:43.156254 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.156232 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-5np8n"] Oct 30 04:02:43.260029 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.259951 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ca9a92d-d5d1-4476-9211-0e27e9be58df-kube-api-access-vxhjw\" (UniqueName: \"kubernetes.io/projected/0ca9a92d-d5d1-4476-9211-0e27e9be58df-kube-api-access-vxhjw\") pod \"0ca9a92d-d5d1-4476-9211-0e27e9be58df\" (UID: \"0ca9a92d-d5d1-4476-9211-0e27e9be58df\") " Oct 30 04:02:43.260029 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.260004 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ca9a92d-d5d1-4476-9211-0e27e9be58df-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca9a92d-d5d1-4476-9211-0e27e9be58df-monitoring-tls-certs\") pod \"0ca9a92d-d5d1-4476-9211-0e27e9be58df\" (UID: \"0ca9a92d-d5d1-4476-9211-0e27e9be58df\") " Oct 30 04:02:43.260253 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.260111 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ca9a92d-d5d1-4476-9211-0e27e9be58df-otc-internal\" (UniqueName: \"kubernetes.io/configmap/0ca9a92d-d5d1-4476-9211-0e27e9be58df-otc-internal\") pod \"0ca9a92d-d5d1-4476-9211-0e27e9be58df\" (UID: \"0ca9a92d-d5d1-4476-9211-0e27e9be58df\") " Oct 30 04:02:43.260413 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.260394 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ca9a92d-d5d1-4476-9211-0e27e9be58df-otc-internal" pod "0ca9a92d-d5d1-4476-9211-0e27e9be58df" (UID: "0ca9a92d-d5d1-4476-9211-0e27e9be58df"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:02:43.262177 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.262145 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ca9a92d-d5d1-4476-9211-0e27e9be58df-monitoring-tls-certs" pod "0ca9a92d-d5d1-4476-9211-0e27e9be58df" (UID: "0ca9a92d-d5d1-4476-9211-0e27e9be58df"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:02:43.262288 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.262217 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca9a92d-d5d1-4476-9211-0e27e9be58df-kube-api-access-vxhjw" pod "0ca9a92d-d5d1-4476-9211-0e27e9be58df" (UID: "0ca9a92d-d5d1-4476-9211-0e27e9be58df"). InnerVolumeSpecName "kube-api-access-vxhjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:02:43.360839 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.360805 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0ca9a92d-d5d1-4476-9211-0e27e9be58df-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:43.360839 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.360832 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxhjw\" (UniqueName: \"kubernetes.io/projected/0ca9a92d-d5d1-4476-9211-0e27e9be58df-kube-api-access-vxhjw\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:43.360839 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.360844 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca9a92d-d5d1-4476-9211-0e27e9be58df-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:43.508158 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.508123 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp"] Oct 30 04:02:43.508603 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.508338 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" podUID="2437ead6-6d92-4324-9cca-a355c73a6646" containerName="tempo" containerID="cri-o://efaa7ceb7d5929deffe440f046904a07c52d2d8c05b05aa44c6f35da4d49c0db" gracePeriod=30 Oct 30 04:02:43.510836 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.510728 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4"] Oct 30 04:02:43.511024 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.510998 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" podUID="d4fec03a-ad3e-4bfc-9910-41485267ff19" containerName="tempo-gateway" containerID="cri-o://02f6972dc85b69a4abb45211ab0187e6dcf4e0011025ce0e0c7e0e7ecb5ee96e" gracePeriod=30 Oct 30 04:02:43.511119 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.511032 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" podUID="d4fec03a-ad3e-4bfc-9910-41485267ff19" containerName="tempo-gateway-opa" containerID="cri-o://a5660ca25a190b08c34e24c02f2f465cb1e00302472395006c77a8c57556f54d" gracePeriod=30 Oct 30 04:02:43.512871 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.512843 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4"] Oct 30 04:02:43.513048 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.513024 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" podUID="d52093ca-d97a-4e66-ac82-3a52d050525f" containerName="tempo" containerID="cri-o://aafc5b90548175086e4e71d442fa0fc8f9b812d6191d6d9ac2a52cb1b3193cb9" gracePeriod=30 Oct 30 04:02:43.516506 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.516480 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx"] Oct 30 04:02:43.516626 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.516610 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" podUID="d0480bc1-0f02-40a0-9ff9-dd717975b349" containerName="tempo" containerID="cri-o://8db4e5eb58c261dc5722836620e56eee0229dfb44a0f76491b5898fc8a7a2209" gracePeriod=30 Oct 30 04:02:43.520321 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.520301 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq"] Oct 30 04:02:43.520499 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.520481 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" podUID="e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed" containerName="tempo" containerID="cri-o://de9fda749df78c14966905ecd10717c4ac9ff817243c02793ac554a4d0de4dee" gracePeriod=30 Oct 30 04:02:43.599139 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.599098 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" podUID="2437ead6-6d92-4324-9cca-a355c73a6646" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:02:43.599139 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.599122 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" podUID="d0480bc1-0f02-40a0-9ff9-dd717975b349" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:02:43.956737 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:43.956714 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:02:44.068833 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.068798 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-tmp-storage\") pod \"2437ead6-6d92-4324-9cca-a355c73a6646\" (UID: \"2437ead6-6d92-4324-9cca-a355c73a6646\") " Oct 30 04:02:44.069008 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.068847 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-data-science-tempostack-distributor-mtls\") pod \"2437ead6-6d92-4324-9cca-a355c73a6646\" (UID: \"2437ead6-6d92-4324-9cca-a355c73a6646\") " Oct 30 04:02:44.069008 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.068882 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-conf\") pod \"2437ead6-6d92-4324-9cca-a355c73a6646\" (UID: \"2437ead6-6d92-4324-9cca-a355c73a6646\") " Oct 30 04:02:44.069008 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.068952 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2437ead6-6d92-4324-9cca-a355c73a6646-kube-api-access-qmcdr\" (UniqueName: \"kubernetes.io/projected/2437ead6-6d92-4324-9cca-a355c73a6646-kube-api-access-qmcdr\") pod \"2437ead6-6d92-4324-9cca-a355c73a6646\" (UID: \"2437ead6-6d92-4324-9cca-a355c73a6646\") " Oct 30 04:02:44.069008 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.068982 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-data-science-tempostack-ca-bundle\") pod \"2437ead6-6d92-4324-9cca-a355c73a6646\" (UID: \"2437ead6-6d92-4324-9cca-a355c73a6646\") " Oct 30 04:02:44.069243 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.069125 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-tmp-storage" pod "2437ead6-6d92-4324-9cca-a355c73a6646" (UID: "2437ead6-6d92-4324-9cca-a355c73a6646"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:02:44.069390 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.069369 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-data-science-tempostack-ca-bundle" pod "2437ead6-6d92-4324-9cca-a355c73a6646" (UID: "2437ead6-6d92-4324-9cca-a355c73a6646"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:02:44.069454 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.069418 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-conf" pod "2437ead6-6d92-4324-9cca-a355c73a6646" (UID: "2437ead6-6d92-4324-9cca-a355c73a6646"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:02:44.071152 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.071129 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2437ead6-6d92-4324-9cca-a355c73a6646-kube-api-access-qmcdr" pod "2437ead6-6d92-4324-9cca-a355c73a6646" (UID: "2437ead6-6d92-4324-9cca-a355c73a6646"). InnerVolumeSpecName "kube-api-access-qmcdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:02:44.071229 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.071131 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-data-science-tempostack-distributor-mtls" pod "2437ead6-6d92-4324-9cca-a355c73a6646" (UID: "2437ead6-6d92-4324-9cca-a355c73a6646"). InnerVolumeSpecName "tempo-data-science-tempostack-distributor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:02:44.106564 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.106491 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:44.106564 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.106522 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"0ca9a92d-d5d1-4476-9211-0e27e9be58df","Type":"ContainerDied","Data":"85d5030b8c27b931da7318f056790f7d3f3d2a1c39103ee743fc5c2f9d00b319"} Oct 30 04:02:44.106735 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.106596 4925 scope.go:122] "RemoveContainer" containerID="e99000f962990002572b3c70d6229a1eb59203e2c959b28763b106a723467091" Oct 30 04:02:44.118128 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.118096 4925 generic.go:356] "Generic (PLEG): container finished" podID="d4fec03a-ad3e-4bfc-9910-41485267ff19" containerID="a5660ca25a190b08c34e24c02f2f465cb1e00302472395006c77a8c57556f54d" exitCode=0 Oct 30 04:02:44.118235 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.118131 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" event={"ID":"d4fec03a-ad3e-4bfc-9910-41485267ff19","Type":"ContainerDied","Data":"a5660ca25a190b08c34e24c02f2f465cb1e00302472395006c77a8c57556f54d"} Oct 30 04:02:44.119588 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.119567 4925 generic.go:356] "Generic (PLEG): container finished" podID="2437ead6-6d92-4324-9cca-a355c73a6646" containerID="efaa7ceb7d5929deffe440f046904a07c52d2d8c05b05aa44c6f35da4d49c0db" exitCode=0 Oct 30 04:02:44.119685 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.119612 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" event={"ID":"2437ead6-6d92-4324-9cca-a355c73a6646","Type":"ContainerDied","Data":"efaa7ceb7d5929deffe440f046904a07c52d2d8c05b05aa44c6f35da4d49c0db"} Oct 30 04:02:44.119685 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.119617 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" Oct 30 04:02:44.119685 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.119638 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp" event={"ID":"2437ead6-6d92-4324-9cca-a355c73a6646","Type":"ContainerDied","Data":"c9408d72eb03cb3494eac70d839b89f482c2bedbbaa00a08f6976555a39d2016"} Oct 30 04:02:44.123960 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.123938 4925 scope.go:122] "RemoveContainer" containerID="efaa7ceb7d5929deffe440f046904a07c52d2d8c05b05aa44c6f35da4d49c0db" Oct 30 04:02:44.132492 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.132458 4925 scope.go:122] "RemoveContainer" containerID="efaa7ceb7d5929deffe440f046904a07c52d2d8c05b05aa44c6f35da4d49c0db" Oct 30 04:02:44.132758 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:02:44.132724 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efaa7ceb7d5929deffe440f046904a07c52d2d8c05b05aa44c6f35da4d49c0db\": container with ID starting with efaa7ceb7d5929deffe440f046904a07c52d2d8c05b05aa44c6f35da4d49c0db not found: ID does not exist" containerID="efaa7ceb7d5929deffe440f046904a07c52d2d8c05b05aa44c6f35da4d49c0db" Oct 30 04:02:44.132831 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.132769 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efaa7ceb7d5929deffe440f046904a07c52d2d8c05b05aa44c6f35da4d49c0db"} err="failed to get container status \"efaa7ceb7d5929deffe440f046904a07c52d2d8c05b05aa44c6f35da4d49c0db\": rpc error: code = NotFound desc = could not find container \"efaa7ceb7d5929deffe440f046904a07c52d2d8c05b05aa44c6f35da4d49c0db\": container with ID starting with efaa7ceb7d5929deffe440f046904a07c52d2d8c05b05aa44c6f35da4d49c0db not found: ID does not exist" Oct 30 04:02:44.159864 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.159832 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:02:44.165234 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.165212 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:02:44.169957 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.169937 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmcdr\" (UniqueName: \"kubernetes.io/projected/2437ead6-6d92-4324-9cca-a355c73a6646-kube-api-access-qmcdr\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:44.170023 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.169960 4925 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:44.170023 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.169971 4925 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-tmp-storage\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:44.170023 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.169981 4925 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-data-science-tempostack-distributor-mtls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:44.170023 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.169990 4925 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2437ead6-6d92-4324-9cca-a355c73a6646-tempo-conf\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:44.214906 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.214879 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp"] Oct 30 04:02:44.218732 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.218708 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5bc5dfb56b-hlmsp"] Oct 30 04:02:44.630714 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.630674 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" podUID="d52093ca-d97a-4e66-ac82-3a52d050525f" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:02:44.759077 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.759053 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:02:44.875589 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.875556 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-rbac\" (UniqueName: \"kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-rbac\") pod \"d4fec03a-ad3e-4bfc-9910-41485267ff19\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " Oct 30 04:02:44.875589 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.875591 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-cabundle\" (UniqueName: \"kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-cabundle\") pod \"d4fec03a-ad3e-4bfc-9910-41485267ff19\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " Oct 30 04:02:44.875854 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.875608 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-serving-certs\" (UniqueName: \"kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-serving-certs\") pod \"d4fec03a-ad3e-4bfc-9910-41485267ff19\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " Oct 30 04:02:44.875854 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.875643 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-tempo-data-science-tempostack-gateway-mtls\") pod \"d4fec03a-ad3e-4bfc-9910-41485267ff19\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " Oct 30 04:02:44.875854 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.875669 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-tenant\" (UniqueName: \"kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-tenant\") pod \"d4fec03a-ad3e-4bfc-9910-41485267ff19\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " Oct 30 04:02:44.875854 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.875736 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4fec03a-ad3e-4bfc-9910-41485267ff19-kube-api-access-ghfwd\" (UniqueName: \"kubernetes.io/projected/d4fec03a-ad3e-4bfc-9910-41485267ff19-kube-api-access-ghfwd\") pod \"d4fec03a-ad3e-4bfc-9910-41485267ff19\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " Oct 30 04:02:44.875854 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.875795 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-tempo-data-science-tempostack-ca-bundle\") pod \"d4fec03a-ad3e-4bfc-9910-41485267ff19\" (UID: \"d4fec03a-ad3e-4bfc-9910-41485267ff19\") " Oct 30 04:02:44.876111 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.876032 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-rbac" pod "d4fec03a-ad3e-4bfc-9910-41485267ff19" (UID: "d4fec03a-ad3e-4bfc-9910-41485267ff19"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:02:44.876227 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.876202 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-cabundle" pod "d4fec03a-ad3e-4bfc-9910-41485267ff19" (UID: "d4fec03a-ad3e-4bfc-9910-41485267ff19"). InnerVolumeSpecName "cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:02:44.876288 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.876254 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-tempo-data-science-tempostack-ca-bundle" pod "d4fec03a-ad3e-4bfc-9910-41485267ff19" (UID: "d4fec03a-ad3e-4bfc-9910-41485267ff19"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:02:44.877856 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.877824 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-serving-certs" pod "d4fec03a-ad3e-4bfc-9910-41485267ff19" (UID: "d4fec03a-ad3e-4bfc-9910-41485267ff19"). InnerVolumeSpecName "serving-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:02:44.878287 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.878263 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-tempo-data-science-tempostack-gateway-mtls" pod "d4fec03a-ad3e-4bfc-9910-41485267ff19" (UID: "d4fec03a-ad3e-4bfc-9910-41485267ff19"). InnerVolumeSpecName "tempo-data-science-tempostack-gateway-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:02:44.878406 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.878300 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fec03a-ad3e-4bfc-9910-41485267ff19-kube-api-access-ghfwd" pod "d4fec03a-ad3e-4bfc-9910-41485267ff19" (UID: "d4fec03a-ad3e-4bfc-9910-41485267ff19"). InnerVolumeSpecName "kube-api-access-ghfwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:02:44.881997 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.881977 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-tenant" pod "d4fec03a-ad3e-4bfc-9910-41485267ff19" (UID: "d4fec03a-ad3e-4bfc-9910-41485267ff19"). InnerVolumeSpecName "tenant". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:02:44.976828 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.976800 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghfwd\" (UniqueName: \"kubernetes.io/projected/d4fec03a-ad3e-4bfc-9910-41485267ff19-kube-api-access-ghfwd\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:44.976828 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.976826 4925 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:44.976998 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.976837 4925 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-rbac\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:44.976998 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.976846 4925 reconciler_common.go:299] "Volume detached for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/d4fec03a-ad3e-4bfc-9910-41485267ff19-cabundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:44.976998 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.976854 4925 reconciler_common.go:299] "Volume detached for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-serving-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:44.976998 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.976863 4925 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-tempo-data-science-tempostack-gateway-mtls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:44.976998 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:44.976870 4925 reconciler_common.go:299] "Volume detached for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/d4fec03a-ad3e-4bfc-9910-41485267ff19-tenant\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:45.119028 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:45.118986 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="052f928b-5f7e-409d-b037-ed9af947a8fe" path="/var/lib/kubelet/pods/052f928b-5f7e-409d-b037-ed9af947a8fe/volumes" Oct 30 04:02:45.119472 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:45.119450 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ca9a92d-d5d1-4476-9211-0e27e9be58df" path="/var/lib/kubelet/pods/0ca9a92d-d5d1-4476-9211-0e27e9be58df/volumes" Oct 30 04:02:45.119932 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:45.119910 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2437ead6-6d92-4324-9cca-a355c73a6646" path="/var/lib/kubelet/pods/2437ead6-6d92-4324-9cca-a355c73a6646/volumes" Oct 30 04:02:45.131151 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:45.131129 4925 generic.go:356] "Generic (PLEG): container finished" podID="d4fec03a-ad3e-4bfc-9910-41485267ff19" containerID="02f6972dc85b69a4abb45211ab0187e6dcf4e0011025ce0e0c7e0e7ecb5ee96e" exitCode=0 Oct 30 04:02:45.131233 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:45.131182 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" Oct 30 04:02:45.131233 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:45.131203 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" event={"ID":"d4fec03a-ad3e-4bfc-9910-41485267ff19","Type":"ContainerDied","Data":"02f6972dc85b69a4abb45211ab0187e6dcf4e0011025ce0e0c7e0e7ecb5ee96e"} Oct 30 04:02:45.131407 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:45.131245 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4" event={"ID":"d4fec03a-ad3e-4bfc-9910-41485267ff19","Type":"ContainerDied","Data":"d36beecbca7f6145b5113de58d6027179a6eb04b977964efc60b00a91f95fca0"} Oct 30 04:02:45.131407 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:45.131261 4925 scope.go:122] "RemoveContainer" containerID="a5660ca25a190b08c34e24c02f2f465cb1e00302472395006c77a8c57556f54d" Oct 30 04:02:45.139699 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:45.139682 4925 scope.go:122] "RemoveContainer" containerID="02f6972dc85b69a4abb45211ab0187e6dcf4e0011025ce0e0c7e0e7ecb5ee96e" Oct 30 04:02:45.148493 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:45.148469 4925 scope.go:122] "RemoveContainer" containerID="a5660ca25a190b08c34e24c02f2f465cb1e00302472395006c77a8c57556f54d" Oct 30 04:02:45.148745 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:02:45.148724 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5660ca25a190b08c34e24c02f2f465cb1e00302472395006c77a8c57556f54d\": container with ID starting with a5660ca25a190b08c34e24c02f2f465cb1e00302472395006c77a8c57556f54d not found: ID does not exist" containerID="a5660ca25a190b08c34e24c02f2f465cb1e00302472395006c77a8c57556f54d" Oct 30 04:02:45.148870 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:45.148851 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5660ca25a190b08c34e24c02f2f465cb1e00302472395006c77a8c57556f54d"} err="failed to get container status \"a5660ca25a190b08c34e24c02f2f465cb1e00302472395006c77a8c57556f54d\": rpc error: code = NotFound desc = could not find container \"a5660ca25a190b08c34e24c02f2f465cb1e00302472395006c77a8c57556f54d\": container with ID starting with a5660ca25a190b08c34e24c02f2f465cb1e00302472395006c77a8c57556f54d not found: ID does not exist" Oct 30 04:02:45.148921 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:45.148872 4925 scope.go:122] "RemoveContainer" containerID="02f6972dc85b69a4abb45211ab0187e6dcf4e0011025ce0e0c7e0e7ecb5ee96e" Oct 30 04:02:45.149147 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:02:45.149125 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02f6972dc85b69a4abb45211ab0187e6dcf4e0011025ce0e0c7e0e7ecb5ee96e\": container with ID starting with 02f6972dc85b69a4abb45211ab0187e6dcf4e0011025ce0e0c7e0e7ecb5ee96e not found: ID does not exist" containerID="02f6972dc85b69a4abb45211ab0187e6dcf4e0011025ce0e0c7e0e7ecb5ee96e" Oct 30 04:02:45.149226 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:45.149161 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f6972dc85b69a4abb45211ab0187e6dcf4e0011025ce0e0c7e0e7ecb5ee96e"} err="failed to get container status \"02f6972dc85b69a4abb45211ab0187e6dcf4e0011025ce0e0c7e0e7ecb5ee96e\": rpc error: code = NotFound desc = could not find container \"02f6972dc85b69a4abb45211ab0187e6dcf4e0011025ce0e0c7e0e7ecb5ee96e\": container with ID starting with 02f6972dc85b69a4abb45211ab0187e6dcf4e0011025ce0e0c7e0e7ecb5ee96e not found: ID does not exist" Oct 30 04:02:45.184762 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:45.184729 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4"] Oct 30 04:02:45.197302 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:45.197279 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-746f7f6687-rsdl4"] Oct 30 04:02:45.607775 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:45.607712 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" podUID="e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:02:46.465940 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:46.465913 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:02:46.591771 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:46.591724 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-storage-gcs-key\") pod \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " Oct 30 04:02:46.591949 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:46.591784 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-data-science-tempostack-ca-bundle\") pod \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " Oct 30 04:02:46.591949 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:46.591820 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-tmp-storage\") pod \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " Oct 30 04:02:46.591949 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:46.591924 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-kube-api-access-wggpw\" (UniqueName: \"kubernetes.io/projected/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-kube-api-access-wggpw\") pod \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " Oct 30 04:02:46.592152 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:46.591997 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-conf\") pod \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " Oct 30 04:02:46.592152 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:46.592069 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-data-science-tempostack-compactor-mtls\") pod \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\" (UID: \"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed\") " Oct 30 04:02:46.592262 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:46.592203 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-tmp-storage" pod "e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed" (UID: "e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:02:46.592330 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:46.592305 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-data-science-tempostack-ca-bundle" pod "e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed" (UID: "e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:02:46.592393 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:46.592384 4925 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:46.592456 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:46.592396 4925 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-tmp-storage\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:46.592456 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:46.592399 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-conf" pod "e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed" (UID: "e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:02:46.594125 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:46.594099 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-storage-gcs-key" pod "e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed" (UID: "e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:02:46.594261 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:46.594213 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-data-science-tempostack-compactor-mtls" pod "e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed" (UID: "e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed"). InnerVolumeSpecName "tempo-data-science-tempostack-compactor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:02:46.594328 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:46.594291 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-kube-api-access-wggpw" pod "e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed" (UID: "e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed"). InnerVolumeSpecName "kube-api-access-wggpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:02:46.692811 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:46.692779 4925 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-storage-gcs-key\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:46.692811 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:46.692806 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wggpw\" (UniqueName: \"kubernetes.io/projected/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-kube-api-access-wggpw\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:46.692811 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:46.692815 4925 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-conf\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:46.693021 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:46.692825 4925 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed-tempo-data-science-tempostack-compactor-mtls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:02:47.119190 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.119155 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4fec03a-ad3e-4bfc-9910-41485267ff19" path="/var/lib/kubelet/pods/d4fec03a-ad3e-4bfc-9910-41485267ff19/volumes" Oct 30 04:02:47.144842 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.144809 4925 generic.go:356] "Generic (PLEG): container finished" podID="e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed" containerID="de9fda749df78c14966905ecd10717c4ac9ff817243c02793ac554a4d0de4dee" exitCode=0 Oct 30 04:02:47.144965 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.144858 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" Oct 30 04:02:47.144965 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.144889 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" event={"ID":"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed","Type":"ContainerDied","Data":"de9fda749df78c14966905ecd10717c4ac9ff817243c02793ac554a4d0de4dee"} Oct 30 04:02:47.144965 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.144941 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq" event={"ID":"e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed","Type":"ContainerDied","Data":"05eb3686933a4067bbaa8898ec8f442002dfa250b33732d0f85914ec6cd79943"} Oct 30 04:02:47.145080 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.144967 4925 scope.go:122] "RemoveContainer" containerID="de9fda749df78c14966905ecd10717c4ac9ff817243c02793ac554a4d0de4dee" Oct 30 04:02:47.153689 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.153669 4925 scope.go:122] "RemoveContainer" containerID="de9fda749df78c14966905ecd10717c4ac9ff817243c02793ac554a4d0de4dee" Oct 30 04:02:47.153975 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:02:47.153957 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de9fda749df78c14966905ecd10717c4ac9ff817243c02793ac554a4d0de4dee\": container with ID starting with de9fda749df78c14966905ecd10717c4ac9ff817243c02793ac554a4d0de4dee not found: ID does not exist" containerID="de9fda749df78c14966905ecd10717c4ac9ff817243c02793ac554a4d0de4dee" Oct 30 04:02:47.154024 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.153984 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de9fda749df78c14966905ecd10717c4ac9ff817243c02793ac554a4d0de4dee"} err="failed to get container status \"de9fda749df78c14966905ecd10717c4ac9ff817243c02793ac554a4d0de4dee\": rpc error: code = NotFound desc = could not find container \"de9fda749df78c14966905ecd10717c4ac9ff817243c02793ac554a4d0de4dee\": container with ID starting with de9fda749df78c14966905ecd10717c4ac9ff817243c02793ac554a4d0de4dee not found: ID does not exist" Oct 30 04:02:47.197219 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.197191 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq"] Oct 30 04:02:47.209889 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.209862 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-78c9757bcb-p2gkq"] Oct 30 04:02:47.500735 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.500658 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-kkt7w"] Oct 30 04:02:47.501123 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.501015 4925 cpu_manager.go:414] "containerMap: removing container" podUID="052f928b-5f7e-409d-b037-ed9af947a8fe" containerName="korrel8r" Oct 30 04:02:47.501123 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.501026 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052f928b-5f7e-409d-b037-ed9af947a8fe" containerName="korrel8r" Oct 30 04:02:47.501123 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.501040 4925 cpu_manager.go:414] "containerMap: removing container" podUID="2437ead6-6d92-4324-9cca-a355c73a6646" containerName="tempo" Oct 30 04:02:47.501123 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.501047 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2437ead6-6d92-4324-9cca-a355c73a6646" containerName="tempo" Oct 30 04:02:47.501123 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.501055 4925 cpu_manager.go:414] "containerMap: removing container" podUID="e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed" containerName="tempo" Oct 30 04:02:47.501123 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.501064 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed" containerName="tempo" Oct 30 04:02:47.501123 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.501087 4925 cpu_manager.go:414] "containerMap: removing container" podUID="d4fec03a-ad3e-4bfc-9910-41485267ff19" containerName="tempo-gateway-opa" Oct 30 04:02:47.501123 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.501091 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4fec03a-ad3e-4bfc-9910-41485267ff19" containerName="tempo-gateway-opa" Oct 30 04:02:47.501123 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.501102 4925 cpu_manager.go:414] "containerMap: removing container" podUID="d4fec03a-ad3e-4bfc-9910-41485267ff19" containerName="tempo-gateway" Oct 30 04:02:47.501123 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.501106 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4fec03a-ad3e-4bfc-9910-41485267ff19" containerName="tempo-gateway" Oct 30 04:02:47.501123 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.501119 4925 cpu_manager.go:414] "containerMap: removing container" podUID="0ca9a92d-d5d1-4476-9211-0e27e9be58df" containerName="otc-container" Oct 30 04:02:47.501123 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.501125 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ca9a92d-d5d1-4476-9211-0e27e9be58df" containerName="otc-container" Oct 30 04:02:47.501474 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.501181 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4fec03a-ad3e-4bfc-9910-41485267ff19" containerName="tempo-gateway-opa" Oct 30 04:02:47.501474 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.501188 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4fec03a-ad3e-4bfc-9910-41485267ff19" containerName="tempo-gateway" Oct 30 04:02:47.501474 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.501193 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ca9a92d-d5d1-4476-9211-0e27e9be58df" containerName="otc-container" Oct 30 04:02:47.501474 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.501202 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="052f928b-5f7e-409d-b037-ed9af947a8fe" containerName="korrel8r" Oct 30 04:02:47.501474 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.501208 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2437ead6-6d92-4324-9cca-a355c73a6646" containerName="tempo" Oct 30 04:02:47.501474 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.501213 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed" containerName="tempo" Oct 30 04:02:47.505710 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.505687 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" Oct 30 04:02:47.524609 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.524590 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-dgkfk\"" Oct 30 04:02:47.524729 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.524673 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 04:02:47.524833 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.524739 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 04:02:47.526457 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.526437 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-kkt7w"] Oct 30 04:02:47.600344 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.600297 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfzgm\" (UniqueName: \"kubernetes.io/projected/eda907d6-ab62-4242-8498-41c56f845eaf-kube-api-access-bfzgm\") pod \"korrel8r-6644769df5-kkt7w\" (UID: \"eda907d6-ab62-4242-8498-41c56f845eaf\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" Oct 30 04:02:47.600498 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.600378 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eda907d6-ab62-4242-8498-41c56f845eaf-config\") pod \"korrel8r-6644769df5-kkt7w\" (UID: \"eda907d6-ab62-4242-8498-41c56f845eaf\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" Oct 30 04:02:47.600498 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.600408 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/eda907d6-ab62-4242-8498-41c56f845eaf-tls\") pod \"korrel8r-6644769df5-kkt7w\" (UID: \"eda907d6-ab62-4242-8498-41c56f845eaf\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" Oct 30 04:02:47.701411 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.701376 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfzgm\" (UniqueName: \"kubernetes.io/projected/eda907d6-ab62-4242-8498-41c56f845eaf-kube-api-access-bfzgm\") pod \"korrel8r-6644769df5-kkt7w\" (UID: \"eda907d6-ab62-4242-8498-41c56f845eaf\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" Oct 30 04:02:47.701411 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.701413 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eda907d6-ab62-4242-8498-41c56f845eaf-config\") pod \"korrel8r-6644769df5-kkt7w\" (UID: \"eda907d6-ab62-4242-8498-41c56f845eaf\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" Oct 30 04:02:47.701605 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.701438 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/eda907d6-ab62-4242-8498-41c56f845eaf-tls\") pod \"korrel8r-6644769df5-kkt7w\" (UID: \"eda907d6-ab62-4242-8498-41c56f845eaf\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" Oct 30 04:02:47.702229 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.702207 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eda907d6-ab62-4242-8498-41c56f845eaf-config\") pod \"korrel8r-6644769df5-kkt7w\" (UID: \"eda907d6-ab62-4242-8498-41c56f845eaf\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" Oct 30 04:02:47.703882 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.703864 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/eda907d6-ab62-4242-8498-41c56f845eaf-tls\") pod \"korrel8r-6644769df5-kkt7w\" (UID: \"eda907d6-ab62-4242-8498-41c56f845eaf\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" Oct 30 04:02:47.725194 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.725172 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfzgm\" (UniqueName: \"kubernetes.io/projected/eda907d6-ab62-4242-8498-41c56f845eaf-kube-api-access-bfzgm\") pod \"korrel8r-6644769df5-kkt7w\" (UID: \"eda907d6-ab62-4242-8498-41c56f845eaf\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" Oct 30 04:02:47.818309 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:47.818243 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" Oct 30 04:02:48.123350 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:48.123326 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-kkt7w"] Oct 30 04:02:48.133796 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:48.133331 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:02:48.138675 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:48.138652 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:48.161603 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:48.160796 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-645168e1\"" Oct 30 04:02:48.161603 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:48.161526 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:02:48.166880 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:48.166835 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" event={"ID":"eda907d6-ab62-4242-8498-41c56f845eaf","Type":"ContainerStarted","Data":"9d2407da920b5f68e677f325376c47f6c3a42cd307cbea2cc7da8c16b0fdb223"} Oct 30 04:02:48.174165 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:02:48.174132 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:02:48.177345 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:48.177317 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:02:48.206547 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:48.206514 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbn4h\" (UniqueName: \"kubernetes.io/projected/4d989035-dcf0-4a72-9c1e-ab7161858f53-kube-api-access-bbn4h\") pod \"data-science-collector-collector-0\" (UID: \"4d989035-dcf0-4a72-9c1e-ab7161858f53\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:48.206682 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:48.206567 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4d989035-dcf0-4a72-9c1e-ab7161858f53-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4d989035-dcf0-4a72-9c1e-ab7161858f53\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:48.206682 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:48.206598 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d989035-dcf0-4a72-9c1e-ab7161858f53-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4d989035-dcf0-4a72-9c1e-ab7161858f53\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:48.307276 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:48.307237 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbn4h\" (UniqueName: \"kubernetes.io/projected/4d989035-dcf0-4a72-9c1e-ab7161858f53-kube-api-access-bbn4h\") pod \"data-science-collector-collector-0\" (UID: \"4d989035-dcf0-4a72-9c1e-ab7161858f53\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:48.307276 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:48.307277 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4d989035-dcf0-4a72-9c1e-ab7161858f53-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4d989035-dcf0-4a72-9c1e-ab7161858f53\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:48.307514 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:48.307296 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d989035-dcf0-4a72-9c1e-ab7161858f53-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4d989035-dcf0-4a72-9c1e-ab7161858f53\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:48.308064 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:48.308037 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4d989035-dcf0-4a72-9c1e-ab7161858f53-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4d989035-dcf0-4a72-9c1e-ab7161858f53\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:48.309933 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:48.309913 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d989035-dcf0-4a72-9c1e-ab7161858f53-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4d989035-dcf0-4a72-9c1e-ab7161858f53\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:48.330668 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:48.330641 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbn4h\" (UniqueName: \"kubernetes.io/projected/4d989035-dcf0-4a72-9c1e-ab7161858f53-kube-api-access-bbn4h\") pod \"data-science-collector-collector-0\" (UID: \"4d989035-dcf0-4a72-9c1e-ab7161858f53\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:48.478075 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:48.477974 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:02:48.777929 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:48.777901 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:02:48.779977 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:02:48.779948 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d989035_dcf0_4a72_9c1e_ab7161858f53.slice/crio-20d88eb3d75cc2aaa3fc5466a7cd87bbbbb5bbb41f600ee8ea12e73ed2af9eaa WatchSource:0}: Error finding container 20d88eb3d75cc2aaa3fc5466a7cd87bbbbb5bbb41f600ee8ea12e73ed2af9eaa: Status 404 returned error can't find the container with id 20d88eb3d75cc2aaa3fc5466a7cd87bbbbb5bbb41f600ee8ea12e73ed2af9eaa Oct 30 04:02:48.782263 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:48.782234 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:02:48.782358 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:48.782288 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:02:49.121494 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:49.121458 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed" path="/var/lib/kubelet/pods/e68f2b05-a8fc-4c2f-bf74-b4c0f11e76ed/volumes" Oct 30 04:02:49.176308 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:49.176268 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" event={"ID":"eda907d6-ab62-4242-8498-41c56f845eaf","Type":"ContainerStarted","Data":"b29bd35f23442c6185ab6f4762d2b6fd9f92792584262478ea11d182e00f18d4"} Oct 30 04:02:49.176474 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:49.176336 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" Oct 30 04:02:49.177988 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:49.177960 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4d989035-dcf0-4a72-9c1e-ab7161858f53","Type":"ContainerStarted","Data":"a203ba1776750812c65f7504512d87ff4f6ece0c401fca016394982f78e1dea5"} Oct 30 04:02:49.178166 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:49.178014 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4d989035-dcf0-4a72-9c1e-ab7161858f53","Type":"ContainerStarted","Data":"20d88eb3d75cc2aaa3fc5466a7cd87bbbbb5bbb41f600ee8ea12e73ed2af9eaa"} Oct 30 04:02:49.219423 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:49.219366 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" podStartSLOduration=2.219350433 podStartE2EDuration="2.219350433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:02:49.21708563 +0000 UTC m=+4930.230363733" watchObservedRunningTime="2026-10-30 04:02:49.219350433 +0000 UTC m=+4930.232628537" Oct 30 04:02:49.259739 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:49.259687 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.25967306 podStartE2EDuration="1.25967306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:02:49.25864942 +0000 UTC m=+4930.271927528" watchObservedRunningTime="2026-10-30 04:02:49.25967306 +0000 UTC m=+4930.272951163" Oct 30 04:02:53.597864 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:53.597822 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" podUID="d0480bc1-0f02-40a0-9ff9-dd717975b349" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:02:54.603122 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:02:54.603084 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" podUID="d52093ca-d97a-4e66-ac82-3a52d050525f" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:03:00.185313 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:00.185284 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" Oct 30 04:03:03.598078 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:03.598037 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" podUID="d0480bc1-0f02-40a0-9ff9-dd717975b349" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:03:03.598468 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:03.598144 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:03:04.603044 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:04.603007 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" podUID="d52093ca-d97a-4e66-ac82-3a52d050525f" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:03:04.603397 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:04.603105 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:03:05.311769 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:05.311722 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-kkt7w"] Oct 30 04:03:05.311954 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:05.311930 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" podUID="eda907d6-ab62-4242-8498-41c56f845eaf" containerName="korrel8r" containerID="cri-o://b29bd35f23442c6185ab6f4762d2b6fd9f92792584262478ea11d182e00f18d4" gracePeriod=30 Oct 30 04:03:05.553289 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:05.553263 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" Oct 30 04:03:05.646832 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:05.646787 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eda907d6-ab62-4242-8498-41c56f845eaf-kube-api-access-bfzgm\" (UniqueName: \"kubernetes.io/projected/eda907d6-ab62-4242-8498-41c56f845eaf-kube-api-access-bfzgm\") pod \"eda907d6-ab62-4242-8498-41c56f845eaf\" (UID: \"eda907d6-ab62-4242-8498-41c56f845eaf\") " Oct 30 04:03:05.647327 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:05.646853 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eda907d6-ab62-4242-8498-41c56f845eaf-tls\" (UniqueName: \"kubernetes.io/secret/eda907d6-ab62-4242-8498-41c56f845eaf-tls\") pod \"eda907d6-ab62-4242-8498-41c56f845eaf\" (UID: \"eda907d6-ab62-4242-8498-41c56f845eaf\") " Oct 30 04:03:05.647327 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:05.646913 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eda907d6-ab62-4242-8498-41c56f845eaf-config\" (UniqueName: \"kubernetes.io/configmap/eda907d6-ab62-4242-8498-41c56f845eaf-config\") pod \"eda907d6-ab62-4242-8498-41c56f845eaf\" (UID: \"eda907d6-ab62-4242-8498-41c56f845eaf\") " Oct 30 04:03:05.647327 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:05.647293 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda907d6-ab62-4242-8498-41c56f845eaf-config" pod "eda907d6-ab62-4242-8498-41c56f845eaf" (UID: "eda907d6-ab62-4242-8498-41c56f845eaf"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:03:05.648973 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:05.648934 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eda907d6-ab62-4242-8498-41c56f845eaf-tls" pod "eda907d6-ab62-4242-8498-41c56f845eaf" (UID: "eda907d6-ab62-4242-8498-41c56f845eaf"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:03:05.649133 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:05.649004 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda907d6-ab62-4242-8498-41c56f845eaf-kube-api-access-bfzgm" pod "eda907d6-ab62-4242-8498-41c56f845eaf" (UID: "eda907d6-ab62-4242-8498-41c56f845eaf"). InnerVolumeSpecName "kube-api-access-bfzgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:03:05.734509 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:05.734470 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:03:05.734731 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:05.734704 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="4d989035-dcf0-4a72-9c1e-ab7161858f53" containerName="otc-container" containerID="cri-o://a203ba1776750812c65f7504512d87ff4f6ece0c401fca016394982f78e1dea5" gracePeriod=30 Oct 30 04:03:05.748883 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:05.748853 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/eda907d6-ab62-4242-8498-41c56f845eaf-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:05.748883 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:05.748881 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eda907d6-ab62-4242-8498-41c56f845eaf-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:05.749118 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:05.748890 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfzgm\" (UniqueName: \"kubernetes.io/projected/eda907d6-ab62-4242-8498-41c56f845eaf-kube-api-access-bfzgm\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:05.977309 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:05.977284 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:06.151597 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.151562 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d989035-dcf0-4a72-9c1e-ab7161858f53-otc-internal\" (UniqueName: \"kubernetes.io/configmap/4d989035-dcf0-4a72-9c1e-ab7161858f53-otc-internal\") pod \"4d989035-dcf0-4a72-9c1e-ab7161858f53\" (UID: \"4d989035-dcf0-4a72-9c1e-ab7161858f53\") " Oct 30 04:03:06.151823 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.151654 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d989035-dcf0-4a72-9c1e-ab7161858f53-kube-api-access-bbn4h\" (UniqueName: \"kubernetes.io/projected/4d989035-dcf0-4a72-9c1e-ab7161858f53-kube-api-access-bbn4h\") pod \"4d989035-dcf0-4a72-9c1e-ab7161858f53\" (UID: \"4d989035-dcf0-4a72-9c1e-ab7161858f53\") " Oct 30 04:03:06.151823 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.151684 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d989035-dcf0-4a72-9c1e-ab7161858f53-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d989035-dcf0-4a72-9c1e-ab7161858f53-monitoring-tls-certs\") pod \"4d989035-dcf0-4a72-9c1e-ab7161858f53\" (UID: \"4d989035-dcf0-4a72-9c1e-ab7161858f53\") " Oct 30 04:03:06.151983 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.151956 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d989035-dcf0-4a72-9c1e-ab7161858f53-otc-internal" pod "4d989035-dcf0-4a72-9c1e-ab7161858f53" (UID: "4d989035-dcf0-4a72-9c1e-ab7161858f53"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:03:06.153839 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.153815 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d989035-dcf0-4a72-9c1e-ab7161858f53-monitoring-tls-certs" pod "4d989035-dcf0-4a72-9c1e-ab7161858f53" (UID: "4d989035-dcf0-4a72-9c1e-ab7161858f53"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:03:06.153922 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.153865 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d989035-dcf0-4a72-9c1e-ab7161858f53-kube-api-access-bbn4h" pod "4d989035-dcf0-4a72-9c1e-ab7161858f53" (UID: "4d989035-dcf0-4a72-9c1e-ab7161858f53"). InnerVolumeSpecName "kube-api-access-bbn4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:03:06.252864 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.252778 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4d989035-dcf0-4a72-9c1e-ab7161858f53-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:06.252864 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.252812 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbn4h\" (UniqueName: \"kubernetes.io/projected/4d989035-dcf0-4a72-9c1e-ab7161858f53-kube-api-access-bbn4h\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:06.252864 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.252825 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d989035-dcf0-4a72-9c1e-ab7161858f53-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:06.266318 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.266284 4925 generic.go:356] "Generic (PLEG): container finished" podID="eda907d6-ab62-4242-8498-41c56f845eaf" containerID="b29bd35f23442c6185ab6f4762d2b6fd9f92792584262478ea11d182e00f18d4" exitCode=0 Oct 30 04:03:06.266472 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.266328 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" Oct 30 04:03:06.266472 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.266361 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" event={"ID":"eda907d6-ab62-4242-8498-41c56f845eaf","Type":"ContainerDied","Data":"b29bd35f23442c6185ab6f4762d2b6fd9f92792584262478ea11d182e00f18d4"} Oct 30 04:03:06.266472 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.266416 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-kkt7w" event={"ID":"eda907d6-ab62-4242-8498-41c56f845eaf","Type":"ContainerDied","Data":"9d2407da920b5f68e677f325376c47f6c3a42cd307cbea2cc7da8c16b0fdb223"} Oct 30 04:03:06.266472 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.266447 4925 scope.go:122] "RemoveContainer" containerID="b29bd35f23442c6185ab6f4762d2b6fd9f92792584262478ea11d182e00f18d4" Oct 30 04:03:06.267862 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.267837 4925 generic.go:356] "Generic (PLEG): container finished" podID="4d989035-dcf0-4a72-9c1e-ab7161858f53" containerID="a203ba1776750812c65f7504512d87ff4f6ece0c401fca016394982f78e1dea5" exitCode=0 Oct 30 04:03:06.267992 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.267878 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4d989035-dcf0-4a72-9c1e-ab7161858f53","Type":"ContainerDied","Data":"a203ba1776750812c65f7504512d87ff4f6ece0c401fca016394982f78e1dea5"} Oct 30 04:03:06.267992 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.267931 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4d989035-dcf0-4a72-9c1e-ab7161858f53","Type":"ContainerDied","Data":"20d88eb3d75cc2aaa3fc5466a7cd87bbbbb5bbb41f600ee8ea12e73ed2af9eaa"} Oct 30 04:03:06.267992 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.267891 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:06.280239 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.280030 4925 scope.go:122] "RemoveContainer" containerID="b29bd35f23442c6185ab6f4762d2b6fd9f92792584262478ea11d182e00f18d4" Oct 30 04:03:06.280557 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:03:06.280530 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b29bd35f23442c6185ab6f4762d2b6fd9f92792584262478ea11d182e00f18d4\": container with ID starting with b29bd35f23442c6185ab6f4762d2b6fd9f92792584262478ea11d182e00f18d4 not found: ID does not exist" containerID="b29bd35f23442c6185ab6f4762d2b6fd9f92792584262478ea11d182e00f18d4" Oct 30 04:03:06.280646 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.280570 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b29bd35f23442c6185ab6f4762d2b6fd9f92792584262478ea11d182e00f18d4"} err="failed to get container status \"b29bd35f23442c6185ab6f4762d2b6fd9f92792584262478ea11d182e00f18d4\": rpc error: code = NotFound desc = could not find container \"b29bd35f23442c6185ab6f4762d2b6fd9f92792584262478ea11d182e00f18d4\": container with ID starting with b29bd35f23442c6185ab6f4762d2b6fd9f92792584262478ea11d182e00f18d4 not found: ID does not exist" Oct 30 04:03:06.280646 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.280611 4925 scope.go:122] "RemoveContainer" containerID="a203ba1776750812c65f7504512d87ff4f6ece0c401fca016394982f78e1dea5" Oct 30 04:03:06.290145 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.290124 4925 scope.go:122] "RemoveContainer" containerID="a203ba1776750812c65f7504512d87ff4f6ece0c401fca016394982f78e1dea5" Oct 30 04:03:06.290427 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:03:06.290407 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a203ba1776750812c65f7504512d87ff4f6ece0c401fca016394982f78e1dea5\": container with ID starting with a203ba1776750812c65f7504512d87ff4f6ece0c401fca016394982f78e1dea5 not found: ID does not exist" containerID="a203ba1776750812c65f7504512d87ff4f6ece0c401fca016394982f78e1dea5" Oct 30 04:03:06.290474 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.290435 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a203ba1776750812c65f7504512d87ff4f6ece0c401fca016394982f78e1dea5"} err="failed to get container status \"a203ba1776750812c65f7504512d87ff4f6ece0c401fca016394982f78e1dea5\": rpc error: code = NotFound desc = could not find container \"a203ba1776750812c65f7504512d87ff4f6ece0c401fca016394982f78e1dea5\": container with ID starting with a203ba1776750812c65f7504512d87ff4f6ece0c401fca016394982f78e1dea5 not found: ID does not exist" Oct 30 04:03:06.320346 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.320308 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:03:06.324825 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.324796 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:03:06.359435 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.359392 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-kkt7w"] Oct 30 04:03:06.363913 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:06.363882 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-kkt7w"] Oct 30 04:03:07.119065 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:07.119034 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d989035-dcf0-4a72-9c1e-ab7161858f53" path="/var/lib/kubelet/pods/4d989035-dcf0-4a72-9c1e-ab7161858f53/volumes" Oct 30 04:03:07.119445 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:07.119420 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eda907d6-ab62-4242-8498-41c56f845eaf" path="/var/lib/kubelet/pods/eda907d6-ab62-4242-8498-41c56f845eaf/volumes" Oct 30 04:03:10.139425 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.139386 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8"] Oct 30 04:03:10.139866 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.139744 4925 cpu_manager.go:414] "containerMap: removing container" podUID="eda907d6-ab62-4242-8498-41c56f845eaf" containerName="korrel8r" Oct 30 04:03:10.139866 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.139771 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eda907d6-ab62-4242-8498-41c56f845eaf" containerName="korrel8r" Oct 30 04:03:10.139866 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.139820 4925 cpu_manager.go:414] "containerMap: removing container" podUID="4d989035-dcf0-4a72-9c1e-ab7161858f53" containerName="otc-container" Oct 30 04:03:10.139866 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.139826 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d989035-dcf0-4a72-9c1e-ab7161858f53" containerName="otc-container" Oct 30 04:03:10.139995 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.139890 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d989035-dcf0-4a72-9c1e-ab7161858f53" containerName="otc-container" Oct 30 04:03:10.139995 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.139900 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="eda907d6-ab62-4242-8498-41c56f845eaf" containerName="korrel8r" Oct 30 04:03:10.144586 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.144564 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:03:10.168141 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.168082 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 04:03:10.180135 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.180104 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 04:03:10.180323 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.180167 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 04:03:10.180323 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.180176 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 04:03:10.180323 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.180205 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-9ck7r\"" Oct 30 04:03:10.181823 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.181779 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8"] Oct 30 04:03:10.276904 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.276867 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r"] Oct 30 04:03:10.280350 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.280332 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" Oct 30 04:03:10.287848 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.287821 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26q9x\" (UniqueName: \"kubernetes.io/projected/776067d7-1633-44ae-986c-56cafc4be4c4-kube-api-access-26q9x\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-5shw8\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:03:10.287949 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.287870 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-5shw8\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:03:10.287987 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.287968 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-5shw8\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:03:10.288023 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.287998 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-5shw8\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:03:10.288132 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.288108 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/776067d7-1633-44ae-986c-56cafc4be4c4-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-5shw8\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:03:10.303780 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.303730 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r"] Oct 30 04:03:10.368142 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.368104 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 04:03:10.368142 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.368108 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-25ngn\"" Oct 30 04:03:10.368363 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.368182 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 04:03:10.389360 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.389325 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-5shw8\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:03:10.389569 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.389370 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-5shw8\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:03:10.389569 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.389415 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddv4n\" (UniqueName: \"kubernetes.io/projected/b5e58337-f692-4b01-b15d-7642a69d67f2-kube-api-access-ddv4n\") pod \"korrel8r-5cbcc89b8f-5b48r\" (UID: \"b5e58337-f692-4b01-b15d-7642a69d67f2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" Oct 30 04:03:10.389569 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.389442 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/776067d7-1633-44ae-986c-56cafc4be4c4-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-5shw8\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:03:10.389569 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:03:10.389484 4925 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 04:03:10.389569 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.389492 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b5e58337-f692-4b01-b15d-7642a69d67f2-tls\") pod \"korrel8r-5cbcc89b8f-5b48r\" (UID: \"b5e58337-f692-4b01-b15d-7642a69d67f2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" Oct 30 04:03:10.389875 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:03:10.389610 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-prometheus-client-cert podName:776067d7-1633-44ae-986c-56cafc4be4c4 nodeName:}" failed. No retries permitted until 2026-10-30 04:03:10.889585015 +0000 UTC m=+4951.902863107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-5shw8" (UID: "776067d7-1633-44ae-986c-56cafc4be4c4") : secret "prometheus-operated-tls" not found Oct 30 04:03:10.389875 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.389684 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26q9x\" (UniqueName: \"kubernetes.io/projected/776067d7-1633-44ae-986c-56cafc4be4c4-kube-api-access-26q9x\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-5shw8\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:03:10.389875 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.389717 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5e58337-f692-4b01-b15d-7642a69d67f2-config\") pod \"korrel8r-5cbcc89b8f-5b48r\" (UID: \"b5e58337-f692-4b01-b15d-7642a69d67f2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" Oct 30 04:03:10.389875 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.389784 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-5shw8\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:03:10.390174 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.390151 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/776067d7-1633-44ae-986c-56cafc4be4c4-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-5shw8\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:03:10.392100 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.392072 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-5shw8\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:03:10.392196 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.392116 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-5shw8\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:03:10.414257 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.414229 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26q9x\" (UniqueName: \"kubernetes.io/projected/776067d7-1633-44ae-986c-56cafc4be4c4-kube-api-access-26q9x\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-5shw8\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:03:10.490195 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.490161 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5e58337-f692-4b01-b15d-7642a69d67f2-config\") pod \"korrel8r-5cbcc89b8f-5b48r\" (UID: \"b5e58337-f692-4b01-b15d-7642a69d67f2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" Oct 30 04:03:10.490383 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.490258 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddv4n\" (UniqueName: \"kubernetes.io/projected/b5e58337-f692-4b01-b15d-7642a69d67f2-kube-api-access-ddv4n\") pod \"korrel8r-5cbcc89b8f-5b48r\" (UID: \"b5e58337-f692-4b01-b15d-7642a69d67f2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" Oct 30 04:03:10.490383 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.490300 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b5e58337-f692-4b01-b15d-7642a69d67f2-tls\") pod \"korrel8r-5cbcc89b8f-5b48r\" (UID: \"b5e58337-f692-4b01-b15d-7642a69d67f2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" Oct 30 04:03:10.491102 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.491076 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5e58337-f692-4b01-b15d-7642a69d67f2-config\") pod \"korrel8r-5cbcc89b8f-5b48r\" (UID: \"b5e58337-f692-4b01-b15d-7642a69d67f2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" Oct 30 04:03:10.493064 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.493041 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b5e58337-f692-4b01-b15d-7642a69d67f2-tls\") pod \"korrel8r-5cbcc89b8f-5b48r\" (UID: \"b5e58337-f692-4b01-b15d-7642a69d67f2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" Oct 30 04:03:10.513944 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.513916 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddv4n\" (UniqueName: \"kubernetes.io/projected/b5e58337-f692-4b01-b15d-7642a69d67f2-kube-api-access-ddv4n\") pod \"korrel8r-5cbcc89b8f-5b48r\" (UID: \"b5e58337-f692-4b01-b15d-7642a69d67f2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" Oct 30 04:03:10.593367 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.593333 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" Oct 30 04:03:10.892725 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.892604 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r"] Oct 30 04:03:10.893234 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.893208 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-5shw8\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:03:10.895165 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:03:10.895142 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e58337_f692_4b01_b15d_7642a69d67f2.slice/crio-325e04d066e5e0df30f65363d42b6bb6a57b39febf133aa9c03ab4e2e3e70f73 WatchSource:0}: Error finding container 325e04d066e5e0df30f65363d42b6bb6a57b39febf133aa9c03ab4e2e3e70f73: Status 404 returned error can't find the container with id 325e04d066e5e0df30f65363d42b6bb6a57b39febf133aa9c03ab4e2e3e70f73 Oct 30 04:03:10.895846 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:10.895825 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-5shw8\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:03:11.057160 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.057120 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:03:11.302214 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.302129 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" event={"ID":"b5e58337-f692-4b01-b15d-7642a69d67f2","Type":"ContainerStarted","Data":"d4aa3f5e7a4cd3d35825d1f8b3e8529d63aedb36a7ce2364a487a5c45500961c"} Oct 30 04:03:11.302214 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.302179 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" event={"ID":"b5e58337-f692-4b01-b15d-7642a69d67f2","Type":"ContainerStarted","Data":"325e04d066e5e0df30f65363d42b6bb6a57b39febf133aa9c03ab4e2e3e70f73"} Oct 30 04:03:11.302613 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.302229 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" Oct 30 04:03:11.352088 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.352036 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" podStartSLOduration=1.352021296 podStartE2EDuration="1.352021296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:11.343121359 +0000 UTC m=+4952.356399462" watchObservedRunningTime="2026-10-30 04:03:11.352021296 +0000 UTC m=+4952.365299398" Oct 30 04:03:11.353578 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.353557 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8"] Oct 30 04:03:11.359801 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:03:11.359771 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776067d7_1633_44ae_986c_56cafc4be4c4.slice/crio-b420f2a5046bfdd8d48117f33840782bfd898c2c3f90037c9b854ecb68235353 WatchSource:0}: Error finding container b420f2a5046bfdd8d48117f33840782bfd898c2c3f90037c9b854ecb68235353: Status 404 returned error can't find the container with id b420f2a5046bfdd8d48117f33840782bfd898c2c3f90037c9b854ecb68235353 Oct 30 04:03:11.781037 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.781006 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:03:11.786035 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.786012 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:11.800003 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.799967 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:11.800152 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.800087 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:11.800217 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.800161 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg7gh\" (UniqueName: \"kubernetes.io/projected/3603ba78-be34-44f4-92ee-3ad3f5b457bb-kube-api-access-jg7gh\") pod \"data-science-collector-collector-0\" (UID: \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:11.800285 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.800266 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:11.800333 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.800303 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3603ba78-be34-44f4-92ee-3ad3f5b457bb-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:11.808903 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.808509 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:03:11.809386 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.809360 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:03:11.818074 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.818040 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:03:11.818201 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.818140 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:03:11.818201 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.818191 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-45ch8\"" Oct 30 04:03:11.818286 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.818260 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:03:11.901423 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.901330 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:11.901423 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.901382 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg7gh\" (UniqueName: \"kubernetes.io/projected/3603ba78-be34-44f4-92ee-3ad3f5b457bb-kube-api-access-jg7gh\") pod \"data-science-collector-collector-0\" (UID: \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:11.901661 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.901424 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:11.901661 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.901452 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3603ba78-be34-44f4-92ee-3ad3f5b457bb-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:11.901661 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.901502 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:11.903173 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.903139 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3603ba78-be34-44f4-92ee-3ad3f5b457bb-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:11.904806 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.904779 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:11.905700 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.905678 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:11.914465 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.914430 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:11.928801 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:11.928438 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg7gh\" (UniqueName: \"kubernetes.io/projected/3603ba78-be34-44f4-92ee-3ad3f5b457bb-kube-api-access-jg7gh\") pod \"data-science-collector-collector-0\" (UID: \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:12.102487 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:12.102451 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:12.309682 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:12.309635 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" event={"ID":"776067d7-1633-44ae-986c-56cafc4be4c4","Type":"ContainerStarted","Data":"f5edc65e25d7f11d3f41f39e93908b74d309ea4d6cac4b384d9c33ed62d3b931"} Oct 30 04:03:12.310201 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:12.309841 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:03:12.310201 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:12.309864 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" event={"ID":"776067d7-1633-44ae-986c-56cafc4be4c4","Type":"ContainerStarted","Data":"b420f2a5046bfdd8d48117f33840782bfd898c2c3f90037c9b854ecb68235353"} Oct 30 04:03:12.352301 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:12.352252 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" podStartSLOduration=2.352239654 podStartE2EDuration="2.352239654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:12.351610209 +0000 UTC m=+4953.364888315" watchObservedRunningTime="2026-10-30 04:03:12.352239654 +0000 UTC m=+4953.365517757" Oct 30 04:03:12.407979 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:12.407954 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:03:12.410035 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:03:12.410010 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3603ba78_be34_44f4_92ee_3ad3f5b457bb.slice/crio-2973ea1ca50f9f3cfa6e543993914d8b81c05d5d509e25b462d588ad6ed56ce7 WatchSource:0}: Error finding container 2973ea1ca50f9f3cfa6e543993914d8b81c05d5d509e25b462d588ad6ed56ce7: Status 404 returned error can't find the container with id 2973ea1ca50f9f3cfa6e543993914d8b81c05d5d509e25b462d588ad6ed56ce7 Oct 30 04:03:12.412308 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:12.412266 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:03:12.412407 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:12.412360 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:03:13.317193 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.317149 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"3603ba78-be34-44f4-92ee-3ad3f5b457bb","Type":"ContainerStarted","Data":"b89350aa023f79a8e52e807be2aa94a22f9a4c17a3f80e7c5d1b720ff0fbb2f0"} Oct 30 04:03:13.317700 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.317209 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"3603ba78-be34-44f4-92ee-3ad3f5b457bb","Type":"ContainerStarted","Data":"2973ea1ca50f9f3cfa6e543993914d8b81c05d5d509e25b462d588ad6ed56ce7"} Oct 30 04:03:13.360638 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.360578 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.360563316 podStartE2EDuration="2.360563316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:13.359585315 +0000 UTC m=+4954.372863418" watchObservedRunningTime="2026-10-30 04:03:13.360563316 +0000 UTC m=+4954.373841419" Oct 30 04:03:13.594358 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.594080 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" podUID="d0480bc1-0f02-40a0-9ff9-dd717975b349" containerName="tempo" probeResult="failure" output="Get \"https://10.132.0.95:3101/ready\": dial tcp 10.132.0.95:3101: connect: connection refused" Oct 30 04:03:13.730017 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.729992 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:03:13.732500 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.732477 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:03:13.818622 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.818585 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-data-science-tempostack-querier-mtls\") pod \"d0480bc1-0f02-40a0-9ff9-dd717975b349\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " Oct 30 04:03:13.818622 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.818635 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52093ca-d97a-4e66-ac82-3a52d050525f-kube-api-access-l6v4n\" (UniqueName: \"kubernetes.io/projected/d52093ca-d97a-4e66-ac82-3a52d050525f-kube-api-access-l6v4n\") pod \"d52093ca-d97a-4e66-ac82-3a52d050525f\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " Oct 30 04:03:13.820154 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.818654 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-conf\") pod \"d0480bc1-0f02-40a0-9ff9-dd717975b349\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " Oct 30 04:03:13.820154 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.818679 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-conf\") pod \"d52093ca-d97a-4e66-ac82-3a52d050525f\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " Oct 30 04:03:13.820154 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.818722 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-data-science-tempostack-ca-bundle\") pod \"d0480bc1-0f02-40a0-9ff9-dd717975b349\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " Oct 30 04:03:13.820154 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.818780 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-data-science-tempostack-query-frontend-mtls\") pod \"d52093ca-d97a-4e66-ac82-3a52d050525f\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " Oct 30 04:03:13.820154 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.818808 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-tmp-storage\") pod \"d0480bc1-0f02-40a0-9ff9-dd717975b349\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " Oct 30 04:03:13.820154 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.818856 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-data-science-tempostack-ca-bundle\") pod \"d52093ca-d97a-4e66-ac82-3a52d050525f\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " Oct 30 04:03:13.820154 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.818886 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-tmp-storage\") pod \"d52093ca-d97a-4e66-ac82-3a52d050525f\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " Oct 30 04:03:13.820154 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.818917 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d52093ca-d97a-4e66-ac82-3a52d050525f-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/d52093ca-d97a-4e66-ac82-3a52d050525f-storage-gcs-key\") pod \"d52093ca-d97a-4e66-ac82-3a52d050525f\" (UID: \"d52093ca-d97a-4e66-ac82-3a52d050525f\") " Oct 30 04:03:13.820154 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.818949 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0480bc1-0f02-40a0-9ff9-dd717975b349-kube-api-access-5kmd4\" (UniqueName: \"kubernetes.io/projected/d0480bc1-0f02-40a0-9ff9-dd717975b349-kube-api-access-5kmd4\") pod \"d0480bc1-0f02-40a0-9ff9-dd717975b349\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " Oct 30 04:03:13.820154 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.818983 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0480bc1-0f02-40a0-9ff9-dd717975b349-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/d0480bc1-0f02-40a0-9ff9-dd717975b349-storage-gcs-key\") pod \"d0480bc1-0f02-40a0-9ff9-dd717975b349\" (UID: \"d0480bc1-0f02-40a0-9ff9-dd717975b349\") " Oct 30 04:03:13.820154 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.819208 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-conf" pod "d52093ca-d97a-4e66-ac82-3a52d050525f" (UID: "d52093ca-d97a-4e66-ac82-3a52d050525f"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:03:13.820154 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.819290 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-tmp-storage" pod "d52093ca-d97a-4e66-ac82-3a52d050525f" (UID: "d52093ca-d97a-4e66-ac82-3a52d050525f"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:03:13.820154 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.819388 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-data-science-tempostack-ca-bundle" pod "d52093ca-d97a-4e66-ac82-3a52d050525f" (UID: "d52093ca-d97a-4e66-ac82-3a52d050525f"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:03:13.820154 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.819922 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-data-science-tempostack-ca-bundle" pod "d0480bc1-0f02-40a0-9ff9-dd717975b349" (UID: "d0480bc1-0f02-40a0-9ff9-dd717975b349"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:03:13.820944 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.820210 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-conf" pod "d0480bc1-0f02-40a0-9ff9-dd717975b349" (UID: "d0480bc1-0f02-40a0-9ff9-dd717975b349"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:03:13.820944 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.820871 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-tmp-storage" pod "d0480bc1-0f02-40a0-9ff9-dd717975b349" (UID: "d0480bc1-0f02-40a0-9ff9-dd717975b349"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:03:13.822353 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.821582 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-data-science-tempostack-query-frontend-mtls" pod "d52093ca-d97a-4e66-ac82-3a52d050525f" (UID: "d52093ca-d97a-4e66-ac82-3a52d050525f"). InnerVolumeSpecName "tempo-data-science-tempostack-query-frontend-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:03:13.822353 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.822332 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0480bc1-0f02-40a0-9ff9-dd717975b349-storage-gcs-key" pod "d0480bc1-0f02-40a0-9ff9-dd717975b349" (UID: "d0480bc1-0f02-40a0-9ff9-dd717975b349"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:03:13.822838 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.822806 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52093ca-d97a-4e66-ac82-3a52d050525f-kube-api-access-l6v4n" pod "d52093ca-d97a-4e66-ac82-3a52d050525f" (UID: "d52093ca-d97a-4e66-ac82-3a52d050525f"). InnerVolumeSpecName "kube-api-access-l6v4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:03:13.823113 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.823089 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0480bc1-0f02-40a0-9ff9-dd717975b349-kube-api-access-5kmd4" pod "d0480bc1-0f02-40a0-9ff9-dd717975b349" (UID: "d0480bc1-0f02-40a0-9ff9-dd717975b349"). InnerVolumeSpecName "kube-api-access-5kmd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:03:13.823219 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.823200 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-data-science-tempostack-querier-mtls" pod "d0480bc1-0f02-40a0-9ff9-dd717975b349" (UID: "d0480bc1-0f02-40a0-9ff9-dd717975b349"). InnerVolumeSpecName "tempo-data-science-tempostack-querier-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:03:13.823525 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.823506 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d52093ca-d97a-4e66-ac82-3a52d050525f-storage-gcs-key" pod "d52093ca-d97a-4e66-ac82-3a52d050525f" (UID: "d52093ca-d97a-4e66-ac82-3a52d050525f"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:03:13.919567 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.919525 4925 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:13.919567 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.919561 4925 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-tmp-storage\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:13.919567 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.919574 4925 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/d52093ca-d97a-4e66-ac82-3a52d050525f-storage-gcs-key\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:13.919878 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.919586 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kmd4\" (UniqueName: \"kubernetes.io/projected/d0480bc1-0f02-40a0-9ff9-dd717975b349-kube-api-access-5kmd4\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:13.919878 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.919600 4925 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/d0480bc1-0f02-40a0-9ff9-dd717975b349-storage-gcs-key\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:13.919878 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.919612 4925 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-data-science-tempostack-querier-mtls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:13.919878 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.919624 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6v4n\" (UniqueName: \"kubernetes.io/projected/d52093ca-d97a-4e66-ac82-3a52d050525f-kube-api-access-l6v4n\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:13.919878 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.919636 4925 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-conf\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:13.919878 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.919649 4925 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-conf\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:13.919878 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.919660 4925 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:13.919878 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.919673 4925 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/d52093ca-d97a-4e66-ac82-3a52d050525f-tempo-data-science-tempostack-query-frontend-mtls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:13.919878 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:13.919687 4925 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d0480bc1-0f02-40a0-9ff9-dd717975b349-tempo-tmp-storage\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:14.324477 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:14.324440 4925 generic.go:356] "Generic (PLEG): container finished" podID="d0480bc1-0f02-40a0-9ff9-dd717975b349" containerID="8db4e5eb58c261dc5722836620e56eee0229dfb44a0f76491b5898fc8a7a2209" exitCode=137 Oct 30 04:03:14.325003 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:14.324489 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" Oct 30 04:03:14.325003 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:14.324567 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" event={"ID":"d0480bc1-0f02-40a0-9ff9-dd717975b349","Type":"ContainerDied","Data":"8db4e5eb58c261dc5722836620e56eee0229dfb44a0f76491b5898fc8a7a2209"} Oct 30 04:03:14.325003 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:14.324612 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx" event={"ID":"d0480bc1-0f02-40a0-9ff9-dd717975b349","Type":"ContainerDied","Data":"ea7e233140c655bba6f5c3b69c93f55929a170a99bdc9e8f3160b335fb5daefe"} Oct 30 04:03:14.325003 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:14.324628 4925 scope.go:122] "RemoveContainer" containerID="8db4e5eb58c261dc5722836620e56eee0229dfb44a0f76491b5898fc8a7a2209" Oct 30 04:03:14.326125 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:14.326106 4925 generic.go:356] "Generic (PLEG): container finished" podID="d52093ca-d97a-4e66-ac82-3a52d050525f" containerID="aafc5b90548175086e4e71d442fa0fc8f9b812d6191d6d9ac2a52cb1b3193cb9" exitCode=137 Oct 30 04:03:14.326225 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:14.326153 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" Oct 30 04:03:14.326225 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:14.326184 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" event={"ID":"d52093ca-d97a-4e66-ac82-3a52d050525f","Type":"ContainerDied","Data":"aafc5b90548175086e4e71d442fa0fc8f9b812d6191d6d9ac2a52cb1b3193cb9"} Oct 30 04:03:14.326332 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:14.326248 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4" event={"ID":"d52093ca-d97a-4e66-ac82-3a52d050525f","Type":"ContainerDied","Data":"a2f9693605afb6c11e99faa4af866307a433c70536fa649864c9235ec1fe15dd"} Oct 30 04:03:14.335484 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:14.335376 4925 scope.go:122] "RemoveContainer" containerID="8db4e5eb58c261dc5722836620e56eee0229dfb44a0f76491b5898fc8a7a2209" Oct 30 04:03:14.335722 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:03:14.335686 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8db4e5eb58c261dc5722836620e56eee0229dfb44a0f76491b5898fc8a7a2209\": container with ID starting with 8db4e5eb58c261dc5722836620e56eee0229dfb44a0f76491b5898fc8a7a2209 not found: ID does not exist" containerID="8db4e5eb58c261dc5722836620e56eee0229dfb44a0f76491b5898fc8a7a2209" Oct 30 04:03:14.335837 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:14.335732 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8db4e5eb58c261dc5722836620e56eee0229dfb44a0f76491b5898fc8a7a2209"} err="failed to get container status \"8db4e5eb58c261dc5722836620e56eee0229dfb44a0f76491b5898fc8a7a2209\": rpc error: code = NotFound desc = could not find container \"8db4e5eb58c261dc5722836620e56eee0229dfb44a0f76491b5898fc8a7a2209\": container with ID starting with 8db4e5eb58c261dc5722836620e56eee0229dfb44a0f76491b5898fc8a7a2209 not found: ID does not exist" Oct 30 04:03:14.335837 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:14.335762 4925 scope.go:122] "RemoveContainer" containerID="aafc5b90548175086e4e71d442fa0fc8f9b812d6191d6d9ac2a52cb1b3193cb9" Oct 30 04:03:14.346036 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:03:14.346006 4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0480bc1_0f02_40a0_9ff9_dd717975b349.slice\": RecentStats: unable to find data in memory cache]" Oct 30 04:03:14.347003 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:14.346983 4925 scope.go:122] "RemoveContainer" containerID="aafc5b90548175086e4e71d442fa0fc8f9b812d6191d6d9ac2a52cb1b3193cb9" Oct 30 04:03:14.347321 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:03:14.347297 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aafc5b90548175086e4e71d442fa0fc8f9b812d6191d6d9ac2a52cb1b3193cb9\": container with ID starting with aafc5b90548175086e4e71d442fa0fc8f9b812d6191d6d9ac2a52cb1b3193cb9 not found: ID does not exist" containerID="aafc5b90548175086e4e71d442fa0fc8f9b812d6191d6d9ac2a52cb1b3193cb9" Oct 30 04:03:14.347404 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:14.347332 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aafc5b90548175086e4e71d442fa0fc8f9b812d6191d6d9ac2a52cb1b3193cb9"} err="failed to get container status \"aafc5b90548175086e4e71d442fa0fc8f9b812d6191d6d9ac2a52cb1b3193cb9\": rpc error: code = NotFound desc = could not find container \"aafc5b90548175086e4e71d442fa0fc8f9b812d6191d6d9ac2a52cb1b3193cb9\": container with ID starting with aafc5b90548175086e4e71d442fa0fc8f9b812d6191d6d9ac2a52cb1b3193cb9 not found: ID does not exist" Oct 30 04:03:14.377119 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:14.377079 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx"] Oct 30 04:03:14.389637 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:14.389606 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7c5b99b89b-n42qx"] Oct 30 04:03:14.416420 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:14.416385 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4"] Oct 30 04:03:14.420224 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:14.420195 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-5877dc5cbc-dhcd4"] Oct 30 04:03:15.120478 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.120443 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0480bc1-0f02-40a0-9ff9-dd717975b349" path="/var/lib/kubelet/pods/d0480bc1-0f02-40a0-9ff9-dd717975b349/volumes" Oct 30 04:03:15.120889 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.120875 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d52093ca-d97a-4e66-ac82-3a52d050525f" path="/var/lib/kubelet/pods/d52093ca-d97a-4e66-ac82-3a52d050525f/volumes" Oct 30 04:03:15.396022 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.395931 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:03:15.396487 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.396461 4925 cpu_manager.go:414] "containerMap: removing container" podUID="d0480bc1-0f02-40a0-9ff9-dd717975b349" containerName="tempo" Oct 30 04:03:15.396487 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.396480 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0480bc1-0f02-40a0-9ff9-dd717975b349" containerName="tempo" Oct 30 04:03:15.396579 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.396514 4925 cpu_manager.go:414] "containerMap: removing container" podUID="d52093ca-d97a-4e66-ac82-3a52d050525f" containerName="tempo" Oct 30 04:03:15.396579 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.396520 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52093ca-d97a-4e66-ac82-3a52d050525f" containerName="tempo" Oct 30 04:03:15.396579 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.396577 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0480bc1-0f02-40a0-9ff9-dd717975b349" containerName="tempo" Oct 30 04:03:15.396679 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.396587 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52093ca-d97a-4e66-ac82-3a52d050525f" containerName="tempo" Oct 30 04:03:15.401635 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.401612 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.417270 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.417240 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 04:03:15.417270 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.417243 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:03:15.417472 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.417243 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 04:03:15.417472 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.417258 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 04:03:15.429323 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.429271 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-hnkwv\"" Oct 30 04:03:15.430694 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.430668 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.430846 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.430704 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bf40867-6019-4c8c-9488-b9ed3cff2efb-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.430846 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.430723 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.430930 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.430866 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrz2p\" (UniqueName: \"kubernetes.io/projected/8bf40867-6019-4c8c-9488-b9ed3cff2efb-kube-api-access-wrz2p\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.430973 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.430943 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bf40867-6019-4c8c-9488-b9ed3cff2efb-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.431052 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.431026 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.431102 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.431075 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/8bf40867-6019-4c8c-9488-b9ed3cff2efb-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.431592 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.431570 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:03:15.531812 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.531744 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bf40867-6019-4c8c-9488-b9ed3cff2efb-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.531812 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.531813 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.532087 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.531844 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/8bf40867-6019-4c8c-9488-b9ed3cff2efb-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.532087 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.531871 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.532087 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.531896 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bf40867-6019-4c8c-9488-b9ed3cff2efb-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.532265 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.532085 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.532327 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.532185 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrz2p\" (UniqueName: \"kubernetes.io/projected/8bf40867-6019-4c8c-9488-b9ed3cff2efb-kube-api-access-wrz2p\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.532432 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.532408 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/8bf40867-6019-4c8c-9488-b9ed3cff2efb-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.534477 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.534451 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bf40867-6019-4c8c-9488-b9ed3cff2efb-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.534639 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.534609 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.534706 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.534652 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.534912 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.534893 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bf40867-6019-4c8c-9488-b9ed3cff2efb-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.535386 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.535368 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.555792 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.555738 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrz2p\" (UniqueName: \"kubernetes.io/projected/8bf40867-6019-4c8c-9488-b9ed3cff2efb-kube-api-access-wrz2p\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.576818 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.576782 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:03:15.581829 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.581803 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.609088 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.609055 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:03:15.617847 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.617808 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:03:15.617847 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.617834 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 04:03:15.617847 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.617849 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 04:03:15.618146 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.617820 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 04:03:15.618146 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.617823 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 04:03:15.618146 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.617889 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-5fhss\"" Oct 30 04:03:15.674470 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.674374 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 04:03:15.674470 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.674417 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 04:03:15.674670 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.674514 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 04:03:15.719651 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.719614 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:15.734087 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.734045 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.734214 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.734100 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m68gt\" (UniqueName: \"kubernetes.io/projected/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-kube-api-access-m68gt\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.734262 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.734202 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.734317 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.734255 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.734369 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.734342 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.734446 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.734427 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.734540 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.734513 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.734602 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.734576 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.734651 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.734601 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.734651 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.734627 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.734793 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.734767 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.734905 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.734826 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.835420 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.835383 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.835420 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.835433 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.835692 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.835456 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m68gt\" (UniqueName: \"kubernetes.io/projected/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-kube-api-access-m68gt\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.835692 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.835486 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.835692 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.835506 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.835692 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.835527 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.835692 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.835598 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.835692 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.835654 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.835692 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.835681 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.836074 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.835705 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.836074 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.835730 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.836074 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.835785 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.836404 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.836375 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.836553 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.836523 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.837161 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.837129 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.839193 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.838808 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.839193 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.838853 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.839193 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.838930 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.839193 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.838983 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.839193 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.839000 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.839193 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.839155 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.839491 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.839365 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.840151 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.840127 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:15.859378 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:15.859354 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m68gt\" (UniqueName: \"kubernetes.io/projected/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-kube-api-access-m68gt\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:16.034763 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:16.034724 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:03:16.036723 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:03:16.036691 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf40867_6019_4c8c_9488_b9ed3cff2efb.slice/crio-9822a8e9a965693da09d7951521a43a5a16a056328338dbcd646fd98ba6fea28 WatchSource:0}: Error finding container 9822a8e9a965693da09d7951521a43a5a16a056328338dbcd646fd98ba6fea28: Status 404 returned error can't find the container with id 9822a8e9a965693da09d7951521a43a5a16a056328338dbcd646fd98ba6fea28 Oct 30 04:03:16.340602 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:16.340567 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"8bf40867-6019-4c8c-9488-b9ed3cff2efb","Type":"ContainerStarted","Data":"9822a8e9a965693da09d7951521a43a5a16a056328338dbcd646fd98ba6fea28"} Oct 30 04:03:17.347136 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:17.347094 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"8bf40867-6019-4c8c-9488-b9ed3cff2efb","Type":"ContainerStarted","Data":"d352db5b2a093942154c4f6c978d81459212200c33195bfeff8d8fe75c903135"} Oct 30 04:03:18.163030 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:03:18.162990 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:03:18.319637 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:18.319604 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:03:19.368384 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:19.368341 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:19.398023 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:19.397990 4925 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:19.469382 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:19.469341 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:19.469541 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:19.469485 4925 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:19.484145 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:19.484112 4925 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") DevicePath \"csi-ec2ccb9dcc2629e5a59ea91641dbb6b15d80e87697977b7c41508ff0003b2f36\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:19.589439 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:19.589408 4925 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a535375d22e374ba333968465471c7604a06f9d95a513910fe4bb083b629e986/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:19.618720 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:19.618653 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:19.814324 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:19.814280 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:20.157777 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:03:20.157731 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3a5697_5bf8_4f0e_8017_e36d0738bbc6.slice/crio-2122fda3a7c1c289c7006077a7fbdf8e514199cd452439782d7801e2cadbeb5a WatchSource:0}: Error finding container 2122fda3a7c1c289c7006077a7fbdf8e514199cd452439782d7801e2cadbeb5a: Status 404 returned error can't find the container with id 2122fda3a7c1c289c7006077a7fbdf8e514199cd452439782d7801e2cadbeb5a Oct 30 04:03:20.170181 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:20.170154 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:03:20.364335 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:20.364283 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6","Type":"ContainerStarted","Data":"2122fda3a7c1c289c7006077a7fbdf8e514199cd452439782d7801e2cadbeb5a"} Oct 30 04:03:20.365914 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:20.365886 4925 generic.go:356] "Generic (PLEG): container finished" podID="8bf40867-6019-4c8c-9488-b9ed3cff2efb" containerID="d352db5b2a093942154c4f6c978d81459212200c33195bfeff8d8fe75c903135" exitCode=0 Oct 30 04:03:20.366025 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:20.365956 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"8bf40867-6019-4c8c-9488-b9ed3cff2efb","Type":"ContainerDied","Data":"d352db5b2a093942154c4f6c978d81459212200c33195bfeff8d8fe75c903135"} Oct 30 04:03:21.373083 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:21.373042 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6","Type":"ContainerStarted","Data":"441ce4cab54f981a440cfb16563d0495bf5f1b200fefaaa267ccf27d09e1e08f"} Oct 30 04:03:21.375268 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:21.375240 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"8bf40867-6019-4c8c-9488-b9ed3cff2efb","Type":"ContainerStarted","Data":"f81733c84e59f8faea08940e57c0da664b647cc0bf6b2295738950944baa9ec7"} Oct 30 04:03:21.375366 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:21.375299 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"8bf40867-6019-4c8c-9488-b9ed3cff2efb","Type":"ContainerStarted","Data":"571a72bcf474a67ce31ee35b1cf1829a4ccae217b602704a1056a27f3f5a9c7b"} Oct 30 04:03:21.375503 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:21.375486 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:21.485870 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:21.485809 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=2.899894399 podStartE2EDuration="6.485796047s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.585901648s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:21.484874373 +0000 UTC m=+4962.498152479" watchObservedRunningTime="2026-10-30 04:03:21.485796047 +0000 UTC m=+4962.499074149" Oct 30 04:03:22.311945 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:22.311911 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" Oct 30 04:03:24.394422 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:24.394388 4925 generic.go:356] "Generic (PLEG): container finished" podID="7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" containerID="441ce4cab54f981a440cfb16563d0495bf5f1b200fefaaa267ccf27d09e1e08f" exitCode=0 Oct 30 04:03:24.394836 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:24.394434 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6","Type":"ContainerDied","Data":"441ce4cab54f981a440cfb16563d0495bf5f1b200fefaaa267ccf27d09e1e08f"} Oct 30 04:03:25.403112 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:25.403069 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6","Type":"ContainerStarted","Data":"1a81157ff602aada80fa0b341c16b4ce7d80e6a31c3871994bb329ce36654a27"} Oct 30 04:03:25.403496 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:25.403142 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6","Type":"ContainerStarted","Data":"ce9867f7ea0cf96ad1f0d0ed22f9fc203e472bf42a9792f6ecb23ebbd3e5e451"} Oct 30 04:03:25.403496 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:25.403167 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6","Type":"ContainerStarted","Data":"065d6459faefe34b553d474d02359f7c13b7cdb563a66a6400675997eebee473"} Oct 30 04:03:25.494429 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:25.494371 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=6.708761949 podStartE2EDuration="10.49435688s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.785594931s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:25.480556146 +0000 UTC m=+4966.493834249" watchObservedRunningTime="2026-10-30 04:03:25.49435688 +0000 UTC m=+4966.507634982" Oct 30 04:03:27.382988 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:27.382956 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:03:29.814676 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:29.814635 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:34.815497 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:34.815455 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:34.820784 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:34.820739 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:35.462270 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:35.462239 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:03:45.957882 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:45.957833 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:03:45.958738 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:45.958100 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="3603ba78-be34-44f4-92ee-3ad3f5b457bb" containerName="otc-container" containerID="cri-o://b89350aa023f79a8e52e807be2aa94a22f9a4c17a3f80e7c5d1b720ff0fbb2f0" gracePeriod=30 Oct 30 04:03:47.217202 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.217171 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:47.318624 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.318530 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3603ba78-be34-44f4-92ee-3ad3f5b457bb-kube-api-access-jg7gh\" (UniqueName: \"kubernetes.io/projected/3603ba78-be34-44f4-92ee-3ad3f5b457bb-kube-api-access-jg7gh\") pod \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\" (UID: \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\") " Oct 30 04:03:47.318624 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.318586 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3603ba78-be34-44f4-92ee-3ad3f5b457bb-otc-internal\" (UniqueName: \"kubernetes.io/configmap/3603ba78-be34-44f4-92ee-3ad3f5b457bb-otc-internal\") pod \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\" (UID: \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\") " Oct 30 04:03:47.318624 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.318609 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-monitoring-tls-certs\") pod \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\" (UID: \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\") " Oct 30 04:03:47.318966 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.318636 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-data-science-collector-ta-client-cert\") pod \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\" (UID: \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\") " Oct 30 04:03:47.318966 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.318679 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-tls-certs\" (UniqueName: \"kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-tls-certs\") pod \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\" (UID: \"3603ba78-be34-44f4-92ee-3ad3f5b457bb\") " Oct 30 04:03:47.318966 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.318942 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3603ba78-be34-44f4-92ee-3ad3f5b457bb-otc-internal" pod "3603ba78-be34-44f4-92ee-3ad3f5b457bb" (UID: "3603ba78-be34-44f4-92ee-3ad3f5b457bb"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:03:47.320995 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.320964 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-tls-certs" pod "3603ba78-be34-44f4-92ee-3ad3f5b457bb" (UID: "3603ba78-be34-44f4-92ee-3ad3f5b457bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:03:47.321131 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.321087 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-data-science-collector-ta-client-cert" pod "3603ba78-be34-44f4-92ee-3ad3f5b457bb" (UID: "3603ba78-be34-44f4-92ee-3ad3f5b457bb"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:03:47.321131 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.321091 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3603ba78-be34-44f4-92ee-3ad3f5b457bb-kube-api-access-jg7gh" pod "3603ba78-be34-44f4-92ee-3ad3f5b457bb" (UID: "3603ba78-be34-44f4-92ee-3ad3f5b457bb"). InnerVolumeSpecName "kube-api-access-jg7gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:03:47.321217 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.321135 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-monitoring-tls-certs" pod "3603ba78-be34-44f4-92ee-3ad3f5b457bb" (UID: "3603ba78-be34-44f4-92ee-3ad3f5b457bb"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:03:47.420106 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.420069 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg7gh\" (UniqueName: \"kubernetes.io/projected/3603ba78-be34-44f4-92ee-3ad3f5b457bb-kube-api-access-jg7gh\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:47.420106 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.420102 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3603ba78-be34-44f4-92ee-3ad3f5b457bb-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:47.420106 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.420114 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:47.420379 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.420124 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:47.420379 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.420134 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3603ba78-be34-44f4-92ee-3ad3f5b457bb-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:03:47.527045 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.527011 4925 generic.go:356] "Generic (PLEG): container finished" podID="3603ba78-be34-44f4-92ee-3ad3f5b457bb" containerID="b89350aa023f79a8e52e807be2aa94a22f9a4c17a3f80e7c5d1b720ff0fbb2f0" exitCode=0 Oct 30 04:03:47.527210 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.527067 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:47.527210 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.527096 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"3603ba78-be34-44f4-92ee-3ad3f5b457bb","Type":"ContainerDied","Data":"b89350aa023f79a8e52e807be2aa94a22f9a4c17a3f80e7c5d1b720ff0fbb2f0"} Oct 30 04:03:47.527210 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.527157 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"3603ba78-be34-44f4-92ee-3ad3f5b457bb","Type":"ContainerDied","Data":"2973ea1ca50f9f3cfa6e543993914d8b81c05d5d509e25b462d588ad6ed56ce7"} Oct 30 04:03:47.527210 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.527178 4925 scope.go:122] "RemoveContainer" containerID="b89350aa023f79a8e52e807be2aa94a22f9a4c17a3f80e7c5d1b720ff0fbb2f0" Oct 30 04:03:47.536925 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.536899 4925 scope.go:122] "RemoveContainer" containerID="b89350aa023f79a8e52e807be2aa94a22f9a4c17a3f80e7c5d1b720ff0fbb2f0" Oct 30 04:03:47.537260 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:03:47.537239 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89350aa023f79a8e52e807be2aa94a22f9a4c17a3f80e7c5d1b720ff0fbb2f0\": container with ID starting with b89350aa023f79a8e52e807be2aa94a22f9a4c17a3f80e7c5d1b720ff0fbb2f0 not found: ID does not exist" containerID="b89350aa023f79a8e52e807be2aa94a22f9a4c17a3f80e7c5d1b720ff0fbb2f0" Oct 30 04:03:47.537316 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.537271 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89350aa023f79a8e52e807be2aa94a22f9a4c17a3f80e7c5d1b720ff0fbb2f0"} err="failed to get container status \"b89350aa023f79a8e52e807be2aa94a22f9a4c17a3f80e7c5d1b720ff0fbb2f0\": rpc error: code = NotFound desc = could not find container \"b89350aa023f79a8e52e807be2aa94a22f9a4c17a3f80e7c5d1b720ff0fbb2f0\": container with ID starting with b89350aa023f79a8e52e807be2aa94a22f9a4c17a3f80e7c5d1b720ff0fbb2f0 not found: ID does not exist" Oct 30 04:03:47.579303 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.579260 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:03:47.591796 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.591761 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:03:47.619742 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.619704 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:03:47.620101 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.620088 4925 cpu_manager.go:414] "containerMap: removing container" podUID="3603ba78-be34-44f4-92ee-3ad3f5b457bb" containerName="otc-container" Oct 30 04:03:47.620154 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.620102 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3603ba78-be34-44f4-92ee-3ad3f5b457bb" containerName="otc-container" Oct 30 04:03:47.620198 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.620179 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="3603ba78-be34-44f4-92ee-3ad3f5b457bb" containerName="otc-container" Oct 30 04:03:47.624447 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.624421 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:47.639931 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.639904 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:03:47.640106 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.639955 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:03:47.640106 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.639963 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:03:47.640106 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.640005 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:03:47.640106 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.640060 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-45ch8\"" Oct 30 04:03:47.645624 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.645593 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:03:47.723422 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.723385 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"acc48730-cedf-4f09-9f31-59862e5d03c2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:47.723641 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.723452 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"acc48730-cedf-4f09-9f31-59862e5d03c2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:47.723641 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.723552 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6flj\" (UniqueName: \"kubernetes.io/projected/acc48730-cedf-4f09-9f31-59862e5d03c2-kube-api-access-n6flj\") pod \"data-science-collector-collector-0\" (UID: \"acc48730-cedf-4f09-9f31-59862e5d03c2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:47.723641 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.723595 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/acc48730-cedf-4f09-9f31-59862e5d03c2-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"acc48730-cedf-4f09-9f31-59862e5d03c2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:47.723801 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.723690 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"acc48730-cedf-4f09-9f31-59862e5d03c2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:47.824599 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.824561 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"acc48730-cedf-4f09-9f31-59862e5d03c2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:47.824846 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.824628 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"acc48730-cedf-4f09-9f31-59862e5d03c2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:47.824846 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.824691 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"acc48730-cedf-4f09-9f31-59862e5d03c2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:47.824846 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.824739 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6flj\" (UniqueName: \"kubernetes.io/projected/acc48730-cedf-4f09-9f31-59862e5d03c2-kube-api-access-n6flj\") pod \"data-science-collector-collector-0\" (UID: \"acc48730-cedf-4f09-9f31-59862e5d03c2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:47.824846 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.824796 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/acc48730-cedf-4f09-9f31-59862e5d03c2-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"acc48730-cedf-4f09-9f31-59862e5d03c2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:47.826256 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.826224 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/acc48730-cedf-4f09-9f31-59862e5d03c2-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"acc48730-cedf-4f09-9f31-59862e5d03c2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:47.828279 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.828245 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"acc48730-cedf-4f09-9f31-59862e5d03c2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:47.828399 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.828284 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"acc48730-cedf-4f09-9f31-59862e5d03c2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:47.835025 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.834969 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"acc48730-cedf-4f09-9f31-59862e5d03c2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:47.848620 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.848593 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6flj\" (UniqueName: \"kubernetes.io/projected/acc48730-cedf-4f09-9f31-59862e5d03c2-kube-api-access-n6flj\") pod \"data-science-collector-collector-0\" (UID: \"acc48730-cedf-4f09-9f31-59862e5d03c2\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:47.937270 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:47.937236 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:03:48.161858 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:03:48.161780 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:03:48.237637 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:48.237609 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:03:48.239995 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:03:48.239967 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc48730_cedf_4f09_9f31_59862e5d03c2.slice/crio-a529ff5694b71c561d1350d633fc76e4c1179092b51633d783d548f7037df358 WatchSource:0}: Error finding container a529ff5694b71c561d1350d633fc76e4c1179092b51633d783d548f7037df358: Status 404 returned error can't find the container with id a529ff5694b71c561d1350d633fc76e4c1179092b51633d783d548f7037df358 Oct 30 04:03:48.242250 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:48.242212 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:03:48.242324 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:48.242293 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:03:48.534111 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:48.534013 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"acc48730-cedf-4f09-9f31-59862e5d03c2","Type":"ContainerStarted","Data":"ddacb1fd51f2d6736d5f49e917e05926d95e696cb599b37555daef8dc8c7b71d"} Oct 30 04:03:48.534111 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:48.534075 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"acc48730-cedf-4f09-9f31-59862e5d03c2","Type":"ContainerStarted","Data":"a529ff5694b71c561d1350d633fc76e4c1179092b51633d783d548f7037df358"} Oct 30 04:03:48.579058 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:48.579000 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.578985019 podStartE2EDuration="1.578985019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:48.576863636 +0000 UTC m=+4989.590141742" watchObservedRunningTime="2026-10-30 04:03:48.578985019 +0000 UTC m=+4989.592263121" Oct 30 04:03:49.120221 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:03:49.120182 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3603ba78-be34-44f4-92ee-3ad3f5b457bb" path="/var/lib/kubelet/pods/3603ba78-be34-44f4-92ee-3ad3f5b457bb/volumes" Oct 30 04:04:06.323089 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:06.323015 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr"] Oct 30 04:04:06.326840 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:06.326819 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" Oct 30 04:04:06.346486 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:06.346461 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr"] Oct 30 04:04:06.352453 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:06.352427 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wzpv\" (UniqueName: \"kubernetes.io/projected/22b328d5-73b0-46a9-8284-325eedc27f8d-kube-api-access-9wzpv\") pod \"korrel8r-66d6f7f696-qkplr\" (UID: \"22b328d5-73b0-46a9-8284-325eedc27f8d\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" Oct 30 04:04:06.352580 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:06.352478 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/22b328d5-73b0-46a9-8284-325eedc27f8d-tls\") pod \"korrel8r-66d6f7f696-qkplr\" (UID: \"22b328d5-73b0-46a9-8284-325eedc27f8d\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" Oct 30 04:04:06.352634 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:06.352585 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/22b328d5-73b0-46a9-8284-325eedc27f8d-config\") pod \"korrel8r-66d6f7f696-qkplr\" (UID: \"22b328d5-73b0-46a9-8284-325eedc27f8d\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" Oct 30 04:04:06.453140 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:06.453098 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/22b328d5-73b0-46a9-8284-325eedc27f8d-tls\") pod \"korrel8r-66d6f7f696-qkplr\" (UID: \"22b328d5-73b0-46a9-8284-325eedc27f8d\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" Oct 30 04:04:06.453140 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:06.453147 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/22b328d5-73b0-46a9-8284-325eedc27f8d-config\") pod \"korrel8r-66d6f7f696-qkplr\" (UID: \"22b328d5-73b0-46a9-8284-325eedc27f8d\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" Oct 30 04:04:06.453355 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:06.453217 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wzpv\" (UniqueName: \"kubernetes.io/projected/22b328d5-73b0-46a9-8284-325eedc27f8d-kube-api-access-9wzpv\") pod \"korrel8r-66d6f7f696-qkplr\" (UID: \"22b328d5-73b0-46a9-8284-325eedc27f8d\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" Oct 30 04:04:06.454016 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:06.453988 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/22b328d5-73b0-46a9-8284-325eedc27f8d-config\") pod \"korrel8r-66d6f7f696-qkplr\" (UID: \"22b328d5-73b0-46a9-8284-325eedc27f8d\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" Oct 30 04:04:06.455599 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:06.455577 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/22b328d5-73b0-46a9-8284-325eedc27f8d-tls\") pod \"korrel8r-66d6f7f696-qkplr\" (UID: \"22b328d5-73b0-46a9-8284-325eedc27f8d\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" Oct 30 04:04:06.476596 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:06.476572 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wzpv\" (UniqueName: \"kubernetes.io/projected/22b328d5-73b0-46a9-8284-325eedc27f8d-kube-api-access-9wzpv\") pod \"korrel8r-66d6f7f696-qkplr\" (UID: \"22b328d5-73b0-46a9-8284-325eedc27f8d\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" Oct 30 04:04:06.638738 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:06.638695 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" Oct 30 04:04:06.940267 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:06.940240 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr"] Oct 30 04:04:06.942296 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:04:06.942264 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b328d5_73b0_46a9_8284_325eedc27f8d.slice/crio-2a9b05d19965139c3d5ca0a9d582ed4954207b4ec660b0e2aa1f71151f71a171 WatchSource:0}: Error finding container 2a9b05d19965139c3d5ca0a9d582ed4954207b4ec660b0e2aa1f71151f71a171: Status 404 returned error can't find the container with id 2a9b05d19965139c3d5ca0a9d582ed4954207b4ec660b0e2aa1f71151f71a171 Oct 30 04:04:07.631650 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:07.631613 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" event={"ID":"22b328d5-73b0-46a9-8284-325eedc27f8d","Type":"ContainerStarted","Data":"ff29d7bacc9e8039169430ee66f731004b559eeaf979a25459c6c4472512cf68"} Oct 30 04:04:07.632081 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:07.631773 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" Oct 30 04:04:07.632081 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:07.631821 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" event={"ID":"22b328d5-73b0-46a9-8284-325eedc27f8d","Type":"ContainerStarted","Data":"2a9b05d19965139c3d5ca0a9d582ed4954207b4ec660b0e2aa1f71151f71a171"} Oct 30 04:04:07.700898 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:07.700841 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" podStartSLOduration=1.700828387 podStartE2EDuration="1.700828387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:07.700396815 +0000 UTC m=+5008.713674915" watchObservedRunningTime="2026-10-30 04:04:07.700828387 +0000 UTC m=+5008.714106490" Oct 30 04:04:12.482543 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:12.482501 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:12.483075 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:12.482777 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="acc48730-cedf-4f09-9f31-59862e5d03c2" containerName="otc-container" containerID="cri-o://ddacb1fd51f2d6736d5f49e917e05926d95e696cb599b37555daef8dc8c7b71d" gracePeriod=30 Oct 30 04:04:14.830412 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:14.830387 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:14.917347 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:14.917268 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-data-science-collector-ta-client-cert\") pod \"acc48730-cedf-4f09-9f31-59862e5d03c2\" (UID: \"acc48730-cedf-4f09-9f31-59862e5d03c2\") " Oct 30 04:04:14.917347 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:14.917304 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acc48730-cedf-4f09-9f31-59862e5d03c2-otc-internal\" (UniqueName: \"kubernetes.io/configmap/acc48730-cedf-4f09-9f31-59862e5d03c2-otc-internal\") pod \"acc48730-cedf-4f09-9f31-59862e5d03c2\" (UID: \"acc48730-cedf-4f09-9f31-59862e5d03c2\") " Oct 30 04:04:14.917347 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:14.917346 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-tls-certs\" (UniqueName: \"kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-tls-certs\") pod \"acc48730-cedf-4f09-9f31-59862e5d03c2\" (UID: \"acc48730-cedf-4f09-9f31-59862e5d03c2\") " Oct 30 04:04:14.917717 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:14.917691 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc48730-cedf-4f09-9f31-59862e5d03c2-otc-internal" pod "acc48730-cedf-4f09-9f31-59862e5d03c2" (UID: "acc48730-cedf-4f09-9f31-59862e5d03c2"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:14.919397 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:14.919372 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-data-science-collector-ta-client-cert" pod "acc48730-cedf-4f09-9f31-59862e5d03c2" (UID: "acc48730-cedf-4f09-9f31-59862e5d03c2"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:14.919518 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:14.919478 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-tls-certs" pod "acc48730-cedf-4f09-9f31-59862e5d03c2" (UID: "acc48730-cedf-4f09-9f31-59862e5d03c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:15.017838 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.017797 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-monitoring-tls-certs\") pod \"acc48730-cedf-4f09-9f31-59862e5d03c2\" (UID: \"acc48730-cedf-4f09-9f31-59862e5d03c2\") " Oct 30 04:04:15.018019 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.017844 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc48730-cedf-4f09-9f31-59862e5d03c2-kube-api-access-n6flj\" (UniqueName: \"kubernetes.io/projected/acc48730-cedf-4f09-9f31-59862e5d03c2-kube-api-access-n6flj\") pod \"acc48730-cedf-4f09-9f31-59862e5d03c2\" (UID: \"acc48730-cedf-4f09-9f31-59862e5d03c2\") " Oct 30 04:04:15.018091 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.018064 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:15.018091 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.018084 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:15.018176 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.018099 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/acc48730-cedf-4f09-9f31-59862e5d03c2-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:15.019868 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.019846 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc48730-cedf-4f09-9f31-59862e5d03c2-kube-api-access-n6flj" pod "acc48730-cedf-4f09-9f31-59862e5d03c2" (UID: "acc48730-cedf-4f09-9f31-59862e5d03c2"). InnerVolumeSpecName "kube-api-access-n6flj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:15.019936 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.019863 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-monitoring-tls-certs" pod "acc48730-cedf-4f09-9f31-59862e5d03c2" (UID: "acc48730-cedf-4f09-9f31-59862e5d03c2"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:15.118561 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.118530 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/acc48730-cedf-4f09-9f31-59862e5d03c2-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:15.118561 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.118558 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6flj\" (UniqueName: \"kubernetes.io/projected/acc48730-cedf-4f09-9f31-59862e5d03c2-kube-api-access-n6flj\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:15.201455 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.201390 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr"] Oct 30 04:04:15.201592 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.201572 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" podUID="22b328d5-73b0-46a9-8284-325eedc27f8d" containerName="korrel8r" containerID="cri-o://ff29d7bacc9e8039169430ee66f731004b559eeaf979a25459c6c4472512cf68" gracePeriod=30 Oct 30 04:04:15.203213 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.203187 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" Oct 30 04:04:15.436968 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.436943 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" Oct 30 04:04:15.520665 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.520586 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22b328d5-73b0-46a9-8284-325eedc27f8d-tls\" (UniqueName: \"kubernetes.io/secret/22b328d5-73b0-46a9-8284-325eedc27f8d-tls\") pod \"22b328d5-73b0-46a9-8284-325eedc27f8d\" (UID: \"22b328d5-73b0-46a9-8284-325eedc27f8d\") " Oct 30 04:04:15.520665 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.520643 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22b328d5-73b0-46a9-8284-325eedc27f8d-config\" (UniqueName: \"kubernetes.io/configmap/22b328d5-73b0-46a9-8284-325eedc27f8d-config\") pod \"22b328d5-73b0-46a9-8284-325eedc27f8d\" (UID: \"22b328d5-73b0-46a9-8284-325eedc27f8d\") " Oct 30 04:04:15.520885 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.520702 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b328d5-73b0-46a9-8284-325eedc27f8d-kube-api-access-9wzpv\" (UniqueName: \"kubernetes.io/projected/22b328d5-73b0-46a9-8284-325eedc27f8d-kube-api-access-9wzpv\") pod \"22b328d5-73b0-46a9-8284-325eedc27f8d\" (UID: \"22b328d5-73b0-46a9-8284-325eedc27f8d\") " Oct 30 04:04:15.521036 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.520975 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b328d5-73b0-46a9-8284-325eedc27f8d-config" pod "22b328d5-73b0-46a9-8284-325eedc27f8d" (UID: "22b328d5-73b0-46a9-8284-325eedc27f8d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:15.522868 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.522847 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b328d5-73b0-46a9-8284-325eedc27f8d-tls" pod "22b328d5-73b0-46a9-8284-325eedc27f8d" (UID: "22b328d5-73b0-46a9-8284-325eedc27f8d"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:15.522868 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.522859 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b328d5-73b0-46a9-8284-325eedc27f8d-kube-api-access-9wzpv" pod "22b328d5-73b0-46a9-8284-325eedc27f8d" (UID: "22b328d5-73b0-46a9-8284-325eedc27f8d"). InnerVolumeSpecName "kube-api-access-9wzpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:15.621633 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.621605 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/22b328d5-73b0-46a9-8284-325eedc27f8d-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:15.621633 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.621631 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/22b328d5-73b0-46a9-8284-325eedc27f8d-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:15.621826 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.621641 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wzpv\" (UniqueName: \"kubernetes.io/projected/22b328d5-73b0-46a9-8284-325eedc27f8d-kube-api-access-9wzpv\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:15.672663 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.672629 4925 generic.go:356] "Generic (PLEG): container finished" podID="22b328d5-73b0-46a9-8284-325eedc27f8d" containerID="ff29d7bacc9e8039169430ee66f731004b559eeaf979a25459c6c4472512cf68" exitCode=0 Oct 30 04:04:15.672798 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.672677 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" Oct 30 04:04:15.672798 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.672723 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" event={"ID":"22b328d5-73b0-46a9-8284-325eedc27f8d","Type":"ContainerDied","Data":"ff29d7bacc9e8039169430ee66f731004b559eeaf979a25459c6c4472512cf68"} Oct 30 04:04:15.672920 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.672803 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr" event={"ID":"22b328d5-73b0-46a9-8284-325eedc27f8d","Type":"ContainerDied","Data":"2a9b05d19965139c3d5ca0a9d582ed4954207b4ec660b0e2aa1f71151f71a171"} Oct 30 04:04:15.672920 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.672833 4925 scope.go:122] "RemoveContainer" containerID="ff29d7bacc9e8039169430ee66f731004b559eeaf979a25459c6c4472512cf68" Oct 30 04:04:15.674367 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.674341 4925 generic.go:356] "Generic (PLEG): container finished" podID="acc48730-cedf-4f09-9f31-59862e5d03c2" containerID="ddacb1fd51f2d6736d5f49e917e05926d95e696cb599b37555daef8dc8c7b71d" exitCode=0 Oct 30 04:04:15.674468 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.674388 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:15.674468 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.674432 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"acc48730-cedf-4f09-9f31-59862e5d03c2","Type":"ContainerDied","Data":"ddacb1fd51f2d6736d5f49e917e05926d95e696cb599b37555daef8dc8c7b71d"} Oct 30 04:04:15.674578 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.674488 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"acc48730-cedf-4f09-9f31-59862e5d03c2","Type":"ContainerDied","Data":"a529ff5694b71c561d1350d633fc76e4c1179092b51633d783d548f7037df358"} Oct 30 04:04:15.682665 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.682649 4925 scope.go:122] "RemoveContainer" containerID="ff29d7bacc9e8039169430ee66f731004b559eeaf979a25459c6c4472512cf68" Oct 30 04:04:15.683009 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:04:15.682990 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff29d7bacc9e8039169430ee66f731004b559eeaf979a25459c6c4472512cf68\": container with ID starting with ff29d7bacc9e8039169430ee66f731004b559eeaf979a25459c6c4472512cf68 not found: ID does not exist" containerID="ff29d7bacc9e8039169430ee66f731004b559eeaf979a25459c6c4472512cf68" Oct 30 04:04:15.683060 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.683016 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff29d7bacc9e8039169430ee66f731004b559eeaf979a25459c6c4472512cf68"} err="failed to get container status \"ff29d7bacc9e8039169430ee66f731004b559eeaf979a25459c6c4472512cf68\": rpc error: code = NotFound desc = could not find container \"ff29d7bacc9e8039169430ee66f731004b559eeaf979a25459c6c4472512cf68\": container with ID starting with ff29d7bacc9e8039169430ee66f731004b559eeaf979a25459c6c4472512cf68 not found: ID does not exist" Oct 30 04:04:15.683060 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.683031 4925 scope.go:122] "RemoveContainer" containerID="ddacb1fd51f2d6736d5f49e917e05926d95e696cb599b37555daef8dc8c7b71d" Oct 30 04:04:15.690716 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.690696 4925 scope.go:122] "RemoveContainer" containerID="ddacb1fd51f2d6736d5f49e917e05926d95e696cb599b37555daef8dc8c7b71d" Oct 30 04:04:15.690972 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:04:15.690954 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddacb1fd51f2d6736d5f49e917e05926d95e696cb599b37555daef8dc8c7b71d\": container with ID starting with ddacb1fd51f2d6736d5f49e917e05926d95e696cb599b37555daef8dc8c7b71d not found: ID does not exist" containerID="ddacb1fd51f2d6736d5f49e917e05926d95e696cb599b37555daef8dc8c7b71d" Oct 30 04:04:15.691061 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.690977 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddacb1fd51f2d6736d5f49e917e05926d95e696cb599b37555daef8dc8c7b71d"} err="failed to get container status \"ddacb1fd51f2d6736d5f49e917e05926d95e696cb599b37555daef8dc8c7b71d\": rpc error: code = NotFound desc = could not find container \"ddacb1fd51f2d6736d5f49e917e05926d95e696cb599b37555daef8dc8c7b71d\": container with ID starting with ddacb1fd51f2d6736d5f49e917e05926d95e696cb599b37555daef8dc8c7b71d not found: ID does not exist" Oct 30 04:04:15.724602 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.724576 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:15.728971 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.728950 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:15.764689 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.764664 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr"] Oct 30 04:04:15.767813 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.767795 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:15.768148 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.768137 4925 cpu_manager.go:414] "containerMap: removing container" podUID="22b328d5-73b0-46a9-8284-325eedc27f8d" containerName="korrel8r" Oct 30 04:04:15.768199 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.768149 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b328d5-73b0-46a9-8284-325eedc27f8d" containerName="korrel8r" Oct 30 04:04:15.768199 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.768162 4925 cpu_manager.go:414] "containerMap: removing container" podUID="acc48730-cedf-4f09-9f31-59862e5d03c2" containerName="otc-container" Oct 30 04:04:15.768199 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.768167 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acc48730-cedf-4f09-9f31-59862e5d03c2" containerName="otc-container" Oct 30 04:04:15.768291 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.768245 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="22b328d5-73b0-46a9-8284-325eedc27f8d" containerName="korrel8r" Oct 30 04:04:15.768291 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.768252 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="acc48730-cedf-4f09-9f31-59862e5d03c2" containerName="otc-container" Oct 30 04:04:15.772630 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.772567 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:15.776122 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.776096 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-qkplr"] Oct 30 04:04:15.791579 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.791557 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:04:15.791693 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.791659 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-72c79cac\"" Oct 30 04:04:15.791777 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.791728 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-45ch8\"" Oct 30 04:04:15.791842 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.791817 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:04:15.791957 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.791944 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:04:15.793797 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.793774 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:15.823666 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.823633 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:15.823826 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.823688 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/51851878-3d4b-4c44-9b9f-e6c35f6bde01-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:15.823826 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.823742 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8x2f\" (UniqueName: \"kubernetes.io/projected/51851878-3d4b-4c44-9b9f-e6c35f6bde01-kube-api-access-m8x2f\") pod \"data-science-collector-collector-0\" (UID: \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:15.823928 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.823873 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:15.823928 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.823904 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:15.924556 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.924522 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:15.924933 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.924566 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:15.924933 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.924589 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:15.924933 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.924618 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/51851878-3d4b-4c44-9b9f-e6c35f6bde01-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:15.924933 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.924633 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8x2f\" (UniqueName: \"kubernetes.io/projected/51851878-3d4b-4c44-9b9f-e6c35f6bde01-kube-api-access-m8x2f\") pod \"data-science-collector-collector-0\" (UID: \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:15.925381 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.925358 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/51851878-3d4b-4c44-9b9f-e6c35f6bde01-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:15.927018 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.926990 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:15.927128 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.927075 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:15.927128 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.927080 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:15.947917 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:15.947897 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8x2f\" (UniqueName: \"kubernetes.io/projected/51851878-3d4b-4c44-9b9f-e6c35f6bde01-kube-api-access-m8x2f\") pod \"data-science-collector-collector-0\" (UID: \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:16.089814 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:16.089786 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:16.389523 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:16.389497 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:16.396696 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:16.396661 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:16.396810 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:16.396739 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:16.683650 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:16.683575 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"51851878-3d4b-4c44-9b9f-e6c35f6bde01","Type":"ContainerStarted","Data":"992d5dde1d92885417260b76dc884e290d20d04b8de79ef61a04219b33c7cc58"} Oct 30 04:04:16.683650 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:16.683640 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"51851878-3d4b-4c44-9b9f-e6c35f6bde01","Type":"ContainerStarted","Data":"babffd25ba56feac9401b03ae6aef38d9e22ee17fd8099bb265d50da34f9ca66"} Oct 30 04:04:16.748054 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:16.747917 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.74789826 podStartE2EDuration="1.74789826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:16.746399975 +0000 UTC m=+5017.759678094" watchObservedRunningTime="2026-10-30 04:04:16.74789826 +0000 UTC m=+5017.761176364" Oct 30 04:04:17.119927 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:17.119892 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22b328d5-73b0-46a9-8284-325eedc27f8d" path="/var/lib/kubelet/pods/22b328d5-73b0-46a9-8284-325eedc27f8d/volumes" Oct 30 04:04:17.120303 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:17.120277 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acc48730-cedf-4f09-9f31-59862e5d03c2" path="/var/lib/kubelet/pods/acc48730-cedf-4f09-9f31-59862e5d03c2/volumes" Oct 30 04:04:18.163645 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:04:18.163605 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:04:18.556139 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:18.556046 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:18.694471 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:18.694432 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="51851878-3d4b-4c44-9b9f-e6c35f6bde01" containerName="otc-container" containerID="cri-o://992d5dde1d92885417260b76dc884e290d20d04b8de79ef61a04219b33c7cc58" gracePeriod=30 Oct 30 04:04:18.955489 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:18.955460 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:19.048497 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.048463 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51851878-3d4b-4c44-9b9f-e6c35f6bde01-kube-api-access-m8x2f\" (UniqueName: \"kubernetes.io/projected/51851878-3d4b-4c44-9b9f-e6c35f6bde01-kube-api-access-m8x2f\") pod \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\" (UID: \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\") " Oct 30 04:04:19.048698 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.048517 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-data-science-collector-ta-client-cert\") pod \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\" (UID: \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\") " Oct 30 04:04:19.048698 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.048543 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51851878-3d4b-4c44-9b9f-e6c35f6bde01-otc-internal\" (UniqueName: \"kubernetes.io/configmap/51851878-3d4b-4c44-9b9f-e6c35f6bde01-otc-internal\") pod \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\" (UID: \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\") " Oct 30 04:04:19.048698 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.048596 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-monitoring-tls-certs\") pod \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\" (UID: \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\") " Oct 30 04:04:19.048698 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.048615 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-tls-certs\" (UniqueName: \"kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-tls-certs\") pod \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\" (UID: \"51851878-3d4b-4c44-9b9f-e6c35f6bde01\") " Oct 30 04:04:19.049037 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.048992 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51851878-3d4b-4c44-9b9f-e6c35f6bde01-otc-internal" pod "51851878-3d4b-4c44-9b9f-e6c35f6bde01" (UID: "51851878-3d4b-4c44-9b9f-e6c35f6bde01"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:19.050921 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.050886 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-tls-certs" pod "51851878-3d4b-4c44-9b9f-e6c35f6bde01" (UID: "51851878-3d4b-4c44-9b9f-e6c35f6bde01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:19.051027 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.050894 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-monitoring-tls-certs" pod "51851878-3d4b-4c44-9b9f-e6c35f6bde01" (UID: "51851878-3d4b-4c44-9b9f-e6c35f6bde01"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:19.051027 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.050996 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-data-science-collector-ta-client-cert" pod "51851878-3d4b-4c44-9b9f-e6c35f6bde01" (UID: "51851878-3d4b-4c44-9b9f-e6c35f6bde01"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:19.051097 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.051029 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51851878-3d4b-4c44-9b9f-e6c35f6bde01-kube-api-access-m8x2f" pod "51851878-3d4b-4c44-9b9f-e6c35f6bde01" (UID: "51851878-3d4b-4c44-9b9f-e6c35f6bde01"). InnerVolumeSpecName "kube-api-access-m8x2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:19.149902 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.149866 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:19.149902 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.149900 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:19.149902 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.149913 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8x2f\" (UniqueName: \"kubernetes.io/projected/51851878-3d4b-4c44-9b9f-e6c35f6bde01-kube-api-access-m8x2f\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:19.150214 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.149924 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/51851878-3d4b-4c44-9b9f-e6c35f6bde01-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:19.150214 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.149933 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/51851878-3d4b-4c44-9b9f-e6c35f6bde01-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:19.707921 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.707887 4925 generic.go:356] "Generic (PLEG): container finished" podID="51851878-3d4b-4c44-9b9f-e6c35f6bde01" containerID="992d5dde1d92885417260b76dc884e290d20d04b8de79ef61a04219b33c7cc58" exitCode=0 Oct 30 04:04:19.708316 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.708002 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"51851878-3d4b-4c44-9b9f-e6c35f6bde01","Type":"ContainerDied","Data":"992d5dde1d92885417260b76dc884e290d20d04b8de79ef61a04219b33c7cc58"} Oct 30 04:04:19.708316 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.708012 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:19.708316 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.708071 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"51851878-3d4b-4c44-9b9f-e6c35f6bde01","Type":"ContainerDied","Data":"babffd25ba56feac9401b03ae6aef38d9e22ee17fd8099bb265d50da34f9ca66"} Oct 30 04:04:19.708316 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.708101 4925 scope.go:122] "RemoveContainer" containerID="992d5dde1d92885417260b76dc884e290d20d04b8de79ef61a04219b33c7cc58" Oct 30 04:04:19.718532 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.718512 4925 scope.go:122] "RemoveContainer" containerID="992d5dde1d92885417260b76dc884e290d20d04b8de79ef61a04219b33c7cc58" Oct 30 04:04:19.718825 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:04:19.718805 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"992d5dde1d92885417260b76dc884e290d20d04b8de79ef61a04219b33c7cc58\": container with ID starting with 992d5dde1d92885417260b76dc884e290d20d04b8de79ef61a04219b33c7cc58 not found: ID does not exist" containerID="992d5dde1d92885417260b76dc884e290d20d04b8de79ef61a04219b33c7cc58" Oct 30 04:04:19.718903 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.718831 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"992d5dde1d92885417260b76dc884e290d20d04b8de79ef61a04219b33c7cc58"} err="failed to get container status \"992d5dde1d92885417260b76dc884e290d20d04b8de79ef61a04219b33c7cc58\": rpc error: code = NotFound desc = could not find container \"992d5dde1d92885417260b76dc884e290d20d04b8de79ef61a04219b33c7cc58\": container with ID starting with 992d5dde1d92885417260b76dc884e290d20d04b8de79ef61a04219b33c7cc58 not found: ID does not exist" Oct 30 04:04:19.761111 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.761075 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:19.772767 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.772727 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:19.802282 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.802257 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:19.802623 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.802611 4925 cpu_manager.go:414] "containerMap: removing container" podUID="51851878-3d4b-4c44-9b9f-e6c35f6bde01" containerName="otc-container" Oct 30 04:04:19.802664 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.802624 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51851878-3d4b-4c44-9b9f-e6c35f6bde01" containerName="otc-container" Oct 30 04:04:19.802699 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.802692 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="51851878-3d4b-4c44-9b9f-e6c35f6bde01" containerName="otc-container" Oct 30 04:04:19.807444 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.807423 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:19.825141 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.825114 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:04:19.825330 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.825243 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:04:19.825330 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.825325 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-45ch8\"" Oct 30 04:04:19.825458 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.825435 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:04:19.825512 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.825490 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:04:19.827737 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.827713 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:19.855853 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.855819 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"213e6521-9211-4568-8645-6136c94b158b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:19.855853 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.855851 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"213e6521-9211-4568-8645-6136c94b158b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:19.856043 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.855874 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/213e6521-9211-4568-8645-6136c94b158b-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"213e6521-9211-4568-8645-6136c94b158b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:19.856043 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.855995 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcm85\" (UniqueName: \"kubernetes.io/projected/213e6521-9211-4568-8645-6136c94b158b-kube-api-access-xcm85\") pod \"data-science-collector-collector-0\" (UID: \"213e6521-9211-4568-8645-6136c94b158b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:19.856139 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.856073 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"213e6521-9211-4568-8645-6136c94b158b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:19.956932 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.956903 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"213e6521-9211-4568-8645-6136c94b158b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:19.957046 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.956938 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"213e6521-9211-4568-8645-6136c94b158b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:19.957046 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.956960 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/213e6521-9211-4568-8645-6136c94b158b-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"213e6521-9211-4568-8645-6136c94b158b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:19.957046 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.957003 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcm85\" (UniqueName: \"kubernetes.io/projected/213e6521-9211-4568-8645-6136c94b158b-kube-api-access-xcm85\") pod \"data-science-collector-collector-0\" (UID: \"213e6521-9211-4568-8645-6136c94b158b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:19.957222 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.957047 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"213e6521-9211-4568-8645-6136c94b158b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:19.957707 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.957683 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/213e6521-9211-4568-8645-6136c94b158b-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"213e6521-9211-4568-8645-6136c94b158b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:19.959460 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.959402 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"213e6521-9211-4568-8645-6136c94b158b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:19.959460 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.959404 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"213e6521-9211-4568-8645-6136c94b158b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:19.959460 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.959441 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"213e6521-9211-4568-8645-6136c94b158b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:19.980776 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:19.980741 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcm85\" (UniqueName: \"kubernetes.io/projected/213e6521-9211-4568-8645-6136c94b158b-kube-api-access-xcm85\") pod \"data-science-collector-collector-0\" (UID: \"213e6521-9211-4568-8645-6136c94b158b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:20.121655 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:20.121623 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:20.420610 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:20.420584 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:20.422524 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:04:20.422491 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213e6521_9211_4568_8645_6136c94b158b.slice/crio-3006d4153550b6e6fd5d7fd0804a444735e58c0438d65531ce8ad513ee50953d WatchSource:0}: Error finding container 3006d4153550b6e6fd5d7fd0804a444735e58c0438d65531ce8ad513ee50953d: Status 404 returned error can't find the container with id 3006d4153550b6e6fd5d7fd0804a444735e58c0438d65531ce8ad513ee50953d Oct 30 04:04:20.424712 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:20.424680 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:20.424831 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:20.424738 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:20.715266 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:20.715166 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"213e6521-9211-4568-8645-6136c94b158b","Type":"ContainerStarted","Data":"51a0b14e1f0b22cae8d4565351a84c6bc404f05d95e5db42a5760ba423cb8bb1"} Oct 30 04:04:20.715266 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:20.715242 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"213e6521-9211-4568-8645-6136c94b158b","Type":"ContainerStarted","Data":"3006d4153550b6e6fd5d7fd0804a444735e58c0438d65531ce8ad513ee50953d"} Oct 30 04:04:20.758962 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:20.758909 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.758896359 podStartE2EDuration="1.758896359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:20.757977711 +0000 UTC m=+5021.771255824" watchObservedRunningTime="2026-10-30 04:04:20.758896359 +0000 UTC m=+5021.772174462" Oct 30 04:04:21.119689 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:21.119656 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51851878-3d4b-4c44-9b9f-e6c35f6bde01" path="/var/lib/kubelet/pods/51851878-3d4b-4c44-9b9f-e6c35f6bde01/volumes" Oct 30 04:04:24.288908 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:24.288876 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk"] Oct 30 04:04:24.296550 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:24.293637 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" Oct 30 04:04:24.312494 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:24.312469 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk"] Oct 30 04:04:24.397993 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:24.397961 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfjw5\" (UniqueName: \"kubernetes.io/projected/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-kube-api-access-mfjw5\") pod \"korrel8r-66d6f7f696-bbmzk\" (UID: \"d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" Oct 30 04:04:24.398137 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:24.398000 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-config\") pod \"korrel8r-66d6f7f696-bbmzk\" (UID: \"d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" Oct 30 04:04:24.398137 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:24.398025 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-tls\") pod \"korrel8r-66d6f7f696-bbmzk\" (UID: \"d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" Oct 30 04:04:24.498923 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:24.498891 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-config\") pod \"korrel8r-66d6f7f696-bbmzk\" (UID: \"d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" Oct 30 04:04:24.498923 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:24.498925 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-tls\") pod \"korrel8r-66d6f7f696-bbmzk\" (UID: \"d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" Oct 30 04:04:24.499094 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:24.498994 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfjw5\" (UniqueName: \"kubernetes.io/projected/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-kube-api-access-mfjw5\") pod \"korrel8r-66d6f7f696-bbmzk\" (UID: \"d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" Oct 30 04:04:24.499844 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:24.499820 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-config\") pod \"korrel8r-66d6f7f696-bbmzk\" (UID: \"d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" Oct 30 04:04:24.501383 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:24.501361 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-tls\") pod \"korrel8r-66d6f7f696-bbmzk\" (UID: \"d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" Oct 30 04:04:24.522632 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:24.522602 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfjw5\" (UniqueName: \"kubernetes.io/projected/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-kube-api-access-mfjw5\") pod \"korrel8r-66d6f7f696-bbmzk\" (UID: \"d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" Oct 30 04:04:24.609683 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:24.609658 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" Oct 30 04:04:24.909980 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:24.909955 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk"] Oct 30 04:04:24.912071 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:04:24.912041 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f6c760_d2d0_4b18_ad0a_0fa60ea7442c.slice/crio-823bd528b4300709d0e04da957d9162d08c0b8d87848cae58da0ff09c9cd9a47 WatchSource:0}: Error finding container 823bd528b4300709d0e04da957d9162d08c0b8d87848cae58da0ff09c9cd9a47: Status 404 returned error can't find the container with id 823bd528b4300709d0e04da957d9162d08c0b8d87848cae58da0ff09c9cd9a47 Oct 30 04:04:25.743647 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:25.743612 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" event={"ID":"d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c","Type":"ContainerStarted","Data":"42e7e9bab5cfbad4ec9758be77634ae26ca37d08d9cf3a0e3306d6d4c412406b"} Oct 30 04:04:25.744066 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:25.743666 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" event={"ID":"d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c","Type":"ContainerStarted","Data":"823bd528b4300709d0e04da957d9162d08c0b8d87848cae58da0ff09c9cd9a47"} Oct 30 04:04:25.744066 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:25.743694 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" Oct 30 04:04:25.786176 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:25.786128 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" podStartSLOduration=1.786115299 podStartE2EDuration="1.786115299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:25.785069451 +0000 UTC m=+5026.798347556" watchObservedRunningTime="2026-10-30 04:04:25.786115299 +0000 UTC m=+5026.799393435" Oct 30 04:04:29.583571 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:29.583537 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:29.584066 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:29.583732 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="213e6521-9211-4568-8645-6136c94b158b" containerName="otc-container" containerID="cri-o://51a0b14e1f0b22cae8d4565351a84c6bc404f05d95e5db42a5760ba423cb8bb1" gracePeriod=30 Oct 30 04:04:29.770132 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:29.770097 4925 generic.go:356] "Generic (PLEG): container finished" podID="213e6521-9211-4568-8645-6136c94b158b" containerID="51a0b14e1f0b22cae8d4565351a84c6bc404f05d95e5db42a5760ba423cb8bb1" exitCode=0 Oct 30 04:04:29.770259 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:29.770160 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"213e6521-9211-4568-8645-6136c94b158b","Type":"ContainerDied","Data":"51a0b14e1f0b22cae8d4565351a84c6bc404f05d95e5db42a5760ba423cb8bb1"} Oct 30 04:04:29.831885 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:29.831861 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:29.946806 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:29.946703 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-tls-certs\" (UniqueName: \"kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-tls-certs\") pod \"213e6521-9211-4568-8645-6136c94b158b\" (UID: \"213e6521-9211-4568-8645-6136c94b158b\") " Oct 30 04:04:29.946806 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:29.946742 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/213e6521-9211-4568-8645-6136c94b158b-otc-internal\" (UniqueName: \"kubernetes.io/configmap/213e6521-9211-4568-8645-6136c94b158b-otc-internal\") pod \"213e6521-9211-4568-8645-6136c94b158b\" (UID: \"213e6521-9211-4568-8645-6136c94b158b\") " Oct 30 04:04:29.946806 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:29.946803 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/213e6521-9211-4568-8645-6136c94b158b-kube-api-access-xcm85\" (UniqueName: \"kubernetes.io/projected/213e6521-9211-4568-8645-6136c94b158b-kube-api-access-xcm85\") pod \"213e6521-9211-4568-8645-6136c94b158b\" (UID: \"213e6521-9211-4568-8645-6136c94b158b\") " Oct 30 04:04:29.947087 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:29.946828 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-monitoring-tls-certs\") pod \"213e6521-9211-4568-8645-6136c94b158b\" (UID: \"213e6521-9211-4568-8645-6136c94b158b\") " Oct 30 04:04:29.947087 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:29.946858 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-data-science-collector-ta-client-cert\") pod \"213e6521-9211-4568-8645-6136c94b158b\" (UID: \"213e6521-9211-4568-8645-6136c94b158b\") " Oct 30 04:04:29.947243 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:29.947203 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213e6521-9211-4568-8645-6136c94b158b-otc-internal" pod "213e6521-9211-4568-8645-6136c94b158b" (UID: "213e6521-9211-4568-8645-6136c94b158b"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:29.949033 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:29.948997 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-monitoring-tls-certs" pod "213e6521-9211-4568-8645-6136c94b158b" (UID: "213e6521-9211-4568-8645-6136c94b158b"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:29.949161 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:29.949052 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-tls-certs" pod "213e6521-9211-4568-8645-6136c94b158b" (UID: "213e6521-9211-4568-8645-6136c94b158b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:29.949333 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:29.949312 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213e6521-9211-4568-8645-6136c94b158b-kube-api-access-xcm85" pod "213e6521-9211-4568-8645-6136c94b158b" (UID: "213e6521-9211-4568-8645-6136c94b158b"). InnerVolumeSpecName "kube-api-access-xcm85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:29.949414 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:29.949324 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-data-science-collector-ta-client-cert" pod "213e6521-9211-4568-8645-6136c94b158b" (UID: "213e6521-9211-4568-8645-6136c94b158b"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:30.047840 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.047807 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:30.047840 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.047836 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/213e6521-9211-4568-8645-6136c94b158b-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:30.047840 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.047845 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcm85\" (UniqueName: \"kubernetes.io/projected/213e6521-9211-4568-8645-6136c94b158b-kube-api-access-xcm85\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:30.048072 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.047854 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:30.048072 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.047864 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/213e6521-9211-4568-8645-6136c94b158b-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:30.777213 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.777180 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:30.777638 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.777227 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"213e6521-9211-4568-8645-6136c94b158b","Type":"ContainerDied","Data":"3006d4153550b6e6fd5d7fd0804a444735e58c0438d65531ce8ad513ee50953d"} Oct 30 04:04:30.777638 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.777291 4925 scope.go:122] "RemoveContainer" containerID="51a0b14e1f0b22cae8d4565351a84c6bc404f05d95e5db42a5760ba423cb8bb1" Oct 30 04:04:30.829582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.829553 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:30.842318 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.842292 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:30.871773 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.871735 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:30.872151 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.872139 4925 cpu_manager.go:414] "containerMap: removing container" podUID="213e6521-9211-4568-8645-6136c94b158b" containerName="otc-container" Oct 30 04:04:30.872510 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.872152 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="213e6521-9211-4568-8645-6136c94b158b" containerName="otc-container" Oct 30 04:04:30.872510 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.872221 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="213e6521-9211-4568-8645-6136c94b158b" containerName="otc-container" Oct 30 04:04:30.876991 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.876969 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:30.897363 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.897327 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:04:30.897487 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.897445 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:04:30.897568 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.897559 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-72c79cac\"" Oct 30 04:04:30.897620 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.897612 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:04:30.897668 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.897655 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-45ch8\"" Oct 30 04:04:30.897935 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.897911 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:30.955066 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.955037 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2070094a-1ed2-4df8-af0e-d3333bf26035\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:30.955066 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.955068 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2070094a-1ed2-4df8-af0e-d3333bf26035-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"2070094a-1ed2-4df8-af0e-d3333bf26035\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:30.955437 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.955088 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2070094a-1ed2-4df8-af0e-d3333bf26035\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:30.955437 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.955153 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk8ll\" (UniqueName: \"kubernetes.io/projected/2070094a-1ed2-4df8-af0e-d3333bf26035-kube-api-access-qk8ll\") pod \"data-science-collector-collector-0\" (UID: \"2070094a-1ed2-4df8-af0e-d3333bf26035\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:30.955437 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:30.955270 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"2070094a-1ed2-4df8-af0e-d3333bf26035\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:31.056114 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:31.056034 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"2070094a-1ed2-4df8-af0e-d3333bf26035\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:31.056273 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:31.056122 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2070094a-1ed2-4df8-af0e-d3333bf26035\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:31.056273 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:31.056152 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2070094a-1ed2-4df8-af0e-d3333bf26035-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"2070094a-1ed2-4df8-af0e-d3333bf26035\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:31.056273 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:31.056176 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2070094a-1ed2-4df8-af0e-d3333bf26035\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:31.056419 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:31.056318 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk8ll\" (UniqueName: \"kubernetes.io/projected/2070094a-1ed2-4df8-af0e-d3333bf26035-kube-api-access-qk8ll\") pod \"data-science-collector-collector-0\" (UID: \"2070094a-1ed2-4df8-af0e-d3333bf26035\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:31.056935 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:31.056913 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2070094a-1ed2-4df8-af0e-d3333bf26035-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"2070094a-1ed2-4df8-af0e-d3333bf26035\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:31.058645 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:31.058616 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2070094a-1ed2-4df8-af0e-d3333bf26035\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:31.058960 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:31.058943 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2070094a-1ed2-4df8-af0e-d3333bf26035\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:31.059057 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:31.059041 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"2070094a-1ed2-4df8-af0e-d3333bf26035\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:31.080108 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:31.080087 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk8ll\" (UniqueName: \"kubernetes.io/projected/2070094a-1ed2-4df8-af0e-d3333bf26035-kube-api-access-qk8ll\") pod \"data-science-collector-collector-0\" (UID: \"2070094a-1ed2-4df8-af0e-d3333bf26035\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:31.121689 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:31.121650 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="213e6521-9211-4568-8645-6136c94b158b" path="/var/lib/kubelet/pods/213e6521-9211-4568-8645-6136c94b158b/volumes" Oct 30 04:04:31.189989 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:31.189955 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:31.489244 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:31.489215 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:31.490713 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:04:31.490681 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2070094a_1ed2_4df8_af0e_d3333bf26035.slice/crio-d9fc1488f7af99fb1cbdfa1ad9d4c47f23d76bd0fe01e2b2265aaf9bf2a81a02 WatchSource:0}: Error finding container d9fc1488f7af99fb1cbdfa1ad9d4c47f23d76bd0fe01e2b2265aaf9bf2a81a02: Status 404 returned error can't find the container with id d9fc1488f7af99fb1cbdfa1ad9d4c47f23d76bd0fe01e2b2265aaf9bf2a81a02 Oct 30 04:04:31.492847 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:31.492816 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:31.492925 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:31.492877 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:31.786011 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:31.785931 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"2070094a-1ed2-4df8-af0e-d3333bf26035","Type":"ContainerStarted","Data":"030b977863a28eaaf955505c873690868cddeaa237083d33f26ac90656911f4d"} Oct 30 04:04:31.786011 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:31.785993 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"2070094a-1ed2-4df8-af0e-d3333bf26035","Type":"ContainerStarted","Data":"d9fc1488f7af99fb1cbdfa1ad9d4c47f23d76bd0fe01e2b2265aaf9bf2a81a02"} Oct 30 04:04:31.828862 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:31.828817 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.8288021269999999 podStartE2EDuration="1.828802127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:31.828179467 +0000 UTC m=+5032.841457569" watchObservedRunningTime="2026-10-30 04:04:31.828802127 +0000 UTC m=+5032.842080232" Oct 30 04:04:33.387585 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.387551 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk"] Oct 30 04:04:33.388104 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.387731 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" podUID="d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c" containerName="korrel8r" containerID="cri-o://42e7e9bab5cfbad4ec9758be77634ae26ca37d08d9cf3a0e3306d6d4c412406b" gracePeriod=30 Oct 30 04:04:33.389534 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.389508 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" Oct 30 04:04:33.634203 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.634177 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" Oct 30 04:04:33.680828 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.680731 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-kube-api-access-mfjw5\" (UniqueName: \"kubernetes.io/projected/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-kube-api-access-mfjw5\") pod \"d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c\" (UID: \"d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c\") " Oct 30 04:04:33.680828 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.680781 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-tls\" (UniqueName: \"kubernetes.io/secret/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-tls\") pod \"d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c\" (UID: \"d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c\") " Oct 30 04:04:33.681037 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.680869 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-config\" (UniqueName: \"kubernetes.io/configmap/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-config\") pod \"d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c\" (UID: \"d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c\") " Oct 30 04:04:33.681494 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.681466 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-config" pod "d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c" (UID: "d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:33.683547 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.683318 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-kube-api-access-mfjw5" pod "d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c" (UID: "d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c"). InnerVolumeSpecName "kube-api-access-mfjw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:33.683547 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.683371 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-tls" pod "d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c" (UID: "d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:33.781829 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.781790 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfjw5\" (UniqueName: \"kubernetes.io/projected/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-kube-api-access-mfjw5\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:33.781829 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.781826 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:33.781829 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.781838 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:33.796265 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.796232 4925 generic.go:356] "Generic (PLEG): container finished" podID="d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c" containerID="42e7e9bab5cfbad4ec9758be77634ae26ca37d08d9cf3a0e3306d6d4c412406b" exitCode=0 Oct 30 04:04:33.796373 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.796277 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" Oct 30 04:04:33.796373 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.796308 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" event={"ID":"d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c","Type":"ContainerDied","Data":"42e7e9bab5cfbad4ec9758be77634ae26ca37d08d9cf3a0e3306d6d4c412406b"} Oct 30 04:04:33.796373 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.796355 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk" event={"ID":"d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c","Type":"ContainerDied","Data":"823bd528b4300709d0e04da957d9162d08c0b8d87848cae58da0ff09c9cd9a47"} Oct 30 04:04:33.796475 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.796377 4925 scope.go:122] "RemoveContainer" containerID="42e7e9bab5cfbad4ec9758be77634ae26ca37d08d9cf3a0e3306d6d4c412406b" Oct 30 04:04:33.805435 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.805420 4925 scope.go:122] "RemoveContainer" containerID="42e7e9bab5cfbad4ec9758be77634ae26ca37d08d9cf3a0e3306d6d4c412406b" Oct 30 04:04:33.805707 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:04:33.805689 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e7e9bab5cfbad4ec9758be77634ae26ca37d08d9cf3a0e3306d6d4c412406b\": container with ID starting with 42e7e9bab5cfbad4ec9758be77634ae26ca37d08d9cf3a0e3306d6d4c412406b not found: ID does not exist" containerID="42e7e9bab5cfbad4ec9758be77634ae26ca37d08d9cf3a0e3306d6d4c412406b" Oct 30 04:04:33.805789 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.805713 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e7e9bab5cfbad4ec9758be77634ae26ca37d08d9cf3a0e3306d6d4c412406b"} err="failed to get container status \"42e7e9bab5cfbad4ec9758be77634ae26ca37d08d9cf3a0e3306d6d4c412406b\": rpc error: code = NotFound desc = could not find container \"42e7e9bab5cfbad4ec9758be77634ae26ca37d08d9cf3a0e3306d6d4c412406b\": container with ID starting with 42e7e9bab5cfbad4ec9758be77634ae26ca37d08d9cf3a0e3306d6d4c412406b not found: ID does not exist" Oct 30 04:04:33.848171 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.848146 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk"] Oct 30 04:04:33.852458 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:33.852430 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-bbmzk"] Oct 30 04:04:35.120384 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:35.120350 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c" path="/var/lib/kubelet/pods/d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c/volumes" Oct 30 04:04:37.647902 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:37.647865 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:37.648385 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:37.648083 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="2070094a-1ed2-4df8-af0e-d3333bf26035" containerName="otc-container" containerID="cri-o://030b977863a28eaaf955505c873690868cddeaa237083d33f26ac90656911f4d" gracePeriod=30 Oct 30 04:04:37.821588 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:37.821556 4925 generic.go:356] "Generic (PLEG): container finished" podID="2070094a-1ed2-4df8-af0e-d3333bf26035" containerID="030b977863a28eaaf955505c873690868cddeaa237083d33f26ac90656911f4d" exitCode=0 Oct 30 04:04:37.821733 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:37.821639 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"2070094a-1ed2-4df8-af0e-d3333bf26035","Type":"ContainerDied","Data":"030b977863a28eaaf955505c873690868cddeaa237083d33f26ac90656911f4d"} Oct 30 04:04:37.900005 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:37.899947 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:38.019933 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.019904 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-tls-certs\" (UniqueName: \"kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-tls-certs\") pod \"2070094a-1ed2-4df8-af0e-d3333bf26035\" (UID: \"2070094a-1ed2-4df8-af0e-d3333bf26035\") " Oct 30 04:04:38.019933 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.019936 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2070094a-1ed2-4df8-af0e-d3333bf26035-kube-api-access-qk8ll\" (UniqueName: \"kubernetes.io/projected/2070094a-1ed2-4df8-af0e-d3333bf26035-kube-api-access-qk8ll\") pod \"2070094a-1ed2-4df8-af0e-d3333bf26035\" (UID: \"2070094a-1ed2-4df8-af0e-d3333bf26035\") " Oct 30 04:04:38.020166 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.019965 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2070094a-1ed2-4df8-af0e-d3333bf26035-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2070094a-1ed2-4df8-af0e-d3333bf26035-otc-internal\") pod \"2070094a-1ed2-4df8-af0e-d3333bf26035\" (UID: \"2070094a-1ed2-4df8-af0e-d3333bf26035\") " Oct 30 04:04:38.020166 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.020087 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-monitoring-tls-certs\") pod \"2070094a-1ed2-4df8-af0e-d3333bf26035\" (UID: \"2070094a-1ed2-4df8-af0e-d3333bf26035\") " Oct 30 04:04:38.020277 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.020190 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-data-science-collector-ta-client-cert\") pod \"2070094a-1ed2-4df8-af0e-d3333bf26035\" (UID: \"2070094a-1ed2-4df8-af0e-d3333bf26035\") " Oct 30 04:04:38.020346 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.020318 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2070094a-1ed2-4df8-af0e-d3333bf26035-otc-internal" pod "2070094a-1ed2-4df8-af0e-d3333bf26035" (UID: "2070094a-1ed2-4df8-af0e-d3333bf26035"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:38.020436 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.020421 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2070094a-1ed2-4df8-af0e-d3333bf26035-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:38.022198 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.022172 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-tls-certs" pod "2070094a-1ed2-4df8-af0e-d3333bf26035" (UID: "2070094a-1ed2-4df8-af0e-d3333bf26035"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:38.022198 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.022184 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2070094a-1ed2-4df8-af0e-d3333bf26035-kube-api-access-qk8ll" pod "2070094a-1ed2-4df8-af0e-d3333bf26035" (UID: "2070094a-1ed2-4df8-af0e-d3333bf26035"). InnerVolumeSpecName "kube-api-access-qk8ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:38.022493 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.022464 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-data-science-collector-ta-client-cert" pod "2070094a-1ed2-4df8-af0e-d3333bf26035" (UID: "2070094a-1ed2-4df8-af0e-d3333bf26035"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:38.022493 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.022481 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-monitoring-tls-certs" pod "2070094a-1ed2-4df8-af0e-d3333bf26035" (UID: "2070094a-1ed2-4df8-af0e-d3333bf26035"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:38.121472 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.121439 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:38.121472 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.121467 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk8ll\" (UniqueName: \"kubernetes.io/projected/2070094a-1ed2-4df8-af0e-d3333bf26035-kube-api-access-qk8ll\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:38.121472 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.121478 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:38.121679 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.121488 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2070094a-1ed2-4df8-af0e-d3333bf26035-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:38.829058 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.829024 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"2070094a-1ed2-4df8-af0e-d3333bf26035","Type":"ContainerDied","Data":"d9fc1488f7af99fb1cbdfa1ad9d4c47f23d76bd0fe01e2b2265aaf9bf2a81a02"} Oct 30 04:04:38.829058 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.829058 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:38.829568 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.829090 4925 scope.go:122] "RemoveContainer" containerID="030b977863a28eaaf955505c873690868cddeaa237083d33f26ac90656911f4d" Oct 30 04:04:38.883341 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.883312 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:38.894761 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.894727 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:38.928744 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.928718 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:38.929123 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.929109 4925 cpu_manager.go:414] "containerMap: removing container" podUID="2070094a-1ed2-4df8-af0e-d3333bf26035" containerName="otc-container" Oct 30 04:04:38.929177 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.929124 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2070094a-1ed2-4df8-af0e-d3333bf26035" containerName="otc-container" Oct 30 04:04:38.929177 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.929139 4925 cpu_manager.go:414] "containerMap: removing container" podUID="d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c" containerName="korrel8r" Oct 30 04:04:38.929177 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.929144 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c" containerName="korrel8r" Oct 30 04:04:38.929268 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.929226 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2070094a-1ed2-4df8-af0e-d3333bf26035" containerName="otc-container" Oct 30 04:04:38.929268 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.929233 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f6c760-d2d0-4b18-ad0a-0fa60ea7442c" containerName="korrel8r" Oct 30 04:04:38.934114 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.934094 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:38.952812 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.952790 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:04:38.953401 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.952845 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:04:38.953401 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.952968 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:04:38.953401 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.953023 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:04:38.953587 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.953109 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-45ch8\"" Oct 30 04:04:38.954459 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:38.954437 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:39.028834 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.028804 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:39.028954 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.028848 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:39.028954 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.028877 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgjv4\" (UniqueName: \"kubernetes.io/projected/0df97dd1-be9a-43e3-b52a-cad79d3192f3-kube-api-access-vgjv4\") pod \"data-science-collector-collector-0\" (UID: \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:39.028954 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.028949 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0df97dd1-be9a-43e3-b52a-cad79d3192f3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:39.029078 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.028991 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:39.125998 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.125922 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2070094a-1ed2-4df8-af0e-d3333bf26035" path="/var/lib/kubelet/pods/2070094a-1ed2-4df8-af0e-d3333bf26035/volumes" Oct 30 04:04:39.129308 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.129287 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:39.129374 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.129317 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgjv4\" (UniqueName: \"kubernetes.io/projected/0df97dd1-be9a-43e3-b52a-cad79d3192f3-kube-api-access-vgjv4\") pod \"data-science-collector-collector-0\" (UID: \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:39.129374 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.129346 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0df97dd1-be9a-43e3-b52a-cad79d3192f3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:39.129439 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.129399 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:39.129471 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.129441 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:39.143591 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.143566 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:04:39.143772 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.143694 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:04:39.143862 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.143844 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:04:39.143921 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.143864 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:04:39.151197 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.151156 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0df97dd1-be9a-43e3-b52a-cad79d3192f3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:39.152718 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.152687 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:39.152831 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.152737 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:39.152916 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.152895 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgjv4\" (UniqueName: \"kubernetes.io/projected/0df97dd1-be9a-43e3-b52a-cad79d3192f3-kube-api-access-vgjv4\") pod \"data-science-collector-collector-0\" (UID: \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:39.152955 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.152897 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:39.261157 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.261128 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-45ch8\"" Oct 30 04:04:39.267338 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.267320 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:39.570983 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.570961 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:39.573005 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:04:39.572968 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df97dd1_be9a_43e3_b52a_cad79d3192f3.slice/crio-e7f655c273be7e49fae44a83d5ebd7f1e932f0044532d7d81920edd35d835b8b WatchSource:0}: Error finding container e7f655c273be7e49fae44a83d5ebd7f1e932f0044532d7d81920edd35d835b8b: Status 404 returned error can't find the container with id e7f655c273be7e49fae44a83d5ebd7f1e932f0044532d7d81920edd35d835b8b Oct 30 04:04:39.575203 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.575171 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:39.575276 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.575233 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:39.836275 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.836232 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"0df97dd1-be9a-43e3-b52a-cad79d3192f3","Type":"ContainerStarted","Data":"5ee9dbe4d05d277fe15244b437f66bc34b9ca9aa2e63b400721fa692488b84e1"} Oct 30 04:04:39.836724 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.836302 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"0df97dd1-be9a-43e3-b52a-cad79d3192f3","Type":"ContainerStarted","Data":"e7f655c273be7e49fae44a83d5ebd7f1e932f0044532d7d81920edd35d835b8b"} Oct 30 04:04:39.882552 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:39.882497 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.8824834080000001 podStartE2EDuration="1.882483408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:39.88226558 +0000 UTC m=+5040.895543683" watchObservedRunningTime="2026-10-30 04:04:39.882483408 +0000 UTC m=+5040.895761511" Oct 30 04:04:40.094536 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.094460 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8"] Oct 30 04:04:40.094717 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.094692 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" podUID="776067d7-1633-44ae-986c-56cafc4be4c4" containerName="kube-rbac-proxy" containerID="cri-o://f5edc65e25d7f11d3f41f39e93908b74d309ea4d6cac4b384d9c33ed62d3b931" gracePeriod=30 Oct 30 04:04:40.161263 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.161231 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r"] Oct 30 04:04:40.161465 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.161425 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" podUID="b5e58337-f692-4b01-b15d-7642a69d67f2" containerName="korrel8r" containerID="cri-o://d4aa3f5e7a4cd3d35825d1f8b3e8529d63aedb36a7ce2364a487a5c45500961c" gracePeriod=30 Oct 30 04:04:40.411636 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.411613 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:04:40.432232 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.432212 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" Oct 30 04:04:40.442234 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.441475 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/776067d7-1633-44ae-986c-56cafc4be4c4-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/776067d7-1633-44ae-986c-56cafc4be4c4-prometheus-ca\") pod \"776067d7-1633-44ae-986c-56cafc4be4c4\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " Oct 30 04:04:40.442234 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.441535 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-tls\" (UniqueName: \"kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-tls\") pod \"776067d7-1633-44ae-986c-56cafc4be4c4\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " Oct 30 04:04:40.442234 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.441587 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-kube-rbac-proxy-config\") pod \"776067d7-1633-44ae-986c-56cafc4be4c4\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " Oct 30 04:04:40.442234 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.441635 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/776067d7-1633-44ae-986c-56cafc4be4c4-kube-api-access-26q9x\" (UniqueName: \"kubernetes.io/projected/776067d7-1633-44ae-986c-56cafc4be4c4-kube-api-access-26q9x\") pod \"776067d7-1633-44ae-986c-56cafc4be4c4\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " Oct 30 04:04:40.442234 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.441682 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-prometheus-client-cert\") pod \"776067d7-1633-44ae-986c-56cafc4be4c4\" (UID: \"776067d7-1633-44ae-986c-56cafc4be4c4\") " Oct 30 04:04:40.444487 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.444445 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/776067d7-1633-44ae-986c-56cafc4be4c4-prometheus-ca" pod "776067d7-1633-44ae-986c-56cafc4be4c4" (UID: "776067d7-1633-44ae-986c-56cafc4be4c4"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:40.449222 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.449178 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-prometheus-client-cert" pod "776067d7-1633-44ae-986c-56cafc4be4c4" (UID: "776067d7-1633-44ae-986c-56cafc4be4c4"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:40.449222 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.449211 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-tls" pod "776067d7-1633-44ae-986c-56cafc4be4c4" (UID: "776067d7-1633-44ae-986c-56cafc4be4c4"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:40.449353 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.449211 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776067d7-1633-44ae-986c-56cafc4be4c4-kube-api-access-26q9x" pod "776067d7-1633-44ae-986c-56cafc4be4c4" (UID: "776067d7-1633-44ae-986c-56cafc4be4c4"). InnerVolumeSpecName "kube-api-access-26q9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:40.449353 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.449254 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-kube-rbac-proxy-config" pod "776067d7-1633-44ae-986c-56cafc4be4c4" (UID: "776067d7-1633-44ae-986c-56cafc4be4c4"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:40.543011 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.542981 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5e58337-f692-4b01-b15d-7642a69d67f2-config\" (UniqueName: \"kubernetes.io/configmap/b5e58337-f692-4b01-b15d-7642a69d67f2-config\") pod \"b5e58337-f692-4b01-b15d-7642a69d67f2\" (UID: \"b5e58337-f692-4b01-b15d-7642a69d67f2\") " Oct 30 04:04:40.543189 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.543036 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5e58337-f692-4b01-b15d-7642a69d67f2-tls\" (UniqueName: \"kubernetes.io/secret/b5e58337-f692-4b01-b15d-7642a69d67f2-tls\") pod \"b5e58337-f692-4b01-b15d-7642a69d67f2\" (UID: \"b5e58337-f692-4b01-b15d-7642a69d67f2\") " Oct 30 04:04:40.543189 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.543061 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5e58337-f692-4b01-b15d-7642a69d67f2-kube-api-access-ddv4n\" (UniqueName: \"kubernetes.io/projected/b5e58337-f692-4b01-b15d-7642a69d67f2-kube-api-access-ddv4n\") pod \"b5e58337-f692-4b01-b15d-7642a69d67f2\" (UID: \"b5e58337-f692-4b01-b15d-7642a69d67f2\") " Oct 30 04:04:40.543328 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.543266 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/776067d7-1633-44ae-986c-56cafc4be4c4-prometheus-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:40.543328 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.543278 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:40.543328 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.543287 4925 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-kube-rbac-proxy-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:40.543328 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.543296 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26q9x\" (UniqueName: \"kubernetes.io/projected/776067d7-1633-44ae-986c-56cafc4be4c4-kube-api-access-26q9x\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:40.543328 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.543305 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/776067d7-1633-44ae-986c-56cafc4be4c4-prometheus-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:40.543588 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.543436 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e58337-f692-4b01-b15d-7642a69d67f2-config" pod "b5e58337-f692-4b01-b15d-7642a69d67f2" (UID: "b5e58337-f692-4b01-b15d-7642a69d67f2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:40.545437 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.545409 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e58337-f692-4b01-b15d-7642a69d67f2-tls" pod "b5e58337-f692-4b01-b15d-7642a69d67f2" (UID: "b5e58337-f692-4b01-b15d-7642a69d67f2"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:40.549303 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.549275 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e58337-f692-4b01-b15d-7642a69d67f2-kube-api-access-ddv4n" pod "b5e58337-f692-4b01-b15d-7642a69d67f2" (UID: "b5e58337-f692-4b01-b15d-7642a69d67f2"). InnerVolumeSpecName "kube-api-access-ddv4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:40.644212 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.644139 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5e58337-f692-4b01-b15d-7642a69d67f2-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:40.644212 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.644164 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b5e58337-f692-4b01-b15d-7642a69d67f2-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:40.644212 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.644174 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddv4n\" (UniqueName: \"kubernetes.io/projected/b5e58337-f692-4b01-b15d-7642a69d67f2-kube-api-access-ddv4n\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:40.663026 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.662997 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:04:40.663237 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.663211 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="8bf40867-6019-4c8c-9488-b9ed3cff2efb" containerName="alertmanager" containerID="cri-o://571a72bcf474a67ce31ee35b1cf1829a4ccae217b602704a1056a27f3f5a9c7b" gracePeriod=120 Oct 30 04:04:40.663387 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.663243 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="8bf40867-6019-4c8c-9488-b9ed3cff2efb" containerName="config-reloader" containerID="cri-o://f81733c84e59f8faea08940e57c0da664b647cc0bf6b2295738950944baa9ec7" gracePeriod=120 Oct 30 04:04:40.679507 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.679478 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:04:40.679722 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.679698 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" containerName="thanos-sidecar" containerID="cri-o://1a81157ff602aada80fa0b341c16b4ce7d80e6a31c3871994bb329ce36654a27" gracePeriod=600 Oct 30 04:04:40.679825 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.679714 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" containerName="config-reloader" containerID="cri-o://ce9867f7ea0cf96ad1f0d0ed22f9fc203e472bf42a9792f6ecb23ebbd3e5e451" gracePeriod=600 Oct 30 04:04:40.679825 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.679698 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" containerName="prometheus" containerID="cri-o://065d6459faefe34b553d474d02359f7c13b7cdb563a66a6400675997eebee473" gracePeriod=600 Oct 30 04:04:40.845438 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.845403 4925 generic.go:356] "Generic (PLEG): container finished" podID="776067d7-1633-44ae-986c-56cafc4be4c4" containerID="f5edc65e25d7f11d3f41f39e93908b74d309ea4d6cac4b384d9c33ed62d3b931" exitCode=0 Oct 30 04:04:40.845911 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.845461 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" Oct 30 04:04:40.845911 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.845494 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" event={"ID":"776067d7-1633-44ae-986c-56cafc4be4c4","Type":"ContainerDied","Data":"f5edc65e25d7f11d3f41f39e93908b74d309ea4d6cac4b384d9c33ed62d3b931"} Oct 30 04:04:40.845911 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.845574 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8" event={"ID":"776067d7-1633-44ae-986c-56cafc4be4c4","Type":"ContainerDied","Data":"b420f2a5046bfdd8d48117f33840782bfd898c2c3f90037c9b854ecb68235353"} Oct 30 04:04:40.845911 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.845629 4925 scope.go:122] "RemoveContainer" containerID="f5edc65e25d7f11d3f41f39e93908b74d309ea4d6cac4b384d9c33ed62d3b931" Oct 30 04:04:40.849530 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.849510 4925 generic.go:356] "Generic (PLEG): container finished" podID="7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" containerID="1a81157ff602aada80fa0b341c16b4ce7d80e6a31c3871994bb329ce36654a27" exitCode=0 Oct 30 04:04:40.849530 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.849528 4925 generic.go:356] "Generic (PLEG): container finished" podID="7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" containerID="065d6459faefe34b553d474d02359f7c13b7cdb563a66a6400675997eebee473" exitCode=0 Oct 30 04:04:40.849717 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.849571 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6","Type":"ContainerDied","Data":"1a81157ff602aada80fa0b341c16b4ce7d80e6a31c3871994bb329ce36654a27"} Oct 30 04:04:40.849717 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.849647 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6","Type":"ContainerDied","Data":"065d6459faefe34b553d474d02359f7c13b7cdb563a66a6400675997eebee473"} Oct 30 04:04:40.851129 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.851103 4925 generic.go:356] "Generic (PLEG): container finished" podID="b5e58337-f692-4b01-b15d-7642a69d67f2" containerID="d4aa3f5e7a4cd3d35825d1f8b3e8529d63aedb36a7ce2364a487a5c45500961c" exitCode=0 Oct 30 04:04:40.851235 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.851142 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" Oct 30 04:04:40.851235 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.851191 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" event={"ID":"b5e58337-f692-4b01-b15d-7642a69d67f2","Type":"ContainerDied","Data":"d4aa3f5e7a4cd3d35825d1f8b3e8529d63aedb36a7ce2364a487a5c45500961c"} Oct 30 04:04:40.851367 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.851238 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r" event={"ID":"b5e58337-f692-4b01-b15d-7642a69d67f2","Type":"ContainerDied","Data":"325e04d066e5e0df30f65363d42b6bb6a57b39febf133aa9c03ab4e2e3e70f73"} Oct 30 04:04:40.857523 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.857505 4925 scope.go:122] "RemoveContainer" containerID="f5edc65e25d7f11d3f41f39e93908b74d309ea4d6cac4b384d9c33ed62d3b931" Oct 30 04:04:40.857799 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:04:40.857773 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5edc65e25d7f11d3f41f39e93908b74d309ea4d6cac4b384d9c33ed62d3b931\": container with ID starting with f5edc65e25d7f11d3f41f39e93908b74d309ea4d6cac4b384d9c33ed62d3b931 not found: ID does not exist" containerID="f5edc65e25d7f11d3f41f39e93908b74d309ea4d6cac4b384d9c33ed62d3b931" Oct 30 04:04:40.857879 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.857809 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5edc65e25d7f11d3f41f39e93908b74d309ea4d6cac4b384d9c33ed62d3b931"} err="failed to get container status \"f5edc65e25d7f11d3f41f39e93908b74d309ea4d6cac4b384d9c33ed62d3b931\": rpc error: code = NotFound desc = could not find container \"f5edc65e25d7f11d3f41f39e93908b74d309ea4d6cac4b384d9c33ed62d3b931\": container with ID starting with f5edc65e25d7f11d3f41f39e93908b74d309ea4d6cac4b384d9c33ed62d3b931 not found: ID does not exist" Oct 30 04:04:40.857879 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.857825 4925 scope.go:122] "RemoveContainer" containerID="d4aa3f5e7a4cd3d35825d1f8b3e8529d63aedb36a7ce2364a487a5c45500961c" Oct 30 04:04:40.866031 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.866012 4925 scope.go:122] "RemoveContainer" containerID="d4aa3f5e7a4cd3d35825d1f8b3e8529d63aedb36a7ce2364a487a5c45500961c" Oct 30 04:04:40.866296 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:04:40.866279 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4aa3f5e7a4cd3d35825d1f8b3e8529d63aedb36a7ce2364a487a5c45500961c\": container with ID starting with d4aa3f5e7a4cd3d35825d1f8b3e8529d63aedb36a7ce2364a487a5c45500961c not found: ID does not exist" containerID="d4aa3f5e7a4cd3d35825d1f8b3e8529d63aedb36a7ce2364a487a5c45500961c" Oct 30 04:04:40.866371 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.866300 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4aa3f5e7a4cd3d35825d1f8b3e8529d63aedb36a7ce2364a487a5c45500961c"} err="failed to get container status \"d4aa3f5e7a4cd3d35825d1f8b3e8529d63aedb36a7ce2364a487a5c45500961c\": rpc error: code = NotFound desc = could not find container \"d4aa3f5e7a4cd3d35825d1f8b3e8529d63aedb36a7ce2364a487a5c45500961c\": container with ID starting with d4aa3f5e7a4cd3d35825d1f8b3e8529d63aedb36a7ce2364a487a5c45500961c not found: ID does not exist" Oct 30 04:04:40.898449 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.898318 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8"] Oct 30 04:04:40.900184 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.900168 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-5shw8"] Oct 30 04:04:40.934742 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.934722 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r"] Oct 30 04:04:40.939131 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:40.939112 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-5b48r"] Oct 30 04:04:41.112141 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.112115 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:04:41.119730 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.119694 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="776067d7-1633-44ae-986c-56cafc4be4c4" path="/var/lib/kubelet/pods/776067d7-1633-44ae-986c-56cafc4be4c4/volumes" Oct 30 04:04:41.120126 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.120107 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5e58337-f692-4b01-b15d-7642a69d67f2" path="/var/lib/kubelet/pods/b5e58337-f692-4b01-b15d-7642a69d67f2/volumes" Oct 30 04:04:41.148100 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.148067 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-config\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-config\") pod \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " Oct 30 04:04:41.148100 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.148101 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-tls-assets\" (UniqueName: \"kubernetes.io/projected/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-tls-assets\") pod \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " Oct 30 04:04:41.148277 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.148125 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-thanos-prometheus-http-client-file\") pod \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " Oct 30 04:04:41.148277 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.148148 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " Oct 30 04:04:41.148277 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.148176 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-secret-prometheus-web-tls-ca\") pod \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " Oct 30 04:04:41.148277 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.148208 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-config-out\") pod \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " Oct 30 04:04:41.148277 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.148241 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " Oct 30 04:04:41.148559 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.148282 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " Oct 30 04:04:41.148559 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.148335 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " Oct 30 04:04:41.148559 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.148432 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " Oct 30 04:04:41.148559 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.148471 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config\") pod \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " Oct 30 04:04:41.148559 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.148545 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " Oct 30 04:04:41.148860 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.148577 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-kube-api-access-m68gt\" (UniqueName: \"kubernetes.io/projected/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-kube-api-access-m68gt\") pod \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\" (UID: \"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6\") " Oct 30 04:04:41.148860 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.148578 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-2" pod "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" (UID: "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:41.148979 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.148908 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.149303 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.149259 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-1" pod "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" (UID: "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:41.149640 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.149611 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-0" pod "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" (UID: "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:41.150872 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.150834 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-secret-prometheus-web-tls-ca" pod "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" (UID: "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:41.151405 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.151380 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-tls-assets" pod "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" (UID: "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:41.151491 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.151447 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-config" pod "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" (UID: "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:41.151722 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.151685 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" (UID: "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:41.152049 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.152024 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" (UID: "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:41.152678 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.152653 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-thanos-prometheus-http-client-file" pod "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" (UID: "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:41.152678 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.152671 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-config-out" pod "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" (UID: "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:04:41.152864 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.152789 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-kube-api-access-m68gt" pod "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" (UID: "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6"). InnerVolumeSpecName "kube-api-access-m68gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:41.154495 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.154475 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4" pod "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" (UID: "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6"). InnerVolumeSpecName "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:04:41.165800 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.164981 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config" pod "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" (UID: "7f3a5697-5bf8-4f0e-8017-e36d0738bbc6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:41.250114 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.250081 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.250114 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.250112 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m68gt\" (UniqueName: \"kubernetes.io/projected/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-kube-api-access-m68gt\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.250305 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.250123 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.250305 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.250134 4925 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-tls-assets\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.250305 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.250144 4925 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.250305 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.250153 4925 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-secret-prometheus-web-tls-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.250305 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.250162 4925 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-config-out\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.250305 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.250171 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.250305 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.250181 4925 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.250305 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.250190 4925 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.250305 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.250220 4925 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-139-19.ec2.internal\" " Oct 30 04:04:41.250305 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.250229 4925 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6-web-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:41.261328 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.261311 4925 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4") on node "ip-10-0-139-19.ec2.internal" Oct 30 04:04:41.350995 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.350952 4925 reconciler_common.go:299] "Volume detached for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"csi-ec2ccb9dcc2629e5a59ea91641dbb6b15d80e87697977b7c41508ff0003b2f36\"" Oct 30 04:04:41.355564 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.355535 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:41.860637 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.860602 4925 generic.go:356] "Generic (PLEG): container finished" podID="7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" containerID="ce9867f7ea0cf96ad1f0d0ed22f9fc203e472bf42a9792f6ecb23ebbd3e5e451" exitCode=0 Oct 30 04:04:41.861151 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.860686 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6","Type":"ContainerDied","Data":"ce9867f7ea0cf96ad1f0d0ed22f9fc203e472bf42a9792f6ecb23ebbd3e5e451"} Oct 30 04:04:41.861151 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.860698 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 04:04:41.861151 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.860798 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"7f3a5697-5bf8-4f0e-8017-e36d0738bbc6","Type":"ContainerDied","Data":"2122fda3a7c1c289c7006077a7fbdf8e514199cd452439782d7801e2cadbeb5a"} Oct 30 04:04:41.861151 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.860816 4925 scope.go:122] "RemoveContainer" containerID="1a81157ff602aada80fa0b341c16b4ce7d80e6a31c3871994bb329ce36654a27" Oct 30 04:04:41.864200 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.864176 4925 generic.go:356] "Generic (PLEG): container finished" podID="8bf40867-6019-4c8c-9488-b9ed3cff2efb" containerID="f81733c84e59f8faea08940e57c0da664b647cc0bf6b2295738950944baa9ec7" exitCode=0 Oct 30 04:04:41.864307 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.864247 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"8bf40867-6019-4c8c-9488-b9ed3cff2efb","Type":"ContainerDied","Data":"f81733c84e59f8faea08940e57c0da664b647cc0bf6b2295738950944baa9ec7"} Oct 30 04:04:41.864367 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.864310 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="0df97dd1-be9a-43e3-b52a-cad79d3192f3" containerName="otc-container" containerID="cri-o://5ee9dbe4d05d277fe15244b437f66bc34b9ca9aa2e63b400721fa692488b84e1" gracePeriod=30 Oct 30 04:04:41.874800 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.874774 4925 scope.go:122] "RemoveContainer" containerID="ce9867f7ea0cf96ad1f0d0ed22f9fc203e472bf42a9792f6ecb23ebbd3e5e451" Oct 30 04:04:41.930970 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.930946 4925 scope.go:122] "RemoveContainer" containerID="065d6459faefe34b553d474d02359f7c13b7cdb563a66a6400675997eebee473" Oct 30 04:04:41.961765 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.961725 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:04:41.974998 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.974972 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 04:04:41.976778 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.976743 4925 scope.go:122] "RemoveContainer" containerID="441ce4cab54f981a440cfb16563d0495bf5f1b200fefaaa267ccf27d09e1e08f" Oct 30 04:04:41.985260 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.985238 4925 scope.go:122] "RemoveContainer" containerID="1a81157ff602aada80fa0b341c16b4ce7d80e6a31c3871994bb329ce36654a27" Oct 30 04:04:41.985569 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:04:41.985545 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a81157ff602aada80fa0b341c16b4ce7d80e6a31c3871994bb329ce36654a27\": container with ID starting with 1a81157ff602aada80fa0b341c16b4ce7d80e6a31c3871994bb329ce36654a27 not found: ID does not exist" containerID="1a81157ff602aada80fa0b341c16b4ce7d80e6a31c3871994bb329ce36654a27" Oct 30 04:04:41.985665 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.985583 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a81157ff602aada80fa0b341c16b4ce7d80e6a31c3871994bb329ce36654a27"} err="failed to get container status \"1a81157ff602aada80fa0b341c16b4ce7d80e6a31c3871994bb329ce36654a27\": rpc error: code = NotFound desc = could not find container \"1a81157ff602aada80fa0b341c16b4ce7d80e6a31c3871994bb329ce36654a27\": container with ID starting with 1a81157ff602aada80fa0b341c16b4ce7d80e6a31c3871994bb329ce36654a27 not found: ID does not exist" Oct 30 04:04:41.985665 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.985608 4925 scope.go:122] "RemoveContainer" containerID="ce9867f7ea0cf96ad1f0d0ed22f9fc203e472bf42a9792f6ecb23ebbd3e5e451" Oct 30 04:04:41.985951 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:04:41.985926 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce9867f7ea0cf96ad1f0d0ed22f9fc203e472bf42a9792f6ecb23ebbd3e5e451\": container with ID starting with ce9867f7ea0cf96ad1f0d0ed22f9fc203e472bf42a9792f6ecb23ebbd3e5e451 not found: ID does not exist" containerID="ce9867f7ea0cf96ad1f0d0ed22f9fc203e472bf42a9792f6ecb23ebbd3e5e451" Oct 30 04:04:41.985990 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.985963 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce9867f7ea0cf96ad1f0d0ed22f9fc203e472bf42a9792f6ecb23ebbd3e5e451"} err="failed to get container status \"ce9867f7ea0cf96ad1f0d0ed22f9fc203e472bf42a9792f6ecb23ebbd3e5e451\": rpc error: code = NotFound desc = could not find container \"ce9867f7ea0cf96ad1f0d0ed22f9fc203e472bf42a9792f6ecb23ebbd3e5e451\": container with ID starting with ce9867f7ea0cf96ad1f0d0ed22f9fc203e472bf42a9792f6ecb23ebbd3e5e451 not found: ID does not exist" Oct 30 04:04:41.985990 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.985986 4925 scope.go:122] "RemoveContainer" containerID="065d6459faefe34b553d474d02359f7c13b7cdb563a66a6400675997eebee473" Oct 30 04:04:41.986272 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:04:41.986250 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"065d6459faefe34b553d474d02359f7c13b7cdb563a66a6400675997eebee473\": container with ID starting with 065d6459faefe34b553d474d02359f7c13b7cdb563a66a6400675997eebee473 not found: ID does not exist" containerID="065d6459faefe34b553d474d02359f7c13b7cdb563a66a6400675997eebee473" Oct 30 04:04:41.986323 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.986281 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"065d6459faefe34b553d474d02359f7c13b7cdb563a66a6400675997eebee473"} err="failed to get container status \"065d6459faefe34b553d474d02359f7c13b7cdb563a66a6400675997eebee473\": rpc error: code = NotFound desc = could not find container \"065d6459faefe34b553d474d02359f7c13b7cdb563a66a6400675997eebee473\": container with ID starting with 065d6459faefe34b553d474d02359f7c13b7cdb563a66a6400675997eebee473 not found: ID does not exist" Oct 30 04:04:41.986323 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.986301 4925 scope.go:122] "RemoveContainer" containerID="441ce4cab54f981a440cfb16563d0495bf5f1b200fefaaa267ccf27d09e1e08f" Oct 30 04:04:41.986600 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:04:41.986578 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"441ce4cab54f981a440cfb16563d0495bf5f1b200fefaaa267ccf27d09e1e08f\": container with ID starting with 441ce4cab54f981a440cfb16563d0495bf5f1b200fefaaa267ccf27d09e1e08f not found: ID does not exist" containerID="441ce4cab54f981a440cfb16563d0495bf5f1b200fefaaa267ccf27d09e1e08f" Oct 30 04:04:41.986667 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:41.986607 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"441ce4cab54f981a440cfb16563d0495bf5f1b200fefaaa267ccf27d09e1e08f"} err="failed to get container status \"441ce4cab54f981a440cfb16563d0495bf5f1b200fefaaa267ccf27d09e1e08f\": rpc error: code = NotFound desc = could not find container \"441ce4cab54f981a440cfb16563d0495bf5f1b200fefaaa267ccf27d09e1e08f\": container with ID starting with 441ce4cab54f981a440cfb16563d0495bf5f1b200fefaaa267ccf27d09e1e08f not found: ID does not exist" Oct 30 04:04:42.106720 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.106693 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:42.157836 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.157784 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-monitoring-tls-certs\") pod \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\" (UID: \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\") " Oct 30 04:04:42.158002 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.157844 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df97dd1-be9a-43e3-b52a-cad79d3192f3-kube-api-access-vgjv4\" (UniqueName: \"kubernetes.io/projected/0df97dd1-be9a-43e3-b52a-cad79d3192f3-kube-api-access-vgjv4\") pod \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\" (UID: \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\") " Oct 30 04:04:42.158002 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.157885 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0df97dd1-be9a-43e3-b52a-cad79d3192f3-otc-internal\" (UniqueName: \"kubernetes.io/configmap/0df97dd1-be9a-43e3-b52a-cad79d3192f3-otc-internal\") pod \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\" (UID: \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\") " Oct 30 04:04:42.158002 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.157908 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-tls-certs\") pod \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\" (UID: \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\") " Oct 30 04:04:42.158002 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.157926 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-data-science-collector-ta-client-cert\") pod \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\" (UID: \"0df97dd1-be9a-43e3-b52a-cad79d3192f3\") " Oct 30 04:04:42.158817 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.158786 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df97dd1-be9a-43e3-b52a-cad79d3192f3-otc-internal" pod "0df97dd1-be9a-43e3-b52a-cad79d3192f3" (UID: "0df97dd1-be9a-43e3-b52a-cad79d3192f3"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:04:42.161388 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.161158 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-monitoring-tls-certs" pod "0df97dd1-be9a-43e3-b52a-cad79d3192f3" (UID: "0df97dd1-be9a-43e3-b52a-cad79d3192f3"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:42.161388 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.161208 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-tls-certs" pod "0df97dd1-be9a-43e3-b52a-cad79d3192f3" (UID: "0df97dd1-be9a-43e3-b52a-cad79d3192f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:42.161388 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.161223 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df97dd1-be9a-43e3-b52a-cad79d3192f3-kube-api-access-vgjv4" pod "0df97dd1-be9a-43e3-b52a-cad79d3192f3" (UID: "0df97dd1-be9a-43e3-b52a-cad79d3192f3"). InnerVolumeSpecName "kube-api-access-vgjv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:42.161388 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.161224 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-data-science-collector-ta-client-cert" pod "0df97dd1-be9a-43e3-b52a-cad79d3192f3" (UID: "0df97dd1-be9a-43e3-b52a-cad79d3192f3"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:42.259260 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.259230 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:42.259260 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.259258 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgjv4\" (UniqueName: \"kubernetes.io/projected/0df97dd1-be9a-43e3-b52a-cad79d3192f3-kube-api-access-vgjv4\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:42.259460 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.259269 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0df97dd1-be9a-43e3-b52a-cad79d3192f3-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:42.259460 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.259278 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:42.259460 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.259288 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0df97dd1-be9a-43e3-b52a-cad79d3192f3-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:42.382556 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.382474 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="8bf40867-6019-4c8c-9488-b9ed3cff2efb" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.104:9093/-/ready\": dial tcp 10.132.0.104:9093: connect: connection refused" Oct 30 04:04:42.869694 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.869653 4925 generic.go:356] "Generic (PLEG): container finished" podID="0df97dd1-be9a-43e3-b52a-cad79d3192f3" containerID="5ee9dbe4d05d277fe15244b437f66bc34b9ca9aa2e63b400721fa692488b84e1" exitCode=0 Oct 30 04:04:42.870156 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.869707 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:42.870156 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.869732 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"0df97dd1-be9a-43e3-b52a-cad79d3192f3","Type":"ContainerDied","Data":"5ee9dbe4d05d277fe15244b437f66bc34b9ca9aa2e63b400721fa692488b84e1"} Oct 30 04:04:42.870156 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.869794 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"0df97dd1-be9a-43e3-b52a-cad79d3192f3","Type":"ContainerDied","Data":"e7f655c273be7e49fae44a83d5ebd7f1e932f0044532d7d81920edd35d835b8b"} Oct 30 04:04:42.870156 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.869816 4925 scope.go:122] "RemoveContainer" containerID="5ee9dbe4d05d277fe15244b437f66bc34b9ca9aa2e63b400721fa692488b84e1" Oct 30 04:04:42.878897 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.878877 4925 scope.go:122] "RemoveContainer" containerID="5ee9dbe4d05d277fe15244b437f66bc34b9ca9aa2e63b400721fa692488b84e1" Oct 30 04:04:42.879164 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:04:42.879146 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee9dbe4d05d277fe15244b437f66bc34b9ca9aa2e63b400721fa692488b84e1\": container with ID starting with 5ee9dbe4d05d277fe15244b437f66bc34b9ca9aa2e63b400721fa692488b84e1 not found: ID does not exist" containerID="5ee9dbe4d05d277fe15244b437f66bc34b9ca9aa2e63b400721fa692488b84e1" Oct 30 04:04:42.879208 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.879175 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee9dbe4d05d277fe15244b437f66bc34b9ca9aa2e63b400721fa692488b84e1"} err="failed to get container status \"5ee9dbe4d05d277fe15244b437f66bc34b9ca9aa2e63b400721fa692488b84e1\": rpc error: code = NotFound desc = could not find container \"5ee9dbe4d05d277fe15244b437f66bc34b9ca9aa2e63b400721fa692488b84e1\": container with ID starting with 5ee9dbe4d05d277fe15244b437f66bc34b9ca9aa2e63b400721fa692488b84e1 not found: ID does not exist" Oct 30 04:04:42.922253 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.922229 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:42.926461 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:42.926439 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:43.119691 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:43.119657 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0df97dd1-be9a-43e3-b52a-cad79d3192f3" path="/var/lib/kubelet/pods/0df97dd1-be9a-43e3-b52a-cad79d3192f3/volumes" Oct 30 04:04:43.120163 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:43.120125 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" path="/var/lib/kubelet/pods/7f3a5697-5bf8-4f0e-8017-e36d0738bbc6/volumes" Oct 30 04:04:47.382898 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:47.382855 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="8bf40867-6019-4c8c-9488-b9ed3cff2efb" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.104:9093/-/ready\": dial tcp 10.132.0.104:9093: connect: connection refused" Oct 30 04:04:48.160655 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:04:48.160616 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:04:50.915058 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:50.915009 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"8bf40867-6019-4c8c-9488-b9ed3cff2efb","Type":"ContainerDied","Data":"571a72bcf474a67ce31ee35b1cf1829a4ccae217b602704a1056a27f3f5a9c7b"} Oct 30 04:04:50.915426 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:50.915092 4925 generic.go:356] "Generic (PLEG): container finished" podID="8bf40867-6019-4c8c-9488-b9ed3cff2efb" containerID="571a72bcf474a67ce31ee35b1cf1829a4ccae217b602704a1056a27f3f5a9c7b" exitCode=0 Oct 30 04:04:50.929036 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:50.928975 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:51.025559 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.025522 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-config-volume\" (UniqueName: \"kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-config-volume\") pod \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " Oct 30 04:04:51.025781 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.025563 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-web-config\" (UniqueName: \"kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-web-config\") pod \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " Oct 30 04:04:51.025781 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.025599 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-cluster-tls-config\") pod \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " Oct 30 04:04:51.025781 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.025625 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf40867-6019-4c8c-9488-b9ed3cff2efb-kube-api-access-wrz2p\" (UniqueName: \"kubernetes.io/projected/8bf40867-6019-4c8c-9488-b9ed3cff2efb-kube-api-access-wrz2p\") pod \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " Oct 30 04:04:51.025781 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.025658 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf40867-6019-4c8c-9488-b9ed3cff2efb-tls-assets\" (UniqueName: \"kubernetes.io/projected/8bf40867-6019-4c8c-9488-b9ed3cff2efb-tls-assets\") pod \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " Oct 30 04:04:51.025781 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.025692 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf40867-6019-4c8c-9488-b9ed3cff2efb-config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bf40867-6019-4c8c-9488-b9ed3cff2efb-config-out\") pod \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " Oct 30 04:04:51.025781 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.025731 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf40867-6019-4c8c-9488-b9ed3cff2efb-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/8bf40867-6019-4c8c-9488-b9ed3cff2efb-alertmanager-data-science-monitoringstack-db\") pod \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\" (UID: \"8bf40867-6019-4c8c-9488-b9ed3cff2efb\") " Oct 30 04:04:51.026233 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.026203 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf40867-6019-4c8c-9488-b9ed3cff2efb-alertmanager-data-science-monitoringstack-db" pod "8bf40867-6019-4c8c-9488-b9ed3cff2efb" (UID: "8bf40867-6019-4c8c-9488-b9ed3cff2efb"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:04:51.027965 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.027933 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-config-volume" pod "8bf40867-6019-4c8c-9488-b9ed3cff2efb" (UID: "8bf40867-6019-4c8c-9488-b9ed3cff2efb"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:51.028084 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.028036 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf40867-6019-4c8c-9488-b9ed3cff2efb-config-out" pod "8bf40867-6019-4c8c-9488-b9ed3cff2efb" (UID: "8bf40867-6019-4c8c-9488-b9ed3cff2efb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:04:51.028274 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.028246 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf40867-6019-4c8c-9488-b9ed3cff2efb-tls-assets" pod "8bf40867-6019-4c8c-9488-b9ed3cff2efb" (UID: "8bf40867-6019-4c8c-9488-b9ed3cff2efb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:51.028344 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.028279 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf40867-6019-4c8c-9488-b9ed3cff2efb-kube-api-access-wrz2p" pod "8bf40867-6019-4c8c-9488-b9ed3cff2efb" (UID: "8bf40867-6019-4c8c-9488-b9ed3cff2efb"). InnerVolumeSpecName "kube-api-access-wrz2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:04:51.033704 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.033678 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-cluster-tls-config" pod "8bf40867-6019-4c8c-9488-b9ed3cff2efb" (UID: "8bf40867-6019-4c8c-9488-b9ed3cff2efb"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:51.040745 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.040721 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-web-config" pod "8bf40867-6019-4c8c-9488-b9ed3cff2efb" (UID: "8bf40867-6019-4c8c-9488-b9ed3cff2efb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:04:51.055011 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.054985 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b"] Oct 30 04:04:51.055315 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055303 4925 cpu_manager.go:414] "containerMap: removing container" podUID="776067d7-1633-44ae-986c-56cafc4be4c4" containerName="kube-rbac-proxy" Oct 30 04:04:51.055359 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055315 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="776067d7-1633-44ae-986c-56cafc4be4c4" containerName="kube-rbac-proxy" Oct 30 04:04:51.055359 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055332 4925 cpu_manager.go:414] "containerMap: removing container" podUID="b5e58337-f692-4b01-b15d-7642a69d67f2" containerName="korrel8r" Oct 30 04:04:51.055359 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055337 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5e58337-f692-4b01-b15d-7642a69d67f2" containerName="korrel8r" Oct 30 04:04:51.055359 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055351 4925 cpu_manager.go:414] "containerMap: removing container" podUID="7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" containerName="init-config-reloader" Oct 30 04:04:51.055359 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055355 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" containerName="init-config-reloader" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055362 4925 cpu_manager.go:414] "containerMap: removing container" podUID="7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" containerName="config-reloader" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055366 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" containerName="config-reloader" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055374 4925 cpu_manager.go:414] "containerMap: removing container" podUID="7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" containerName="prometheus" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055378 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" containerName="prometheus" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055387 4925 cpu_manager.go:414] "containerMap: removing container" podUID="8bf40867-6019-4c8c-9488-b9ed3cff2efb" containerName="init-config-reloader" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055392 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf40867-6019-4c8c-9488-b9ed3cff2efb" containerName="init-config-reloader" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055403 4925 cpu_manager.go:414] "containerMap: removing container" podUID="8bf40867-6019-4c8c-9488-b9ed3cff2efb" containerName="config-reloader" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055408 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf40867-6019-4c8c-9488-b9ed3cff2efb" containerName="config-reloader" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055418 4925 cpu_manager.go:414] "containerMap: removing container" podUID="0df97dd1-be9a-43e3-b52a-cad79d3192f3" containerName="otc-container" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055422 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0df97dd1-be9a-43e3-b52a-cad79d3192f3" containerName="otc-container" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055429 4925 cpu_manager.go:414] "containerMap: removing container" podUID="7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" containerName="thanos-sidecar" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055434 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" containerName="thanos-sidecar" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055441 4925 cpu_manager.go:414] "containerMap: removing container" podUID="8bf40867-6019-4c8c-9488-b9ed3cff2efb" containerName="alertmanager" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055445 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf40867-6019-4c8c-9488-b9ed3cff2efb" containerName="alertmanager" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055492 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" containerName="prometheus" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055502 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf40867-6019-4c8c-9488-b9ed3cff2efb" containerName="config-reloader" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055507 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="0df97dd1-be9a-43e3-b52a-cad79d3192f3" containerName="otc-container" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055513 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" containerName="thanos-sidecar" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055517 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf40867-6019-4c8c-9488-b9ed3cff2efb" containerName="alertmanager" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055522 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="776067d7-1633-44ae-986c-56cafc4be4c4" containerName="kube-rbac-proxy" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055527 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5e58337-f692-4b01-b15d-7642a69d67f2" containerName="korrel8r" Oct 30 04:04:51.055531 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.055535 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f3a5697-5bf8-4f0e-8017-e36d0738bbc6" containerName="config-reloader" Oct 30 04:04:51.058799 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.058780 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:04:51.077339 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.077310 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 04:04:51.077339 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.077325 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 04:04:51.077339 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.077317 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-v4np9\"" Oct 30 04:04:51.077546 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.077416 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 04:04:51.083653 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.083579 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 04:04:51.086035 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.086009 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b"] Oct 30 04:04:51.126532 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.126502 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmvz6\" (UniqueName: \"kubernetes.io/projected/0aefabe9-0871-4f53-a299-22befa91ebf9-kube-api-access-lmvz6\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-xcp6b\" (UID: \"0aefabe9-0871-4f53-a299-22befa91ebf9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:04:51.126711 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.126544 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-xcp6b\" (UID: \"0aefabe9-0871-4f53-a299-22befa91ebf9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:04:51.126711 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.126569 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-xcp6b\" (UID: \"0aefabe9-0871-4f53-a299-22befa91ebf9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:04:51.126711 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.126665 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/0aefabe9-0871-4f53-a299-22befa91ebf9-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-xcp6b\" (UID: \"0aefabe9-0871-4f53-a299-22befa91ebf9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:04:51.126711 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.126712 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-xcp6b\" (UID: \"0aefabe9-0871-4f53-a299-22befa91ebf9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:04:51.126994 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.126812 4925 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/8bf40867-6019-4c8c-9488-b9ed3cff2efb-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:51.126994 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.126826 4925 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-config-volume\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:51.126994 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.126836 4925 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-web-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:51.126994 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.126844 4925 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8bf40867-6019-4c8c-9488-b9ed3cff2efb-cluster-tls-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:51.126994 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.126853 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrz2p\" (UniqueName: \"kubernetes.io/projected/8bf40867-6019-4c8c-9488-b9ed3cff2efb-kube-api-access-wrz2p\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:51.126994 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.126862 4925 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bf40867-6019-4c8c-9488-b9ed3cff2efb-tls-assets\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:51.126994 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.126895 4925 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bf40867-6019-4c8c-9488-b9ed3cff2efb-config-out\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:04:51.186813 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.186718 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f"] Oct 30 04:04:51.190269 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.190250 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" Oct 30 04:04:51.209798 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.209741 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f"] Oct 30 04:04:51.227392 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.227363 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/806f5c61-9a5f-487a-94ec-e3577431dd83-config\") pod \"korrel8r-5cbcc89b8f-jp42f\" (UID: \"806f5c61-9a5f-487a-94ec-e3577431dd83\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" Oct 30 04:04:51.227565 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.227411 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmvz6\" (UniqueName: \"kubernetes.io/projected/0aefabe9-0871-4f53-a299-22befa91ebf9-kube-api-access-lmvz6\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-xcp6b\" (UID: \"0aefabe9-0871-4f53-a299-22befa91ebf9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:04:51.227565 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.227474 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/806f5c61-9a5f-487a-94ec-e3577431dd83-tls\") pod \"korrel8r-5cbcc89b8f-jp42f\" (UID: \"806f5c61-9a5f-487a-94ec-e3577431dd83\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" Oct 30 04:04:51.227565 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.227506 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvbqm\" (UniqueName: \"kubernetes.io/projected/806f5c61-9a5f-487a-94ec-e3577431dd83-kube-api-access-hvbqm\") pod \"korrel8r-5cbcc89b8f-jp42f\" (UID: \"806f5c61-9a5f-487a-94ec-e3577431dd83\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" Oct 30 04:04:51.227565 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.227541 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-xcp6b\" (UID: \"0aefabe9-0871-4f53-a299-22befa91ebf9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:04:51.227828 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.227686 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-xcp6b\" (UID: \"0aefabe9-0871-4f53-a299-22befa91ebf9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:04:51.227828 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.227715 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/0aefabe9-0871-4f53-a299-22befa91ebf9-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-xcp6b\" (UID: \"0aefabe9-0871-4f53-a299-22befa91ebf9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:04:51.227828 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.227740 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-xcp6b\" (UID: \"0aefabe9-0871-4f53-a299-22befa91ebf9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:04:51.228637 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.228496 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/0aefabe9-0871-4f53-a299-22befa91ebf9-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-xcp6b\" (UID: \"0aefabe9-0871-4f53-a299-22befa91ebf9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:04:51.230457 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.230433 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-xcp6b\" (UID: \"0aefabe9-0871-4f53-a299-22befa91ebf9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:04:51.230571 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.230457 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-xcp6b\" (UID: \"0aefabe9-0871-4f53-a299-22befa91ebf9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:04:51.230571 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.230530 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-xcp6b\" (UID: \"0aefabe9-0871-4f53-a299-22befa91ebf9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:04:51.251337 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.251314 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmvz6\" (UniqueName: \"kubernetes.io/projected/0aefabe9-0871-4f53-a299-22befa91ebf9-kube-api-access-lmvz6\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-xcp6b\" (UID: \"0aefabe9-0871-4f53-a299-22befa91ebf9\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:04:51.277410 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.277377 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 04:04:51.277410 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.277405 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-75zww\"" Oct 30 04:04:51.277562 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.277408 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 04:04:51.328210 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.328177 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/806f5c61-9a5f-487a-94ec-e3577431dd83-config\") pod \"korrel8r-5cbcc89b8f-jp42f\" (UID: \"806f5c61-9a5f-487a-94ec-e3577431dd83\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" Oct 30 04:04:51.328392 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.328222 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/806f5c61-9a5f-487a-94ec-e3577431dd83-tls\") pod \"korrel8r-5cbcc89b8f-jp42f\" (UID: \"806f5c61-9a5f-487a-94ec-e3577431dd83\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" Oct 30 04:04:51.328392 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.328248 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvbqm\" (UniqueName: \"kubernetes.io/projected/806f5c61-9a5f-487a-94ec-e3577431dd83-kube-api-access-hvbqm\") pod \"korrel8r-5cbcc89b8f-jp42f\" (UID: \"806f5c61-9a5f-487a-94ec-e3577431dd83\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" Oct 30 04:04:51.328392 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:04:51.328336 4925 secret.go:189] Couldn't get secret redhat-ods-monitoring/korrel8r-tls: secret "korrel8r-tls" not found Oct 30 04:04:51.328392 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:04:51.328393 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/806f5c61-9a5f-487a-94ec-e3577431dd83-tls podName:806f5c61-9a5f-487a-94ec-e3577431dd83 nodeName:}" failed. No retries permitted until 2026-10-30 04:04:51.828375681 +0000 UTC m=+5052.841653762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/806f5c61-9a5f-487a-94ec-e3577431dd83-tls") pod "korrel8r-5cbcc89b8f-jp42f" (UID: "806f5c61-9a5f-487a-94ec-e3577431dd83") : secret "korrel8r-tls" not found Oct 30 04:04:51.329087 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.329064 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/806f5c61-9a5f-487a-94ec-e3577431dd83-config\") pod \"korrel8r-5cbcc89b8f-jp42f\" (UID: \"806f5c61-9a5f-487a-94ec-e3577431dd83\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" Oct 30 04:04:51.352132 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.352109 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvbqm\" (UniqueName: \"kubernetes.io/projected/806f5c61-9a5f-487a-94ec-e3577431dd83-kube-api-access-hvbqm\") pod \"korrel8r-5cbcc89b8f-jp42f\" (UID: \"806f5c61-9a5f-487a-94ec-e3577431dd83\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" Oct 30 04:04:51.370961 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.370936 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:04:51.671569 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.671534 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b"] Oct 30 04:04:51.674169 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:04:51.674138 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aefabe9_0871_4f53_a299_22befa91ebf9.slice/crio-db8df5025f9e11aefd48445bd6fd65b9af8f5049250bc5b1cee577f866563a4e WatchSource:0}: Error finding container db8df5025f9e11aefd48445bd6fd65b9af8f5049250bc5b1cee577f866563a4e: Status 404 returned error can't find the container with id db8df5025f9e11aefd48445bd6fd65b9af8f5049250bc5b1cee577f866563a4e Oct 30 04:04:51.832301 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.832272 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/806f5c61-9a5f-487a-94ec-e3577431dd83-tls\") pod \"korrel8r-5cbcc89b8f-jp42f\" (UID: \"806f5c61-9a5f-487a-94ec-e3577431dd83\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" Oct 30 04:04:51.834802 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.834766 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/806f5c61-9a5f-487a-94ec-e3577431dd83-tls\") pod \"korrel8r-5cbcc89b8f-jp42f\" (UID: \"806f5c61-9a5f-487a-94ec-e3577431dd83\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" Oct 30 04:04:51.922962 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.922881 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:51.922962 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.922891 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"8bf40867-6019-4c8c-9488-b9ed3cff2efb","Type":"ContainerDied","Data":"9822a8e9a965693da09d7951521a43a5a16a056328338dbcd646fd98ba6fea28"} Oct 30 04:04:51.923520 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.922972 4925 scope.go:122] "RemoveContainer" containerID="f81733c84e59f8faea08940e57c0da664b647cc0bf6b2295738950944baa9ec7" Oct 30 04:04:51.924309 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.924233 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" event={"ID":"0aefabe9-0871-4f53-a299-22befa91ebf9","Type":"ContainerStarted","Data":"db8df5025f9e11aefd48445bd6fd65b9af8f5049250bc5b1cee577f866563a4e"} Oct 30 04:04:51.931405 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.931387 4925 scope.go:122] "RemoveContainer" containerID="571a72bcf474a67ce31ee35b1cf1829a4ccae217b602704a1056a27f3f5a9c7b" Oct 30 04:04:51.939199 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.939181 4925 scope.go:122] "RemoveContainer" containerID="d352db5b2a093942154c4f6c978d81459212200c33195bfeff8d8fe75c903135" Oct 30 04:04:51.977621 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.977582 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:04:51.983895 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:51.983869 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:04:52.102357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.102312 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" Oct 30 04:04:52.403661 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.403637 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f"] Oct 30 04:04:52.405814 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:04:52.405784 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806f5c61_9a5f_487a_94ec_e3577431dd83.slice/crio-48be7f77a11a207b23ad66249dcdc2138bde5128c610e0150457833dfb886dd7 WatchSource:0}: Error finding container 48be7f77a11a207b23ad66249dcdc2138bde5128c610e0150457833dfb886dd7: Status 404 returned error can't find the container with id 48be7f77a11a207b23ad66249dcdc2138bde5128c610e0150457833dfb886dd7 Oct 30 04:04:52.733520 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.733425 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:52.737286 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.737262 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:52.759337 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.756556 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-js864\"" Oct 30 04:04:52.759337 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.756720 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:04:52.759337 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.756852 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:04:52.759337 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.757016 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:04:52.759337 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.757127 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:04:52.759779 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.759724 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:52.840724 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.840692 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:52.840888 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.840735 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:52.840888 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.840872 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a1e45ba0-0ae1-471a-a337-eed3d34ac818-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:52.840888 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.840905 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfdtg\" (UniqueName: \"kubernetes.io/projected/a1e45ba0-0ae1-471a-a337-eed3d34ac818-kube-api-access-pfdtg\") pod \"data-science-collector-collector-0\" (UID: \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:52.841073 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.840933 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:52.932392 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.932357 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" event={"ID":"0aefabe9-0871-4f53-a299-22befa91ebf9","Type":"ContainerStarted","Data":"eb9e4cb6876b84279c42b827e829d8fba21111c9900297382c79eb96045a4cb4"} Oct 30 04:04:52.932820 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.932441 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:04:52.933967 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.933945 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" event={"ID":"806f5c61-9a5f-487a-94ec-e3577431dd83","Type":"ContainerStarted","Data":"c875a010f990d2b0c541ec88b92b49c86448c59235febf949cb56695f603329e"} Oct 30 04:04:52.934046 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.933980 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" event={"ID":"806f5c61-9a5f-487a-94ec-e3577431dd83","Type":"ContainerStarted","Data":"48be7f77a11a207b23ad66249dcdc2138bde5128c610e0150457833dfb886dd7"} Oct 30 04:04:52.934046 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.934017 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" Oct 30 04:04:52.941439 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.941415 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a1e45ba0-0ae1-471a-a337-eed3d34ac818-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:52.941552 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.941449 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfdtg\" (UniqueName: \"kubernetes.io/projected/a1e45ba0-0ae1-471a-a337-eed3d34ac818-kube-api-access-pfdtg\") pod \"data-science-collector-collector-0\" (UID: \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:52.941552 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.941477 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:52.941552 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.941523 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:52.941552 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.941546 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:52.942185 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.942164 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a1e45ba0-0ae1-471a-a337-eed3d34ac818-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:52.944523 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.944502 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:52.944677 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.944537 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:52.946508 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.946484 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:52.965953 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.965928 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfdtg\" (UniqueName: \"kubernetes.io/projected/a1e45ba0-0ae1-471a-a337-eed3d34ac818-kube-api-access-pfdtg\") pod \"data-science-collector-collector-0\" (UID: \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:52.981278 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:52.981233 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" podStartSLOduration=1.981219834 podStartE2EDuration="1.981219834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:52.980099192 +0000 UTC m=+5053.993377295" watchObservedRunningTime="2026-10-30 04:04:52.981219834 +0000 UTC m=+5053.994497937" Oct 30 04:04:53.022862 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:53.022776 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" podStartSLOduration=2.022764247 podStartE2EDuration="2.022764247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:53.02180968 +0000 UTC m=+5054.035087783" watchObservedRunningTime="2026-10-30 04:04:53.022764247 +0000 UTC m=+5054.036042341" Oct 30 04:04:53.051249 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:53.051224 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:04:53.120340 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:53.120313 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf40867-6019-4c8c-9488-b9ed3cff2efb" path="/var/lib/kubelet/pods/8bf40867-6019-4c8c-9488-b9ed3cff2efb/volumes" Oct 30 04:04:53.351985 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:53.351897 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:04:53.354167 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:04:53.354136 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e45ba0_0ae1_471a_a337_eed3d34ac818.slice/crio-5017d589903dabe6c48e896b2a0708e6fb8898437c5c107358cb6f62a9bb6e6c WatchSource:0}: Error finding container 5017d589903dabe6c48e896b2a0708e6fb8898437c5c107358cb6f62a9bb6e6c: Status 404 returned error can't find the container with id 5017d589903dabe6c48e896b2a0708e6fb8898437c5c107358cb6f62a9bb6e6c Oct 30 04:04:53.356353 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:53.356321 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:53.356467 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:53.356391 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:04:53.940869 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:53.940833 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a1e45ba0-0ae1-471a-a337-eed3d34ac818","Type":"ContainerStarted","Data":"17f26278ee92d5be63edaa68261d6102410594e73010debc34995b0091c4cdd0"} Oct 30 04:04:53.941347 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:53.940890 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a1e45ba0-0ae1-471a-a337-eed3d34ac818","Type":"ContainerStarted","Data":"5017d589903dabe6c48e896b2a0708e6fb8898437c5c107358cb6f62a9bb6e6c"} Oct 30 04:04:53.986216 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:53.986156 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.9861371669999999 podStartE2EDuration="1.986137167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:53.985299802 +0000 UTC m=+5054.998577909" watchObservedRunningTime="2026-10-30 04:04:53.986137167 +0000 UTC m=+5054.999415274" Oct 30 04:04:56.475481 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.475444 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:04:56.479883 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.479852 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.495806 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.495780 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 04:04:56.495930 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.495821 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 04:04:56.507705 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.507664 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-8ntbt\"" Oct 30 04:04:56.509835 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.509809 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 04:04:56.524026 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.523990 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:04:56.524185 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.524134 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.540006 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.539978 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 04:04:56.543236 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.543183 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 04:04:56.543236 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.543254 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 04:04:56.543635 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.543394 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 04:04:56.573777 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.573734 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6654a23a-7079-4f43-8c52-0e07e08bdd90-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.573919 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.573824 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6654a23a-7079-4f43-8c52-0e07e08bdd90-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.573919 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.573853 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.573919 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.573901 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.574048 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.573918 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.574048 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.573940 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmg8t\" (UniqueName: \"kubernetes.io/projected/6654a23a-7079-4f43-8c52-0e07e08bdd90-kube-api-access-nmg8t\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.574048 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.574031 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.574174 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.574074 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/97f68790-292e-4efa-9396-3c4ede68c3bb-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.574174 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.574101 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpsxl\" (UniqueName: \"kubernetes.io/projected/97f68790-292e-4efa-9396-3c4ede68c3bb-kube-api-access-dpsxl\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.574174 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.574153 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.574350 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.574194 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.574350 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.574267 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.574350 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.574323 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.574499 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.574359 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.574499 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.574376 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97f68790-292e-4efa-9396-3c4ede68c3bb-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.574499 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.574467 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.574623 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.574504 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.574623 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.574530 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97f68790-292e-4efa-9396-3c4ede68c3bb-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.574623 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.574561 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.596229 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.596199 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 04:04:56.596434 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.596407 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 04:04:56.596434 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.596431 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 04:04:56.596606 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.596422 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:04:56.596606 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.596454 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 04:04:56.596606 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.596455 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 04:04:56.596606 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.596593 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:04:56.596796 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.596780 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-s78sv\"" Oct 30 04:04:56.675739 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.675696 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.675982 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.675809 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6654a23a-7079-4f43-8c52-0e07e08bdd90-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.675982 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.675838 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6654a23a-7079-4f43-8c52-0e07e08bdd90-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.675982 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.675863 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.675982 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.675888 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.675982 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.675911 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.675982 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.675962 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmg8t\" (UniqueName: \"kubernetes.io/projected/6654a23a-7079-4f43-8c52-0e07e08bdd90-kube-api-access-nmg8t\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.676338 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.675998 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.676338 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.676027 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/97f68790-292e-4efa-9396-3c4ede68c3bb-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.676338 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.676052 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpsxl\" (UniqueName: \"kubernetes.io/projected/97f68790-292e-4efa-9396-3c4ede68c3bb-kube-api-access-dpsxl\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.676338 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.676084 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.676338 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.676106 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.676338 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.676134 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.676338 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.676163 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.676338 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.676204 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.676338 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.676231 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97f68790-292e-4efa-9396-3c4ede68c3bb-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.676338 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.676288 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.676338 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.676318 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.676953 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.676347 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97f68790-292e-4efa-9396-3c4ede68c3bb-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.677277 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.677245 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.677395 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.677337 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/97f68790-292e-4efa-9396-3c4ede68c3bb-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.677715 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.677686 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.679089 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.679040 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.679200 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.679111 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.680057 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.680030 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.680926 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.680491 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6654a23a-7079-4f43-8c52-0e07e08bdd90-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.680926 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.680505 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.680926 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.680713 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6654a23a-7079-4f43-8c52-0e07e08bdd90-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.680926 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.680884 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97f68790-292e-4efa-9396-3c4ede68c3bb-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.681223 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.680992 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.681338 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.681307 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.681462 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.681444 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97f68790-292e-4efa-9396-3c4ede68c3bb-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.681699 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.681681 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.682430 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.682407 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.682552 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.682536 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.682595 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.682580 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.700621 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.700596 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmg8t\" (UniqueName: \"kubernetes.io/projected/6654a23a-7079-4f43-8c52-0e07e08bdd90-kube-api-access-nmg8t\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:04:56.700718 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.700686 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpsxl\" (UniqueName: \"kubernetes.io/projected/97f68790-292e-4efa-9396-3c4ede68c3bb-kube-api-access-dpsxl\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:56.798740 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:56.798650 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:04:57.150191 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:57.150094 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:04:57.152258 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:04:57.152229 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f68790_292e_4efa_9396_3c4ede68c3bb.slice/crio-65dbf1dbe9a1705534c81d3d873b1216bfc509e4dd70f4f92d1f977a6d7a2db6 WatchSource:0}: Error finding container 65dbf1dbe9a1705534c81d3d873b1216bfc509e4dd70f4f92d1f977a6d7a2db6: Status 404 returned error can't find the container with id 65dbf1dbe9a1705534c81d3d873b1216bfc509e4dd70f4f92d1f977a6d7a2db6 Oct 30 04:04:57.962041 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:57.962001 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"97f68790-292e-4efa-9396-3c4ede68c3bb","Type":"ContainerStarted","Data":"6126c7c3933a98d99eb161502058bdf72a8c01b5aad360ed4a09a18b3fa4445c"} Oct 30 04:04:57.962565 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:57.962068 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"97f68790-292e-4efa-9396-3c4ede68c3bb","Type":"ContainerStarted","Data":"65dbf1dbe9a1705534c81d3d873b1216bfc509e4dd70f4f92d1f977a6d7a2db6"} Oct 30 04:04:58.943958 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:04:58.943925 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:05:00.212972 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:00.212938 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:05:00.242834 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:00.242801 4925 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:05:00.313989 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:00.313948 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:05:00.314142 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:00.314087 4925 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:05:00.328884 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:00.328853 4925 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") DevicePath \"csi-77aa142b6cc95d8f79f4aef80349e4d929aed6d462958597286d83ed1cc8fc2c\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:05:00.429844 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:00.429809 4925 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7bd02a6ced0e86eb5067dae735620268cfbf25efbbfc3866c7163efbdf26dd99/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:05:00.458678 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:00.458650 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:05:00.745845 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:00.745808 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:05:00.978267 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:00.978189 4925 generic.go:356] "Generic (PLEG): container finished" podID="97f68790-292e-4efa-9396-3c4ede68c3bb" containerID="6126c7c3933a98d99eb161502058bdf72a8c01b5aad360ed4a09a18b3fa4445c" exitCode=0 Oct 30 04:05:00.978267 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:00.978235 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"97f68790-292e-4efa-9396-3c4ede68c3bb","Type":"ContainerDied","Data":"6126c7c3933a98d99eb161502058bdf72a8c01b5aad360ed4a09a18b3fa4445c"} Oct 30 04:05:01.077823 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:01.077793 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 04:05:01.079212 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:05:01.079189 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6654a23a_7079_4f43_8c52_0e07e08bdd90.slice/crio-0047a7d26ea9e110484b09e70f95455b3534bd1f7f6416803d45b228a997cf1c WatchSource:0}: Error finding container 0047a7d26ea9e110484b09e70f95455b3534bd1f7f6416803d45b228a997cf1c: Status 404 returned error can't find the container with id 0047a7d26ea9e110484b09e70f95455b3534bd1f7f6416803d45b228a997cf1c Oct 30 04:05:01.985765 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:01.985725 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"97f68790-292e-4efa-9396-3c4ede68c3bb","Type":"ContainerStarted","Data":"0ce31577a2679abe8b867430db4f3dff3372172b4f66566ef09e56b8da031bff"} Oct 30 04:05:01.986231 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:01.985810 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"97f68790-292e-4efa-9396-3c4ede68c3bb","Type":"ContainerStarted","Data":"34c12c4161c47fef92a9f1e5b580c30040b32bd208808ce9e4f1c69adfcf4e70"} Oct 30 04:05:01.986231 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:01.986003 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:05:01.987308 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:01.987283 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"6654a23a-7079-4f43-8c52-0e07e08bdd90","Type":"ContainerStarted","Data":"96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f"} Oct 30 04:05:01.987463 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:01.987343 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"6654a23a-7079-4f43-8c52-0e07e08bdd90","Type":"ContainerStarted","Data":"0047a7d26ea9e110484b09e70f95455b3534bd1f7f6416803d45b228a997cf1c"} Oct 30 04:05:02.059356 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:02.059307 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=2.973745706 podStartE2EDuration="6.059292894s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.085547188s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:05:02.045681411 +0000 UTC m=+5063.058959516" watchObservedRunningTime="2026-10-30 04:05:02.059292894 +0000 UTC m=+5063.072571029" Oct 30 04:05:03.942922 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:03.942893 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" Oct 30 04:05:06.009413 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:06.009378 4925 generic.go:356] "Generic (PLEG): container finished" podID="6654a23a-7079-4f43-8c52-0e07e08bdd90" containerID="96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f" exitCode=0 Oct 30 04:05:06.009823 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:06.009442 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"6654a23a-7079-4f43-8c52-0e07e08bdd90","Type":"ContainerDied","Data":"96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f"} Oct 30 04:05:07.016834 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:07.016795 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"6654a23a-7079-4f43-8c52-0e07e08bdd90","Type":"ContainerStarted","Data":"1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac"} Oct 30 04:05:07.017269 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:07.016895 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"6654a23a-7079-4f43-8c52-0e07e08bdd90","Type":"ContainerStarted","Data":"0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b"} Oct 30 04:05:07.017269 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:07.016920 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"6654a23a-7079-4f43-8c52-0e07e08bdd90","Type":"ContainerStarted","Data":"73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a"} Oct 30 04:05:07.094982 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:07.094922 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=7.608142904 podStartE2EDuration="11.09490939s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.486766486s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:05:07.094283586 +0000 UTC m=+5068.107561683" watchObservedRunningTime="2026-10-30 04:05:07.09490939 +0000 UTC m=+5068.108187486" Oct 30 04:05:07.994433 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:07.994398 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:05:10.746347 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:10.746308 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:05:15.746916 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:15.746875 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:05:15.752719 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:15.752692 4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:05:16.073123 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:16.073034 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:05:18.161161 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:05:18.161126 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:05:39.646644 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:39.646548 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 04:05:39.655257 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:39.655236 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 04:05:39.661856 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:39.661837 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 04:05:39.670112 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:05:39.670095 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 04:05:48.163615 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:05:48.163575 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:06:00.889152 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:00.889118 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b"] Oct 30 04:06:00.889584 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:00.889305 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" podUID="0aefabe9-0871-4f53-a299-22befa91ebf9" containerName="kube-rbac-proxy" containerID="cri-o://eb9e4cb6876b84279c42b827e829d8fba21111c9900297382c79eb96045a4cb4" gracePeriod=30 Oct 30 04:06:00.942546 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:00.942491 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f"] Oct 30 04:06:00.942725 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:00.942699 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" podUID="806f5c61-9a5f-487a-94ec-e3577431dd83" containerName="korrel8r" containerID="cri-o://c875a010f990d2b0c541ec88b92b49c86448c59235febf949cb56695f603329e" gracePeriod=30 Oct 30 04:06:01.221593 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.221280 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:06:01.230898 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.230875 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" Oct 30 04:06:01.255523 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.255494 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aefabe9-0871-4f53-a299-22befa91ebf9-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/0aefabe9-0871-4f53-a299-22befa91ebf9-prometheus-ca\") pod \"0aefabe9-0871-4f53-a299-22befa91ebf9\" (UID: \"0aefabe9-0871-4f53-a299-22befa91ebf9\") " Oct 30 04:06:01.255523 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.255535 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-prometheus-client-cert\") pod \"0aefabe9-0871-4f53-a299-22befa91ebf9\" (UID: \"0aefabe9-0871-4f53-a299-22befa91ebf9\") " Oct 30 04:06:01.255832 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.255573 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-tls\" (UniqueName: \"kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-tls\") pod \"0aefabe9-0871-4f53-a299-22befa91ebf9\" (UID: \"0aefabe9-0871-4f53-a299-22befa91ebf9\") " Oct 30 04:06:01.255832 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.255642 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/806f5c61-9a5f-487a-94ec-e3577431dd83-tls\" (UniqueName: \"kubernetes.io/secret/806f5c61-9a5f-487a-94ec-e3577431dd83-tls\") pod \"806f5c61-9a5f-487a-94ec-e3577431dd83\" (UID: \"806f5c61-9a5f-487a-94ec-e3577431dd83\") " Oct 30 04:06:01.255832 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.255723 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/806f5c61-9a5f-487a-94ec-e3577431dd83-config\" (UniqueName: \"kubernetes.io/configmap/806f5c61-9a5f-487a-94ec-e3577431dd83-config\") pod \"806f5c61-9a5f-487a-94ec-e3577431dd83\" (UID: \"806f5c61-9a5f-487a-94ec-e3577431dd83\") " Oct 30 04:06:01.255832 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.255766 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806f5c61-9a5f-487a-94ec-e3577431dd83-kube-api-access-hvbqm\" (UniqueName: \"kubernetes.io/projected/806f5c61-9a5f-487a-94ec-e3577431dd83-kube-api-access-hvbqm\") pod \"806f5c61-9a5f-487a-94ec-e3577431dd83\" (UID: \"806f5c61-9a5f-487a-94ec-e3577431dd83\") " Oct 30 04:06:01.255832 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.255804 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aefabe9-0871-4f53-a299-22befa91ebf9-kube-api-access-lmvz6\" (UniqueName: \"kubernetes.io/projected/0aefabe9-0871-4f53-a299-22befa91ebf9-kube-api-access-lmvz6\") pod \"0aefabe9-0871-4f53-a299-22befa91ebf9\" (UID: \"0aefabe9-0871-4f53-a299-22befa91ebf9\") " Oct 30 04:06:01.256092 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.255851 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-kube-rbac-proxy-config\") pod \"0aefabe9-0871-4f53-a299-22befa91ebf9\" (UID: \"0aefabe9-0871-4f53-a299-22befa91ebf9\") " Oct 30 04:06:01.256092 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.255946 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aefabe9-0871-4f53-a299-22befa91ebf9-prometheus-ca" pod "0aefabe9-0871-4f53-a299-22befa91ebf9" (UID: "0aefabe9-0871-4f53-a299-22befa91ebf9"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:01.256203 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.256122 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/0aefabe9-0871-4f53-a299-22befa91ebf9-prometheus-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:01.257223 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.257177 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806f5c61-9a5f-487a-94ec-e3577431dd83-config" pod "806f5c61-9a5f-487a-94ec-e3577431dd83" (UID: "806f5c61-9a5f-487a-94ec-e3577431dd83"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:01.259591 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.259565 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-prometheus-client-cert" pod "0aefabe9-0871-4f53-a299-22befa91ebf9" (UID: "0aefabe9-0871-4f53-a299-22befa91ebf9"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:01.260341 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.260313 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aefabe9-0871-4f53-a299-22befa91ebf9-kube-api-access-lmvz6" pod "0aefabe9-0871-4f53-a299-22befa91ebf9" (UID: "0aefabe9-0871-4f53-a299-22befa91ebf9"). InnerVolumeSpecName "kube-api-access-lmvz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:01.261238 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.261211 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-tls" pod "0aefabe9-0871-4f53-a299-22befa91ebf9" (UID: "0aefabe9-0871-4f53-a299-22befa91ebf9"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:01.261460 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.261442 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/806f5c61-9a5f-487a-94ec-e3577431dd83-tls" pod "806f5c61-9a5f-487a-94ec-e3577431dd83" (UID: "806f5c61-9a5f-487a-94ec-e3577431dd83"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:01.261534 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.261511 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-kube-rbac-proxy-config" pod "0aefabe9-0871-4f53-a299-22befa91ebf9" (UID: "0aefabe9-0871-4f53-a299-22befa91ebf9"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:01.262982 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.262962 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806f5c61-9a5f-487a-94ec-e3577431dd83-kube-api-access-hvbqm" pod "806f5c61-9a5f-487a-94ec-e3577431dd83" (UID: "806f5c61-9a5f-487a-94ec-e3577431dd83"). InnerVolumeSpecName "kube-api-access-hvbqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:01.298117 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.298083 4925 generic.go:356] "Generic (PLEG): container finished" podID="0aefabe9-0871-4f53-a299-22befa91ebf9" containerID="eb9e4cb6876b84279c42b827e829d8fba21111c9900297382c79eb96045a4cb4" exitCode=0 Oct 30 04:06:01.298290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.298162 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" Oct 30 04:06:01.298290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.298183 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" event={"ID":"0aefabe9-0871-4f53-a299-22befa91ebf9","Type":"ContainerDied","Data":"eb9e4cb6876b84279c42b827e829d8fba21111c9900297382c79eb96045a4cb4"} Oct 30 04:06:01.298290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.298219 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b" event={"ID":"0aefabe9-0871-4f53-a299-22befa91ebf9","Type":"ContainerDied","Data":"db8df5025f9e11aefd48445bd6fd65b9af8f5049250bc5b1cee577f866563a4e"} Oct 30 04:06:01.298290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.298243 4925 scope.go:122] "RemoveContainer" containerID="eb9e4cb6876b84279c42b827e829d8fba21111c9900297382c79eb96045a4cb4" Oct 30 04:06:01.300215 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.300186 4925 generic.go:356] "Generic (PLEG): container finished" podID="806f5c61-9a5f-487a-94ec-e3577431dd83" containerID="c875a010f990d2b0c541ec88b92b49c86448c59235febf949cb56695f603329e" exitCode=0 Oct 30 04:06:01.300346 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.300233 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" Oct 30 04:06:01.300346 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.300300 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" event={"ID":"806f5c61-9a5f-487a-94ec-e3577431dd83","Type":"ContainerDied","Data":"c875a010f990d2b0c541ec88b92b49c86448c59235febf949cb56695f603329e"} Oct 30 04:06:01.300346 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.300342 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f" event={"ID":"806f5c61-9a5f-487a-94ec-e3577431dd83","Type":"ContainerDied","Data":"48be7f77a11a207b23ad66249dcdc2138bde5128c610e0150457833dfb886dd7"} Oct 30 04:06:01.309890 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.309866 4925 scope.go:122] "RemoveContainer" containerID="eb9e4cb6876b84279c42b827e829d8fba21111c9900297382c79eb96045a4cb4" Oct 30 04:06:01.310413 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:06:01.310386 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb9e4cb6876b84279c42b827e829d8fba21111c9900297382c79eb96045a4cb4\": container with ID starting with eb9e4cb6876b84279c42b827e829d8fba21111c9900297382c79eb96045a4cb4 not found: ID does not exist" containerID="eb9e4cb6876b84279c42b827e829d8fba21111c9900297382c79eb96045a4cb4" Oct 30 04:06:01.310413 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.310425 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb9e4cb6876b84279c42b827e829d8fba21111c9900297382c79eb96045a4cb4"} err="failed to get container status \"eb9e4cb6876b84279c42b827e829d8fba21111c9900297382c79eb96045a4cb4\": rpc error: code = NotFound desc = could not find container \"eb9e4cb6876b84279c42b827e829d8fba21111c9900297382c79eb96045a4cb4\": container with ID starting with eb9e4cb6876b84279c42b827e829d8fba21111c9900297382c79eb96045a4cb4 not found: ID does not exist" Oct 30 04:06:01.310603 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.310449 4925 scope.go:122] "RemoveContainer" containerID="c875a010f990d2b0c541ec88b92b49c86448c59235febf949cb56695f603329e" Oct 30 04:06:01.322788 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.322742 4925 scope.go:122] "RemoveContainer" containerID="c875a010f990d2b0c541ec88b92b49c86448c59235febf949cb56695f603329e" Oct 30 04:06:01.323091 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:06:01.323070 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c875a010f990d2b0c541ec88b92b49c86448c59235febf949cb56695f603329e\": container with ID starting with c875a010f990d2b0c541ec88b92b49c86448c59235febf949cb56695f603329e not found: ID does not exist" containerID="c875a010f990d2b0c541ec88b92b49c86448c59235febf949cb56695f603329e" Oct 30 04:06:01.323163 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.323097 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c875a010f990d2b0c541ec88b92b49c86448c59235febf949cb56695f603329e"} err="failed to get container status \"c875a010f990d2b0c541ec88b92b49c86448c59235febf949cb56695f603329e\": rpc error: code = NotFound desc = could not find container \"c875a010f990d2b0c541ec88b92b49c86448c59235febf949cb56695f603329e\": container with ID starting with c875a010f990d2b0c541ec88b92b49c86448c59235febf949cb56695f603329e not found: ID does not exist" Oct 30 04:06:01.357146 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.357115 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/806f5c61-9a5f-487a-94ec-e3577431dd83-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:01.357146 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.357145 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvbqm\" (UniqueName: \"kubernetes.io/projected/806f5c61-9a5f-487a-94ec-e3577431dd83-kube-api-access-hvbqm\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:01.357383 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.357162 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmvz6\" (UniqueName: \"kubernetes.io/projected/0aefabe9-0871-4f53-a299-22befa91ebf9-kube-api-access-lmvz6\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:01.357383 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.357178 4925 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-kube-rbac-proxy-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:01.357383 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.357192 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-prometheus-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:01.357383 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.357227 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0aefabe9-0871-4f53-a299-22befa91ebf9-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:01.357383 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.357241 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/806f5c61-9a5f-487a-94ec-e3577431dd83-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:01.377537 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.377503 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f"] Oct 30 04:06:01.383391 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.383358 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-jp42f"] Oct 30 04:06:01.394373 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.394348 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 04:06:01.394680 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.394613 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="6654a23a-7079-4f43-8c52-0e07e08bdd90" containerName="prometheus" containerID="cri-o://73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a" gracePeriod=600 Oct 30 04:06:01.394680 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.394632 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="6654a23a-7079-4f43-8c52-0e07e08bdd90" containerName="thanos-sidecar" containerID="cri-o://1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac" gracePeriod=600 Oct 30 04:06:01.394680 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.394660 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="6654a23a-7079-4f43-8c52-0e07e08bdd90" containerName="config-reloader" containerID="cri-o://0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b" gracePeriod=600 Oct 30 04:06:01.397604 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.397576 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:06:01.400162 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.400113 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="97f68790-292e-4efa-9396-3c4ede68c3bb" containerName="config-reloader" containerID="cri-o://0ce31577a2679abe8b867430db4f3dff3372172b4f66566ef09e56b8da031bff" gracePeriod=120 Oct 30 04:06:01.400554 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.400524 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="97f68790-292e-4efa-9396-3c4ede68c3bb" containerName="alertmanager" containerID="cri-o://34c12c4161c47fef92a9f1e5b580c30040b32bd208808ce9e4f1c69adfcf4e70" gracePeriod=120 Oct 30 04:06:01.421542 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.421516 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b"] Oct 30 04:06:01.429182 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.429158 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-xcp6b"] Oct 30 04:06:01.650483 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.650416 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:06:01.659393 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.659372 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config\") pod \"6654a23a-7079-4f43-8c52-0e07e08bdd90\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " Oct 30 04:06:01.659513 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.659417 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"6654a23a-7079-4f43-8c52-0e07e08bdd90\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " Oct 30 04:06:01.659513 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.659448 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-config\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-config\") pod \"6654a23a-7079-4f43-8c52-0e07e08bdd90\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " Oct 30 04:06:01.659644 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.659535 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"6654a23a-7079-4f43-8c52-0e07e08bdd90\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " Oct 30 04:06:01.659644 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.659583 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"6654a23a-7079-4f43-8c52-0e07e08bdd90\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " Oct 30 04:06:01.659644 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.659614 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"6654a23a-7079-4f43-8c52-0e07e08bdd90\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " Oct 30 04:06:01.659817 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.659658 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"6654a23a-7079-4f43-8c52-0e07e08bdd90\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " Oct 30 04:06:01.659817 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.659724 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-thanos-prometheus-http-client-file\") pod \"6654a23a-7079-4f43-8c52-0e07e08bdd90\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " Oct 30 04:06:01.659817 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.659743 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-2" pod "6654a23a-7079-4f43-8c52-0e07e08bdd90" (UID: "6654a23a-7079-4f43-8c52-0e07e08bdd90"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:01.660012 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.659982 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-1" pod "6654a23a-7079-4f43-8c52-0e07e08bdd90" (UID: "6654a23a-7079-4f43-8c52-0e07e08bdd90"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:01.660101 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.660083 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6654a23a-7079-4f43-8c52-0e07e08bdd90-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6654a23a-7079-4f43-8c52-0e07e08bdd90-config-out\") pod \"6654a23a-7079-4f43-8c52-0e07e08bdd90\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " Oct 30 04:06:01.660189 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.660135 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-secret-prometheus-web-tls-ca\") pod \"6654a23a-7079-4f43-8c52-0e07e08bdd90\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " Oct 30 04:06:01.660189 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.660164 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6654a23a-7079-4f43-8c52-0e07e08bdd90-tls-assets\" (UniqueName: \"kubernetes.io/projected/6654a23a-7079-4f43-8c52-0e07e08bdd90-tls-assets\") pod \"6654a23a-7079-4f43-8c52-0e07e08bdd90\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " Oct 30 04:06:01.660299 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.660231 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"6654a23a-7079-4f43-8c52-0e07e08bdd90\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " Oct 30 04:06:01.660358 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.660293 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6654a23a-7079-4f43-8c52-0e07e08bdd90-kube-api-access-nmg8t\" (UniqueName: \"kubernetes.io/projected/6654a23a-7079-4f43-8c52-0e07e08bdd90-kube-api-access-nmg8t\") pod \"6654a23a-7079-4f43-8c52-0e07e08bdd90\" (UID: \"6654a23a-7079-4f43-8c52-0e07e08bdd90\") " Oct 30 04:06:01.660683 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.660529 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:01.660683 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.660552 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:01.662171 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.662141 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "6654a23a-7079-4f43-8c52-0e07e08bdd90" (UID: "6654a23a-7079-4f43-8c52-0e07e08bdd90"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:01.662562 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.662537 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "6654a23a-7079-4f43-8c52-0e07e08bdd90" (UID: "6654a23a-7079-4f43-8c52-0e07e08bdd90"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:01.662645 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.662609 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-0" pod "6654a23a-7079-4f43-8c52-0e07e08bdd90" (UID: "6654a23a-7079-4f43-8c52-0e07e08bdd90"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:01.662896 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.662871 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-config" pod "6654a23a-7079-4f43-8c52-0e07e08bdd90" (UID: "6654a23a-7079-4f43-8c52-0e07e08bdd90"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:01.663101 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.663075 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-thanos-prometheus-http-client-file" pod "6654a23a-7079-4f43-8c52-0e07e08bdd90" (UID: "6654a23a-7079-4f43-8c52-0e07e08bdd90"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:01.663398 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.663374 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6654a23a-7079-4f43-8c52-0e07e08bdd90-kube-api-access-nmg8t" pod "6654a23a-7079-4f43-8c52-0e07e08bdd90" (UID: "6654a23a-7079-4f43-8c52-0e07e08bdd90"). InnerVolumeSpecName "kube-api-access-nmg8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:01.663483 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.663459 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-secret-prometheus-web-tls-ca" pod "6654a23a-7079-4f43-8c52-0e07e08bdd90" (UID: "6654a23a-7079-4f43-8c52-0e07e08bdd90"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:01.664163 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.664136 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6654a23a-7079-4f43-8c52-0e07e08bdd90-config-out" pod "6654a23a-7079-4f43-8c52-0e07e08bdd90" (UID: "6654a23a-7079-4f43-8c52-0e07e08bdd90"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:01.664404 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.664381 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6654a23a-7079-4f43-8c52-0e07e08bdd90-tls-assets" pod "6654a23a-7079-4f43-8c52-0e07e08bdd90" (UID: "6654a23a-7079-4f43-8c52-0e07e08bdd90"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:01.665131 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.665105 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085" pod "6654a23a-7079-4f43-8c52-0e07e08bdd90" (UID: "6654a23a-7079-4f43-8c52-0e07e08bdd90"). InnerVolumeSpecName "pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:06:01.675066 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.675044 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config" pod "6654a23a-7079-4f43-8c52-0e07e08bdd90" (UID: "6654a23a-7079-4f43-8c52-0e07e08bdd90"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:01.761832 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.761799 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6654a23a-7079-4f43-8c52-0e07e08bdd90-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:01.761832 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.761832 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmg8t\" (UniqueName: \"kubernetes.io/projected/6654a23a-7079-4f43-8c52-0e07e08bdd90-kube-api-access-nmg8t\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:01.762064 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.761845 4925 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:01.762064 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.761857 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:01.762064 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.761891 4925 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") on node \"ip-10-0-139-19.ec2.internal\" " Oct 30 04:06:01.762064 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.761905 4925 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:01.762064 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.761921 4925 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:01.762064 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.761935 4925 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:01.762064 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.761947 4925 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6654a23a-7079-4f43-8c52-0e07e08bdd90-config-out\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:01.762064 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.761959 4925 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/6654a23a-7079-4f43-8c52-0e07e08bdd90-secret-prometheus-web-tls-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:01.762064 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.761971 4925 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6654a23a-7079-4f43-8c52-0e07e08bdd90-tls-assets\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:01.774349 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.774319 4925 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085") on node "ip-10-0-139-19.ec2.internal" Oct 30 04:06:01.862317 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:01.862273 4925 reconciler_common.go:299] "Volume detached for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"csi-77aa142b6cc95d8f79f4aef80349e4d929aed6d462958597286d83ed1cc8fc2c\"" Oct 30 04:06:02.064090 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.064054 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:06:02.064604 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.064291 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="a1e45ba0-0ae1-471a-a337-eed3d34ac818" containerName="otc-container" containerID="cri-o://17f26278ee92d5be63edaa68261d6102410594e73010debc34995b0091c4cdd0" gracePeriod=30 Oct 30 04:06:02.319104 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.319068 4925 generic.go:356] "Generic (PLEG): container finished" podID="97f68790-292e-4efa-9396-3c4ede68c3bb" containerID="0ce31577a2679abe8b867430db4f3dff3372172b4f66566ef09e56b8da031bff" exitCode=0 Oct 30 04:06:02.319104 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.319097 4925 generic.go:356] "Generic (PLEG): container finished" podID="97f68790-292e-4efa-9396-3c4ede68c3bb" containerID="34c12c4161c47fef92a9f1e5b580c30040b32bd208808ce9e4f1c69adfcf4e70" exitCode=0 Oct 30 04:06:02.319342 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.319137 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"97f68790-292e-4efa-9396-3c4ede68c3bb","Type":"ContainerDied","Data":"0ce31577a2679abe8b867430db4f3dff3372172b4f66566ef09e56b8da031bff"} Oct 30 04:06:02.319342 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.319224 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"97f68790-292e-4efa-9396-3c4ede68c3bb","Type":"ContainerDied","Data":"34c12c4161c47fef92a9f1e5b580c30040b32bd208808ce9e4f1c69adfcf4e70"} Oct 30 04:06:02.321311 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.321283 4925 generic.go:356] "Generic (PLEG): container finished" podID="a1e45ba0-0ae1-471a-a337-eed3d34ac818" containerID="17f26278ee92d5be63edaa68261d6102410594e73010debc34995b0091c4cdd0" exitCode=0 Oct 30 04:06:02.321451 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.321355 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a1e45ba0-0ae1-471a-a337-eed3d34ac818","Type":"ContainerDied","Data":"17f26278ee92d5be63edaa68261d6102410594e73010debc34995b0091c4cdd0"} Oct 30 04:06:02.325257 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.325237 4925 generic.go:356] "Generic (PLEG): container finished" podID="6654a23a-7079-4f43-8c52-0e07e08bdd90" containerID="1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac" exitCode=0 Oct 30 04:06:02.325257 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.325253 4925 generic.go:356] "Generic (PLEG): container finished" podID="6654a23a-7079-4f43-8c52-0e07e08bdd90" containerID="0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b" exitCode=0 Oct 30 04:06:02.325432 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.325262 4925 generic.go:356] "Generic (PLEG): container finished" podID="6654a23a-7079-4f43-8c52-0e07e08bdd90" containerID="73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a" exitCode=0 Oct 30 04:06:02.325432 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.325323 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 04:06:02.325507 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.325320 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"6654a23a-7079-4f43-8c52-0e07e08bdd90","Type":"ContainerDied","Data":"1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac"} Oct 30 04:06:02.325546 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.325508 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"6654a23a-7079-4f43-8c52-0e07e08bdd90","Type":"ContainerDied","Data":"0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b"} Oct 30 04:06:02.325587 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.325550 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"6654a23a-7079-4f43-8c52-0e07e08bdd90","Type":"ContainerDied","Data":"73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a"} Oct 30 04:06:02.325587 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.325574 4925 scope.go:122] "RemoveContainer" containerID="1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac" Oct 30 04:06:02.325676 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.325581 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"6654a23a-7079-4f43-8c52-0e07e08bdd90","Type":"ContainerDied","Data":"0047a7d26ea9e110484b09e70f95455b3534bd1f7f6416803d45b228a997cf1c"} Oct 30 04:06:02.347386 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.347359 4925 scope.go:122] "RemoveContainer" containerID="0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b" Oct 30 04:06:02.353552 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.353527 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:06:02.355792 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.355768 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:06:02.358278 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.358254 4925 scope.go:122] "RemoveContainer" containerID="73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a" Oct 30 04:06:02.366633 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.366605 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-web-config\" (UniqueName: \"kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-web-config\") pod \"97f68790-292e-4efa-9396-3c4ede68c3bb\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " Oct 30 04:06:02.366785 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.366655 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f68790-292e-4efa-9396-3c4ede68c3bb-kube-api-access-dpsxl\" (UniqueName: \"kubernetes.io/projected/97f68790-292e-4efa-9396-3c4ede68c3bb-kube-api-access-dpsxl\") pod \"97f68790-292e-4efa-9396-3c4ede68c3bb\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " Oct 30 04:06:02.366785 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.366691 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-monitoring-tls-certs\") pod \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\" (UID: \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\") " Oct 30 04:06:02.366785 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.366713 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f68790-292e-4efa-9396-3c4ede68c3bb-tls-assets\" (UniqueName: \"kubernetes.io/projected/97f68790-292e-4efa-9396-3c4ede68c3bb-tls-assets\") pod \"97f68790-292e-4efa-9396-3c4ede68c3bb\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " Oct 30 04:06:02.366785 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.366762 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-tls-certs\" (UniqueName: \"kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-tls-certs\") pod \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\" (UID: \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\") " Oct 30 04:06:02.367088 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.366792 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e45ba0-0ae1-471a-a337-eed3d34ac818-kube-api-access-pfdtg\" (UniqueName: \"kubernetes.io/projected/a1e45ba0-0ae1-471a-a337-eed3d34ac818-kube-api-access-pfdtg\") pod \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\" (UID: \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\") " Oct 30 04:06:02.367088 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.366809 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-config-volume\" (UniqueName: \"kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-config-volume\") pod \"97f68790-292e-4efa-9396-3c4ede68c3bb\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " Oct 30 04:06:02.367088 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.366841 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-cluster-tls-config\") pod \"97f68790-292e-4efa-9396-3c4ede68c3bb\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " Oct 30 04:06:02.367088 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.366864 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f68790-292e-4efa-9396-3c4ede68c3bb-config-out\" (UniqueName: \"kubernetes.io/empty-dir/97f68790-292e-4efa-9396-3c4ede68c3bb-config-out\") pod \"97f68790-292e-4efa-9396-3c4ede68c3bb\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " Oct 30 04:06:02.367088 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.366889 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-data-science-collector-ta-client-cert\") pod \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\" (UID: \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\") " Oct 30 04:06:02.367088 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.366916 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f68790-292e-4efa-9396-3c4ede68c3bb-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/97f68790-292e-4efa-9396-3c4ede68c3bb-alertmanager-data-science-monitoringstack-db\") pod \"97f68790-292e-4efa-9396-3c4ede68c3bb\" (UID: \"97f68790-292e-4efa-9396-3c4ede68c3bb\") " Oct 30 04:06:02.367088 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.366943 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1e45ba0-0ae1-471a-a337-eed3d34ac818-otc-internal\" (UniqueName: \"kubernetes.io/configmap/a1e45ba0-0ae1-471a-a337-eed3d34ac818-otc-internal\") pod \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\" (UID: \"a1e45ba0-0ae1-471a-a337-eed3d34ac818\") " Oct 30 04:06:02.367088 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.367025 4925 scope.go:122] "RemoveContainer" containerID="96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f" Oct 30 04:06:02.367537 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.367509 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e45ba0-0ae1-471a-a337-eed3d34ac818-otc-internal" pod "a1e45ba0-0ae1-471a-a337-eed3d34ac818" (UID: "a1e45ba0-0ae1-471a-a337-eed3d34ac818"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:02.367960 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.367935 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f68790-292e-4efa-9396-3c4ede68c3bb-alertmanager-data-science-monitoringstack-db" pod "97f68790-292e-4efa-9396-3c4ede68c3bb" (UID: "97f68790-292e-4efa-9396-3c4ede68c3bb"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:02.370195 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.370174 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f68790-292e-4efa-9396-3c4ede68c3bb-tls-assets" pod "97f68790-292e-4efa-9396-3c4ede68c3bb" (UID: "97f68790-292e-4efa-9396-3c4ede68c3bb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:02.371116 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.370592 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-tls-certs" pod "a1e45ba0-0ae1-471a-a337-eed3d34ac818" (UID: "a1e45ba0-0ae1-471a-a337-eed3d34ac818"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:02.371116 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.370665 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-data-science-collector-ta-client-cert" pod "a1e45ba0-0ae1-471a-a337-eed3d34ac818" (UID: "a1e45ba0-0ae1-471a-a337-eed3d34ac818"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:02.371116 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.370685 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f68790-292e-4efa-9396-3c4ede68c3bb-kube-api-access-dpsxl" pod "97f68790-292e-4efa-9396-3c4ede68c3bb" (UID: "97f68790-292e-4efa-9396-3c4ede68c3bb"). InnerVolumeSpecName "kube-api-access-dpsxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:02.371567 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.371476 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f68790-292e-4efa-9396-3c4ede68c3bb-config-out" pod "97f68790-292e-4efa-9396-3c4ede68c3bb" (UID: "97f68790-292e-4efa-9396-3c4ede68c3bb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:02.371939 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.371910 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-monitoring-tls-certs" pod "a1e45ba0-0ae1-471a-a337-eed3d34ac818" (UID: "a1e45ba0-0ae1-471a-a337-eed3d34ac818"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:02.372179 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.372160 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e45ba0-0ae1-471a-a337-eed3d34ac818-kube-api-access-pfdtg" pod "a1e45ba0-0ae1-471a-a337-eed3d34ac818" (UID: "a1e45ba0-0ae1-471a-a337-eed3d34ac818"). InnerVolumeSpecName "kube-api-access-pfdtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:02.372478 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.372458 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-config-volume" pod "97f68790-292e-4efa-9396-3c4ede68c3bb" (UID: "97f68790-292e-4efa-9396-3c4ede68c3bb"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:02.374998 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.374978 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-cluster-tls-config" pod "97f68790-292e-4efa-9396-3c4ede68c3bb" (UID: "97f68790-292e-4efa-9396-3c4ede68c3bb"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:02.382077 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.382050 4925 scope.go:122] "RemoveContainer" containerID="1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac" Oct 30 04:06:02.382806 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.382610 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-web-config" pod "97f68790-292e-4efa-9396-3c4ede68c3bb" (UID: "97f68790-292e-4efa-9396-3c4ede68c3bb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:02.382806 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:06:02.382611 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac\": container with ID starting with 1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac not found: ID does not exist" containerID="1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac" Oct 30 04:06:02.382806 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.382664 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac"} err="failed to get container status \"1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac\": rpc error: code = NotFound desc = could not find container \"1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac\": container with ID starting with 1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac not found: ID does not exist" Oct 30 04:06:02.382806 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.382694 4925 scope.go:122] "RemoveContainer" containerID="0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b" Oct 30 04:06:02.383523 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:06:02.383495 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b\": container with ID starting with 0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b not found: ID does not exist" containerID="0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b" Oct 30 04:06:02.383628 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.383530 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b"} err="failed to get container status \"0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b\": rpc error: code = NotFound desc = could not find container \"0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b\": container with ID starting with 0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b not found: ID does not exist" Oct 30 04:06:02.383628 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.383552 4925 scope.go:122] "RemoveContainer" containerID="73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a" Oct 30 04:06:02.384214 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:06:02.384090 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a\": container with ID starting with 73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a not found: ID does not exist" containerID="73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a" Oct 30 04:06:02.384214 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.384122 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a"} err="failed to get container status \"73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a\": rpc error: code = NotFound desc = could not find container \"73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a\": container with ID starting with 73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a not found: ID does not exist" Oct 30 04:06:02.384214 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.384142 4925 scope.go:122] "RemoveContainer" containerID="96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f" Oct 30 04:06:02.384536 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:06:02.384500 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f\": container with ID starting with 96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f not found: ID does not exist" containerID="96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f" Oct 30 04:06:02.384589 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.384545 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f"} err="failed to get container status \"96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f\": rpc error: code = NotFound desc = could not find container \"96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f\": container with ID starting with 96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f not found: ID does not exist" Oct 30 04:06:02.384589 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.384565 4925 scope.go:122] "RemoveContainer" containerID="1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac" Oct 30 04:06:02.384965 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.384937 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac"} err="failed to get container status \"1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac\": rpc error: code = NotFound desc = could not find container \"1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac\": container with ID starting with 1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac not found: ID does not exist" Oct 30 04:06:02.385079 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.384967 4925 scope.go:122] "RemoveContainer" containerID="0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b" Oct 30 04:06:02.385279 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.385254 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b"} err="failed to get container status \"0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b\": rpc error: code = NotFound desc = could not find container \"0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b\": container with ID starting with 0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b not found: ID does not exist" Oct 30 04:06:02.385341 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.385280 4925 scope.go:122] "RemoveContainer" containerID="73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a" Oct 30 04:06:02.385539 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.385517 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a"} err="failed to get container status \"73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a\": rpc error: code = NotFound desc = could not find container \"73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a\": container with ID starting with 73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a not found: ID does not exist" Oct 30 04:06:02.385539 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.385538 4925 scope.go:122] "RemoveContainer" containerID="96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f" Oct 30 04:06:02.385825 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.385803 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f"} err="failed to get container status \"96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f\": rpc error: code = NotFound desc = could not find container \"96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f\": container with ID starting with 96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f not found: ID does not exist" Oct 30 04:06:02.385904 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.385825 4925 scope.go:122] "RemoveContainer" containerID="1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac" Oct 30 04:06:02.386074 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.386055 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac"} err="failed to get container status \"1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac\": rpc error: code = NotFound desc = could not find container \"1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac\": container with ID starting with 1ab9d609da20c4a90317b9730d85b69bf91054ee418a54a25e90fe88fcd7ceac not found: ID does not exist" Oct 30 04:06:02.386153 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.386074 4925 scope.go:122] "RemoveContainer" containerID="0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b" Oct 30 04:06:02.386332 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.386308 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 04:06:02.386332 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.386318 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b"} err="failed to get container status \"0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b\": rpc error: code = NotFound desc = could not find container \"0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b\": container with ID starting with 0f363ee90fdb7661a0e25283ec2753fee0f8b3f2846590d5d75022df1085799b not found: ID does not exist" Oct 30 04:06:02.386514 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.386343 4925 scope.go:122] "RemoveContainer" containerID="73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a" Oct 30 04:06:02.386646 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.386622 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a"} err="failed to get container status \"73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a\": rpc error: code = NotFound desc = could not find container \"73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a\": container with ID starting with 73f736d2e84536a1bd50d1e5ea92b6cc3e2054bc0946ec705f85a8239639d91a not found: ID does not exist" Oct 30 04:06:02.386646 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.386645 4925 scope.go:122] "RemoveContainer" containerID="96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f" Oct 30 04:06:02.387105 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.387079 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f"} err="failed to get container status \"96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f\": rpc error: code = NotFound desc = could not find container \"96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f\": container with ID starting with 96facfe0bbecde708885225e432539c727939a39d8244acbdd239f2c226be62f not found: ID does not exist" Oct 30 04:06:02.398543 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.398485 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 04:06:02.467683 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.467599 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.467683 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.467632 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfdtg\" (UniqueName: \"kubernetes.io/projected/a1e45ba0-0ae1-471a-a337-eed3d34ac818-kube-api-access-pfdtg\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.467683 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.467673 4925 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-config-volume\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.467999 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.467689 4925 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-cluster-tls-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.467999 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.467703 4925 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97f68790-292e-4efa-9396-3c4ede68c3bb-config-out\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.467999 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.467719 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.467999 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.467735 4925 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/97f68790-292e-4efa-9396-3c4ede68c3bb-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.467999 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.467779 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a1e45ba0-0ae1-471a-a337-eed3d34ac818-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.467999 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.467794 4925 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97f68790-292e-4efa-9396-3c4ede68c3bb-web-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.467999 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.467808 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpsxl\" (UniqueName: \"kubernetes.io/projected/97f68790-292e-4efa-9396-3c4ede68c3bb-kube-api-access-dpsxl\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.467999 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.467821 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a1e45ba0-0ae1-471a-a337-eed3d34ac818-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:02.467999 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:02.467834 4925 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97f68790-292e-4efa-9396-3c4ede68c3bb-tls-assets\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:06:03.119395 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:03.119354 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aefabe9-0871-4f53-a299-22befa91ebf9" path="/var/lib/kubelet/pods/0aefabe9-0871-4f53-a299-22befa91ebf9/volumes" Oct 30 04:06:03.119895 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:03.119878 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6654a23a-7079-4f43-8c52-0e07e08bdd90" path="/var/lib/kubelet/pods/6654a23a-7079-4f43-8c52-0e07e08bdd90/volumes" Oct 30 04:06:03.120814 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:03.120790 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="806f5c61-9a5f-487a-94ec-e3577431dd83" path="/var/lib/kubelet/pods/806f5c61-9a5f-487a-94ec-e3577431dd83/volumes" Oct 30 04:06:03.333735 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:03.333705 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 04:06:03.333928 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:03.333692 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"97f68790-292e-4efa-9396-3c4ede68c3bb","Type":"ContainerDied","Data":"65dbf1dbe9a1705534c81d3d873b1216bfc509e4dd70f4f92d1f977a6d7a2db6"} Oct 30 04:06:03.333928 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:03.333830 4925 scope.go:122] "RemoveContainer" containerID="0ce31577a2679abe8b867430db4f3dff3372172b4f66566ef09e56b8da031bff" Oct 30 04:06:03.335246 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:03.335223 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 04:06:03.335347 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:03.335221 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a1e45ba0-0ae1-471a-a337-eed3d34ac818","Type":"ContainerDied","Data":"5017d589903dabe6c48e896b2a0708e6fb8898437c5c107358cb6f62a9bb6e6c"} Oct 30 04:06:03.342936 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:03.342915 4925 scope.go:122] "RemoveContainer" containerID="34c12c4161c47fef92a9f1e5b580c30040b32bd208808ce9e4f1c69adfcf4e70" Oct 30 04:06:03.351094 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:03.351077 4925 scope.go:122] "RemoveContainer" containerID="6126c7c3933a98d99eb161502058bdf72a8c01b5aad360ed4a09a18b3fa4445c" Oct 30 04:06:03.358478 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:03.358467 4925 scope.go:122] "RemoveContainer" containerID="17f26278ee92d5be63edaa68261d6102410594e73010debc34995b0091c4cdd0" Oct 30 04:06:03.387422 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:03.387402 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:06:03.400127 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:03.400103 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 04:06:03.425969 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:03.425945 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:06:03.430354 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:03.430333 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 04:06:05.118518 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:05.118487 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f68790-292e-4efa-9396-3c4ede68c3bb" path="/var/lib/kubelet/pods/97f68790-292e-4efa-9396-3c4ede68c3bb/volumes" Oct 30 04:06:05.118993 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:05.118978 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1e45ba0-0ae1-471a-a337-eed3d34ac818" path="/var/lib/kubelet/pods/a1e45ba0-0ae1-471a-a337-eed3d34ac818/volumes" Oct 30 04:06:18.161604 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:06:18.161565 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:06:24.758372 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758332 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m"] Oct 30 04:06:24.758804 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758671 4925 cpu_manager.go:414] "containerMap: removing container" podUID="97f68790-292e-4efa-9396-3c4ede68c3bb" containerName="alertmanager" Oct 30 04:06:24.758804 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758683 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f68790-292e-4efa-9396-3c4ede68c3bb" containerName="alertmanager" Oct 30 04:06:24.758804 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758697 4925 cpu_manager.go:414] "containerMap: removing container" podUID="97f68790-292e-4efa-9396-3c4ede68c3bb" containerName="init-config-reloader" Oct 30 04:06:24.758804 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758703 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f68790-292e-4efa-9396-3c4ede68c3bb" containerName="init-config-reloader" Oct 30 04:06:24.758804 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758713 4925 cpu_manager.go:414] "containerMap: removing container" podUID="0aefabe9-0871-4f53-a299-22befa91ebf9" containerName="kube-rbac-proxy" Oct 30 04:06:24.758804 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758718 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aefabe9-0871-4f53-a299-22befa91ebf9" containerName="kube-rbac-proxy" Oct 30 04:06:24.758804 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758726 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6654a23a-7079-4f43-8c52-0e07e08bdd90" containerName="init-config-reloader" Oct 30 04:06:24.758804 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758730 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6654a23a-7079-4f43-8c52-0e07e08bdd90" containerName="init-config-reloader" Oct 30 04:06:24.758804 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758746 4925 cpu_manager.go:414] "containerMap: removing container" podUID="806f5c61-9a5f-487a-94ec-e3577431dd83" containerName="korrel8r" Oct 30 04:06:24.758804 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758765 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="806f5c61-9a5f-487a-94ec-e3577431dd83" containerName="korrel8r" Oct 30 04:06:24.758804 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758776 4925 cpu_manager.go:414] "containerMap: removing container" podUID="a1e45ba0-0ae1-471a-a337-eed3d34ac818" containerName="otc-container" Oct 30 04:06:24.758804 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758780 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e45ba0-0ae1-471a-a337-eed3d34ac818" containerName="otc-container" Oct 30 04:06:24.758804 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758785 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6654a23a-7079-4f43-8c52-0e07e08bdd90" containerName="config-reloader" Oct 30 04:06:24.758804 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758791 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6654a23a-7079-4f43-8c52-0e07e08bdd90" containerName="config-reloader" Oct 30 04:06:24.758804 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758799 4925 cpu_manager.go:414] "containerMap: removing container" podUID="97f68790-292e-4efa-9396-3c4ede68c3bb" containerName="config-reloader" Oct 30 04:06:24.758804 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758804 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f68790-292e-4efa-9396-3c4ede68c3bb" containerName="config-reloader" Oct 30 04:06:24.758804 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758810 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6654a23a-7079-4f43-8c52-0e07e08bdd90" containerName="prometheus" Oct 30 04:06:24.758804 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758814 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6654a23a-7079-4f43-8c52-0e07e08bdd90" containerName="prometheus" Oct 30 04:06:24.759412 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758820 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6654a23a-7079-4f43-8c52-0e07e08bdd90" containerName="thanos-sidecar" Oct 30 04:06:24.759412 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758824 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6654a23a-7079-4f43-8c52-0e07e08bdd90" containerName="thanos-sidecar" Oct 30 04:06:24.759412 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758877 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f68790-292e-4efa-9396-3c4ede68c3bb" containerName="alertmanager" Oct 30 04:06:24.759412 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758892 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aefabe9-0871-4f53-a299-22befa91ebf9" containerName="kube-rbac-proxy" Oct 30 04:06:24.759412 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758901 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="806f5c61-9a5f-487a-94ec-e3577431dd83" containerName="korrel8r" Oct 30 04:06:24.759412 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758907 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e45ba0-0ae1-471a-a337-eed3d34ac818" containerName="otc-container" Oct 30 04:06:24.759412 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758912 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6654a23a-7079-4f43-8c52-0e07e08bdd90" containerName="config-reloader" Oct 30 04:06:24.759412 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758918 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f68790-292e-4efa-9396-3c4ede68c3bb" containerName="config-reloader" Oct 30 04:06:24.759412 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758923 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6654a23a-7079-4f43-8c52-0e07e08bdd90" containerName="prometheus" Oct 30 04:06:24.759412 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.758928 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6654a23a-7079-4f43-8c52-0e07e08bdd90" containerName="thanos-sidecar" Oct 30 04:06:24.763814 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.763786 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:06:24.789627 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.789596 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m"] Oct 30 04:06:24.789790 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.789780 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 04:06:24.794255 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.794224 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ca-bundle\"" Oct 30 04:06:24.794505 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.794490 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-config\"" Oct 30 04:06:24.794589 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.794563 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 04:06:24.843247 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.843202 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-dockercfg-qxnzq\"" Oct 30 04:06:24.843406 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.843276 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-query-frontend-grpc\"" Oct 30 04:06:24.843406 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.843340 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-query-frontend-http\"" Oct 30 04:06:24.844670 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.844650 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-f746fc66d-7wg8m\" (UID: \"9d91fb0f-d274-4d79-a481-4608ca9d108c\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:06:24.844722 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.844697 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-f746fc66d-7wg8m\" (UID: \"9d91fb0f-d274-4d79-a481-4608ca9d108c\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:06:24.844789 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.844727 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtcws\" (UniqueName: \"kubernetes.io/projected/9d91fb0f-d274-4d79-a481-4608ca9d108c-kube-api-access-mtcws\") pod \"data-science-lokistack-query-frontend-f746fc66d-7wg8m\" (UID: \"9d91fb0f-d274-4d79-a481-4608ca9d108c\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:06:24.844939 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.844912 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-f746fc66d-7wg8m\" (UID: \"9d91fb0f-d274-4d79-a481-4608ca9d108c\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:06:24.844999 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.844975 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d91fb0f-d274-4d79-a481-4608ca9d108c-config\") pod \"data-science-lokistack-query-frontend-f746fc66d-7wg8m\" (UID: \"9d91fb0f-d274-4d79-a481-4608ca9d108c\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:06:24.946113 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.946079 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-f746fc66d-7wg8m\" (UID: \"9d91fb0f-d274-4d79-a481-4608ca9d108c\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:06:24.946305 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.946142 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-f746fc66d-7wg8m\" (UID: \"9d91fb0f-d274-4d79-a481-4608ca9d108c\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:06:24.946305 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.946169 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtcws\" (UniqueName: \"kubernetes.io/projected/9d91fb0f-d274-4d79-a481-4608ca9d108c-kube-api-access-mtcws\") pod \"data-science-lokistack-query-frontend-f746fc66d-7wg8m\" (UID: \"9d91fb0f-d274-4d79-a481-4608ca9d108c\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:06:24.946305 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.946212 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-f746fc66d-7wg8m\" (UID: \"9d91fb0f-d274-4d79-a481-4608ca9d108c\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:06:24.946305 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.946251 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d91fb0f-d274-4d79-a481-4608ca9d108c-config\") pod \"data-science-lokistack-query-frontend-f746fc66d-7wg8m\" (UID: \"9d91fb0f-d274-4d79-a481-4608ca9d108c\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:06:24.946992 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.946966 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-f746fc66d-7wg8m\" (UID: \"9d91fb0f-d274-4d79-a481-4608ca9d108c\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:06:24.947123 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.947103 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d91fb0f-d274-4d79-a481-4608ca9d108c-config\") pod \"data-science-lokistack-query-frontend-f746fc66d-7wg8m\" (UID: \"9d91fb0f-d274-4d79-a481-4608ca9d108c\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:06:24.948604 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.948583 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-f746fc66d-7wg8m\" (UID: \"9d91fb0f-d274-4d79-a481-4608ca9d108c\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:06:24.948709 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.948619 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-f746fc66d-7wg8m\" (UID: \"9d91fb0f-d274-4d79-a481-4608ca9d108c\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:06:24.969387 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:24.969358 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtcws\" (UniqueName: \"kubernetes.io/projected/9d91fb0f-d274-4d79-a481-4608ca9d108c-kube-api-access-mtcws\") pod \"data-science-lokistack-query-frontend-f746fc66d-7wg8m\" (UID: \"9d91fb0f-d274-4d79-a481-4608ca9d108c\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:06:25.020552 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.020483 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx"] Oct 30 04:06:25.024977 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.024951 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.044490 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.044463 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx"] Oct 30 04:06:25.049998 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.049975 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-http\"" Oct 30 04:06:25.050103 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.050076 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-dockercfg-qfmnm\"" Oct 30 04:06:25.050369 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.050354 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\"" Oct 30 04:06:25.050490 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.050479 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-client-http\"" Oct 30 04:06:25.075705 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.075681 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:06:25.110372 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.110335 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\"" Oct 30 04:06:25.110506 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.110386 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-ca-bundle\"" Oct 30 04:06:25.148122 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.148093 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.148279 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.148143 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/27e236f1-2d56-4aa3-be69-d081172c27b1-tenants\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.148279 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.148191 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-rbac\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.148279 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.148217 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.148279 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.148252 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x2mm\" (UniqueName: \"kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-kube-api-access-9x2mm\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.148504 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.148283 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-tls-secret\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.148504 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.148311 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.148504 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.148349 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-lokistack-gateway\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.249613 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.249577 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.249826 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.249625 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/27e236f1-2d56-4aa3-be69-d081172c27b1-tenants\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.249826 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.249665 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-rbac\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.249826 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.249692 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.249826 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.249729 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x2mm\" (UniqueName: \"kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-kube-api-access-9x2mm\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.249826 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.249791 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-tls-secret\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.249826 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.249816 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.250189 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.249855 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-lokistack-gateway\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.250392 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.250363 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.250594 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.250550 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-rbac\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.250714 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.250619 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-lokistack-gateway\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.252352 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.252332 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.252448 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.252425 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-tls-secret\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.252652 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.252634 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/27e236f1-2d56-4aa3-be69-d081172c27b1-tenants\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.252742 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.252721 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.273664 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.273599 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x2mm\" (UniqueName: \"kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-kube-api-access-9x2mm\") pod \"data-science-lokistack-gateway-d7f6645-m4qpx\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.341021 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.340984 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:25.374800 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.374736 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m"] Oct 30 04:06:25.377372 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:06:25.377343 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d91fb0f_d274_4d79_a481_4608ca9d108c.slice/crio-622b4b63c0469aab78a1551706069d39cc6acb098c947d3633e701cfd7d976af WatchSource:0}: Error finding container 622b4b63c0469aab78a1551706069d39cc6acb098c947d3633e701cfd7d976af: Status 404 returned error can't find the container with id 622b4b63c0469aab78a1551706069d39cc6acb098c947d3633e701cfd7d976af Oct 30 04:06:25.379408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.379390 4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:06:25.450383 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.450346 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" event={"ID":"9d91fb0f-d274-4d79-a481-4608ca9d108c","Type":"ContainerStarted","Data":"622b4b63c0469aab78a1551706069d39cc6acb098c947d3633e701cfd7d976af"} Oct 30 04:06:25.658625 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:25.658599 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx"] Oct 30 04:06:25.660365 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:06:25.660339 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e236f1_2d56_4aa3_be69_d081172c27b1.slice/crio-674e36df7c2791e2a92000cc2bfe1422952f937edc873d96ed47cab220bb6c93 WatchSource:0}: Error finding container 674e36df7c2791e2a92000cc2bfe1422952f937edc873d96ed47cab220bb6c93: Status 404 returned error can't find the container with id 674e36df7c2791e2a92000cc2bfe1422952f937edc873d96ed47cab220bb6c93 Oct 30 04:06:26.459424 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:26.459373 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" event={"ID":"27e236f1-2d56-4aa3-be69-d081172c27b1","Type":"ContainerStarted","Data":"674e36df7c2791e2a92000cc2bfe1422952f937edc873d96ed47cab220bb6c93"} Oct 30 04:06:27.579434 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.579401 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 04:06:27.583414 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.583384 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:27.612927 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.612874 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-shared-secret\"" Oct 30 04:06:27.613085 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.612954 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-grpc\"" Oct 30 04:06:27.613085 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.613044 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-http\"" Oct 30 04:06:27.614281 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.614253 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 04:06:27.721986 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.721951 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 04:06:27.727109 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.727084 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:27.742412 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.742385 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-compactor-http\"" Oct 30 04:06:27.742555 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.742511 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-compactor-grpc\"" Oct 30 04:06:27.748180 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.748157 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 04:06:27.773598 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.773555 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:27.773742 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.773691 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfzx4\" (UniqueName: \"kubernetes.io/projected/d50eb753-37ae-4aff-94b6-893a98133df6-kube-api-access-lfzx4\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:27.773835 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.773746 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:27.773835 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.773801 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-test-loki-shared-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:27.773926 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.773834 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d50eb753-37ae-4aff-94b6-893a98133df6-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:27.773926 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.773883 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:27.874549 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.874459 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:27.874710 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.874638 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:27.874710 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.874696 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acff1f0b-ceb6-4222-b151-8a77fcce43ee-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:27.874823 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.874725 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55v74\" (UniqueName: \"kubernetes.io/projected/acff1f0b-ceb6-4222-b151-8a77fcce43ee-kube-api-access-55v74\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:27.874823 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.874782 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfzx4\" (UniqueName: \"kubernetes.io/projected/d50eb753-37ae-4aff-94b6-893a98133df6-kube-api-access-lfzx4\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:27.874823 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.874813 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-test-loki-shared-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:27.874964 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.874841 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:27.874964 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.874879 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:27.874964 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.874923 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-test-loki-shared-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:27.875076 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.874977 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d50eb753-37ae-4aff-94b6-893a98133df6-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:27.875076 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.875008 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:27.875076 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.875046 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:27.875171 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.875124 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:27.875977 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.875947 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d50eb753-37ae-4aff-94b6-893a98133df6-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:27.878363 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.878335 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-test-loki-shared-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:27.880166 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.880145 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:27.880304 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.880283 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:27.902022 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.901991 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfzx4\" (UniqueName: \"kubernetes.io/projected/d50eb753-37ae-4aff-94b6-893a98133df6-kube-api-access-lfzx4\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:27.976102 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.976058 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:27.976290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.976124 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acff1f0b-ceb6-4222-b151-8a77fcce43ee-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:27.976290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.976151 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55v74\" (UniqueName: \"kubernetes.io/projected/acff1f0b-ceb6-4222-b151-8a77fcce43ee-kube-api-access-55v74\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:27.976290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.976181 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-test-loki-shared-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:27.976290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.976197 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:27.976290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.976264 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:27.977805 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.977074 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:27.978563 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.978506 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acff1f0b-ceb6-4222-b151-8a77fcce43ee-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:27.980700 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.980679 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:27.981315 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.981294 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:27.981636 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:27.981616 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-test-loki-shared-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:28.001986 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:28.001961 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55v74\" (UniqueName: \"kubernetes.io/projected/acff1f0b-ceb6-4222-b151-8a77fcce43ee-kube-api-access-55v74\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:29.481007 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:29.480965 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" event={"ID":"9d91fb0f-d274-4d79-a481-4608ca9d108c","Type":"ContainerStarted","Data":"1bda0812aeb9ff07c98f3da316750c1b1b7f1be794272bd52ed5b3e5f5d768cd"} Oct 30 04:06:29.481458 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:29.481062 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:06:29.482519 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:29.482495 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" event={"ID":"27e236f1-2d56-4aa3-be69-d081172c27b1","Type":"ContainerStarted","Data":"16f3797807ad44cec9d51d77b811fd32684228f1a3a16c5307b785793f0aeb3b"} Oct 30 04:06:29.524636 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:29.524589 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" podStartSLOduration=1.7808880980000001 podStartE2EDuration="5.524576289s" totalImagesPullingTime="3.743688191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:06:29.523723731 +0000 UTC m=+5150.537001834" watchObservedRunningTime="2026-10-30 04:06:29.524576289 +0000 UTC m=+5150.537854393" Oct 30 04:06:31.304577 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.304540 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:31.305046 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.304590 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:31.305046 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.304624 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:31.347397 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.347365 4925 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:31.347542 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.347419 4925 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:31.347542 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.347432 4925 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:31.406085 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.406056 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:31.406216 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.406094 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:31.406216 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.406115 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:31.406216 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.406177 4925 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:31.406216 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.406194 4925 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:31.406349 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.406261 4925 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:31.421558 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.421530 4925 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") DevicePath \"csi-1cbd6671aaaf13073dd4ac6088fd398781af87903101ff82770e0956bdbc63f1\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:31.421680 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.421532 4925 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") DevicePath \"csi-3faa7a4919429bfae9e5fe05bb9f7e440b0f7cbd5797a7f1cb9f75e405a5c758\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:31.421837 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.421534 4925 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") DevicePath \"csi-d631f5804e757beb9cee949b1ba08adddfe0f2a6d74da288dfa6a5946ab2fad2\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:31.494733 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.494691 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" event={"ID":"27e236f1-2d56-4aa3-be69-d081172c27b1","Type":"ContainerStarted","Data":"762463fe068d096d95db3eab689023d784a6d75359a5a2506df939e8905ebab5"} Oct 30 04:06:31.494930 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.494869 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:31.500912 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.500893 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:31.540786 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.540724 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" podStartSLOduration=2.7307238959999998 podStartE2EDuration="7.540712627s" totalImagesPullingTime="4.809988731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:06:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:06:31.540213212 +0000 UTC m=+5152.553491315" watchObservedRunningTime="2026-10-30 04:06:31.540712627 +0000 UTC m=+5152.553990729" Oct 30 04:06:31.647027 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.646991 4925 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/ad17c5ee2a7e4881911e4c7c70dd9b4982223d2ab904c958abdf2443e32ecc28/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:31.673314 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.673281 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:31.737178 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.737146 4925 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2a84ccb0e632fc97a562bedb1c2138c4534d44455a54f3f6cb9692580fc3a7df/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:31.764824 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.764793 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:31.943810 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:31.943713 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:32.243944 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:32.243921 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 04:06:32.245905 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:06:32.245881 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacff1f0b_ceb6_4222_b151_8a77fcce43ee.slice/crio-bbf503be312695a47c9897daa9f1be286d9dc914f09679a0cb802777ab9c12be WatchSource:0}: Error finding container bbf503be312695a47c9897daa9f1be286d9dc914f09679a0cb802777ab9c12be: Status 404 returned error can't find the container with id bbf503be312695a47c9897daa9f1be286d9dc914f09679a0cb802777ab9c12be Oct 30 04:06:32.506628 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:32.506502 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"acff1f0b-ceb6-4222-b151-8a77fcce43ee","Type":"ContainerStarted","Data":"78c8b09a3cd0e67e1dcd5795bb27757106654acd4718645ae9e044e6907926b9"} Oct 30 04:06:32.507076 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:32.506655 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:32.507076 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:32.506707 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"acff1f0b-ceb6-4222-b151-8a77fcce43ee","Type":"ContainerStarted","Data":"bbf503be312695a47c9897daa9f1be286d9dc914f09679a0cb802777ab9c12be"} Oct 30 04:06:32.507076 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:32.506733 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:32.515403 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:32.515374 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:06:32.551696 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:32.551651 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" podStartSLOduration=8.551639044 podStartE2EDuration="8.551639044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:06:32.550223819 +0000 UTC m=+5153.563501926" watchObservedRunningTime="2026-10-30 04:06:32.551639044 +0000 UTC m=+5153.564917147" Oct 30 04:06:38.741115 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:38.741076 4925 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4b631ef058d3f818390cfd4c55b829173556e3e1b26e1e76626d152426c8d48c/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:38.765716 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:38.765691 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:38.999205 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:38.999109 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:39.315376 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:39.315352 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 04:06:39.317235 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:06:39.317208 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50eb753_37ae_4aff_94b6_893a98133df6.slice/crio-822d50c5d6aa83bffe8b3978ded2f46d073620179292792e222426d2deab1264 WatchSource:0}: Error finding container 822d50c5d6aa83bffe8b3978ded2f46d073620179292792e222426d2deab1264: Status 404 returned error can't find the container with id 822d50c5d6aa83bffe8b3978ded2f46d073620179292792e222426d2deab1264 Oct 30 04:06:39.548416 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:39.548376 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"d50eb753-37ae-4aff-94b6-893a98133df6","Type":"ContainerStarted","Data":"e14cd81acb74a0e435c11727be5d6d9f47a97a9d3b21bd8213de20c88c44a9e7"} Oct 30 04:06:39.548591 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:39.548437 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"d50eb753-37ae-4aff-94b6-893a98133df6","Type":"ContainerStarted","Data":"822d50c5d6aa83bffe8b3978ded2f46d073620179292792e222426d2deab1264"} Oct 30 04:06:39.548591 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:39.548474 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:06:39.594843 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:39.594794 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podStartSLOduration=15.594781333 podStartE2EDuration="15.594781333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:06:39.59333298 +0000 UTC m=+5160.606611084" watchObservedRunningTime="2026-10-30 04:06:39.594781333 +0000 UTC m=+5160.608059433" Oct 30 04:06:48.161334 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:06:48.161301 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:06:50.493676 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:50.493646 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:06:53.517200 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:53.517132 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:06:58.839800 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:58.839766 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-1"] Oct 30 04:06:58.843168 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:58.843152 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:06:58.870916 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:58.870876 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-grpc\"" Oct 30 04:06:58.871038 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:58.870880 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-http\"" Oct 30 04:06:58.872683 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:58.872662 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-1"] Oct 30 04:06:59.039591 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:59.039555 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:06:59.039766 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:59.039676 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:06:59.039766 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:59.039742 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-config\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:06:59.039883 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:59.039784 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:06:59.039943 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:59.039904 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hmn9\" (UniqueName: \"kubernetes.io/projected/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-kube-api-access-4hmn9\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:06:59.040008 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:59.039944 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-test-loki-shared-secret\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:06:59.140463 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:59.140385 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:06:59.140463 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:59.140460 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:06:59.140691 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:59.140487 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-config\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:06:59.140691 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:59.140503 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:06:59.140691 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:59.140551 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hmn9\" (UniqueName: \"kubernetes.io/projected/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-kube-api-access-4hmn9\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:06:59.140691 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:59.140574 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-test-loki-shared-secret\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:06:59.141242 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:59.141203 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:06:59.141385 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:59.141326 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-config\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:06:59.143039 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:59.143018 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:06:59.143127 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:59.143058 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:06:59.143530 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:59.143509 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-test-loki-shared-secret\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:06:59.164413 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:06:59.164388 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hmn9\" (UniqueName: \"kubernetes.io/projected/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-kube-api-access-4hmn9\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:07:00.558214 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:07:00.558169 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="d50eb753-37ae-4aff-94b6-893a98133df6" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:07:01.966179 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:07:01.966141 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:07:01.996913 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:07:01.996881 4925 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:07:02.067651 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:07:02.067617 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:07:02.067849 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:07:02.067721 4925 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:07:02.082654 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:07:02.082627 4925 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") DevicePath \"csi-8c3a007c96ba81e950cd1bf6462ced09c1187973e7875ac6796d0426158c98a3\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:07:03.251554 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:07:03.251525 4925 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/66d42b06118341040bdf4acaf0eeae99e9dd1cda49be064756edb60338b51e13/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:07:03.278110 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:07:03.278078 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"data-science-lokistack-index-gateway-1\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:07:03.355599 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:07:03.355566 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:07:03.659724 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:07:03.659699 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-1"] Oct 30 04:07:03.661761 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:07:03.661721 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19df8c92_3c72_4731_94fc_bcea4c3bf6e3.slice/crio-341f19274264cbfe990da5420578a47d799fd745e06c5cc9e8d71fddde1799d2 WatchSource:0}: Error finding container 341f19274264cbfe990da5420578a47d799fd745e06c5cc9e8d71fddde1799d2: Status 404 returned error can't find the container with id 341f19274264cbfe990da5420578a47d799fd745e06c5cc9e8d71fddde1799d2 Oct 30 04:07:04.670202 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:07:04.670162 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" event={"ID":"19df8c92-3c72-4731-94fc-bcea4c3bf6e3","Type":"ContainerStarted","Data":"18e9100e927820cf8774975e584379b425625db35c0526e3f878798f15acae19"} Oct 30 04:07:04.670644 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:07:04.670262 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" event={"ID":"19df8c92-3c72-4731-94fc-bcea4c3bf6e3","Type":"ContainerStarted","Data":"341f19274264cbfe990da5420578a47d799fd745e06c5cc9e8d71fddde1799d2"} Oct 30 04:07:04.670644 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:07:04.670291 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:07:10.556582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:07:10.556547 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="d50eb753-37ae-4aff-94b6-893a98133df6" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:07:18.161568 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:07:18.161530 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:07:20.556662 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:07:20.556610 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="d50eb753-37ae-4aff-94b6-893a98133df6" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:07:25.682457 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:07:25.682418 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:07:25.728278 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:07:25.728229 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" podStartSLOduration=30.728216737 podStartE2EDuration="30.728216737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:07:04.714435716 +0000 UTC m=+5185.727713821" watchObservedRunningTime="2026-10-30 04:07:25.728216737 +0000 UTC m=+5206.741494839" Oct 30 04:07:30.556777 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:07:30.556721 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:07:48.162050 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:07:48.162005 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:08:18.161281 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:08:18.161239 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:08:48.161308 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:08:48.161270 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:09:18.161268 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:09:18.161231 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:09:48.162410 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:09:48.162371 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:10:05.200780 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:05.200725 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p"] Oct 30 04:10:05.204457 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:05.204432 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p" Oct 30 04:10:05.232500 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:05.232465 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-6a6878d3\"" Oct 30 04:10:05.234482 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:05.234461 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p"] Oct 30 04:10:05.294963 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:05.294927 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2f25405a-9d8f-447f-8549-51c547f87142-otc-internal\") pod \"data-science-usage-logs-collector-76867dbf9c-zmb4p\" (UID: \"2f25405a-9d8f-447f-8549-51c547f87142\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p" Oct 30 04:10:05.295130 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:05.295098 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpx6s\" (UniqueName: \"kubernetes.io/projected/2f25405a-9d8f-447f-8549-51c547f87142-kube-api-access-qpx6s\") pod \"data-science-usage-logs-collector-76867dbf9c-zmb4p\" (UID: \"2f25405a-9d8f-447f-8549-51c547f87142\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p" Oct 30 04:10:05.396524 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:05.396485 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2f25405a-9d8f-447f-8549-51c547f87142-otc-internal\") pod \"data-science-usage-logs-collector-76867dbf9c-zmb4p\" (UID: \"2f25405a-9d8f-447f-8549-51c547f87142\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p" Oct 30 04:10:05.396702 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:05.396583 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpx6s\" (UniqueName: \"kubernetes.io/projected/2f25405a-9d8f-447f-8549-51c547f87142-kube-api-access-qpx6s\") pod \"data-science-usage-logs-collector-76867dbf9c-zmb4p\" (UID: \"2f25405a-9d8f-447f-8549-51c547f87142\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p" Oct 30 04:10:05.397308 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:05.397286 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2f25405a-9d8f-447f-8549-51c547f87142-otc-internal\") pod \"data-science-usage-logs-collector-76867dbf9c-zmb4p\" (UID: \"2f25405a-9d8f-447f-8549-51c547f87142\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p" Oct 30 04:10:05.421185 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:05.421151 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpx6s\" (UniqueName: \"kubernetes.io/projected/2f25405a-9d8f-447f-8549-51c547f87142-kube-api-access-qpx6s\") pod \"data-science-usage-logs-collector-76867dbf9c-zmb4p\" (UID: \"2f25405a-9d8f-447f-8549-51c547f87142\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p" Oct 30 04:10:05.519788 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:05.519683 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p" Oct 30 04:10:05.819617 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:05.819578 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p"] Oct 30 04:10:05.821307 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:10:05.821277 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f25405a_9d8f_447f_8549_51c547f87142.slice/crio-4d85e558e49f7fdeb2816c34aaeb99eafe320d6382fb7b56c8120ef42124409d WatchSource:0}: Error finding container 4d85e558e49f7fdeb2816c34aaeb99eafe320d6382fb7b56c8120ef42124409d: Status 404 returned error can't find the container with id 4d85e558e49f7fdeb2816c34aaeb99eafe320d6382fb7b56c8120ef42124409d Oct 30 04:10:05.823371 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:05.823340 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:10:05.823486 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:05.823406 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:10:06.559296 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:06.559264 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p" event={"ID":"2f25405a-9d8f-447f-8549-51c547f87142","Type":"ContainerStarted","Data":"9b1fd7237ef21599bf6e777c1ae32321bed78f1668a513f83e381297389cb20c"} Oct 30 04:10:06.559694 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:06.559314 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p" event={"ID":"2f25405a-9d8f-447f-8549-51c547f87142","Type":"ContainerStarted","Data":"4d85e558e49f7fdeb2816c34aaeb99eafe320d6382fb7b56c8120ef42124409d"} Oct 30 04:10:06.601200 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:06.601150 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p" podStartSLOduration=1.601136979 podStartE2EDuration="1.601136979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:10:06.600440034 +0000 UTC m=+5367.613718139" watchObservedRunningTime="2026-10-30 04:10:06.601136979 +0000 UTC m=+5367.614415163" Oct 30 04:10:10.254574 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.254540 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 04:10:10.255033 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.254741 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" podUID="acff1f0b-ceb6-4222-b151-8a77fcce43ee" containerName="loki-compactor" containerID="cri-o://78c8b09a3cd0e67e1dcd5795bb27757106654acd4718645ae9e044e6907926b9" gracePeriod=30 Oct 30 04:10:10.266905 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.266874 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp"] Oct 30 04:10:10.270456 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.270431 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:10.285282 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.285253 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp"] Oct 30 04:10:10.286790 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.286766 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-lifecycle-secret\"" Oct 30 04:10:10.287156 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.287136 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-querier-grpc\"" Oct 30 04:10:10.287245 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.287166 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-querier-http\"" Oct 30 04:10:10.296557 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.296528 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-1"] Oct 30 04:10:10.296775 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.296727 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" podUID="19df8c92-3c72-4731-94fc-bcea4c3bf6e3" containerName="loki-index-gateway" containerID="cri-o://18e9100e927820cf8774975e584379b425625db35c0526e3f878798f15acae19" gracePeriod=30 Oct 30 04:10:10.340812 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.340746 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-nkhbp\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:10.340955 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.340820 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3652c6da-ee63-46ca-8d7f-78fd47177fe0-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-nkhbp\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:10.340955 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.340897 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-nkhbp\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:10.341036 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.341009 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-nkhbp\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:10.341067 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.341050 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgpdl\" (UniqueName: \"kubernetes.io/projected/3652c6da-ee63-46ca-8d7f-78fd47177fe0-kube-api-access-kgpdl\") pod \"data-science-lokistack-querier-56ffbd5fdc-nkhbp\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:10.341116 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.341073 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-nkhbp\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:10.441924 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.441886 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-nkhbp\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:10.441924 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.441928 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3652c6da-ee63-46ca-8d7f-78fd47177fe0-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-nkhbp\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:10.442142 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.441946 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-nkhbp\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:10.442142 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.442088 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-nkhbp\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:10.442230 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.442159 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgpdl\" (UniqueName: \"kubernetes.io/projected/3652c6da-ee63-46ca-8d7f-78fd47177fe0-kube-api-access-kgpdl\") pod \"data-science-lokistack-querier-56ffbd5fdc-nkhbp\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:10.442230 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.442198 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-nkhbp\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:10.442958 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.442927 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3652c6da-ee63-46ca-8d7f-78fd47177fe0-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-nkhbp\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:10.443131 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.443091 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-nkhbp\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:10.445017 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.444986 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-nkhbp\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:10.445229 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.445205 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-nkhbp\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:10.445476 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.445447 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-nkhbp\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:10.466836 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.466803 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgpdl\" (UniqueName: \"kubernetes.io/projected/3652c6da-ee63-46ca-8d7f-78fd47177fe0-kube-api-access-kgpdl\") pod \"data-science-lokistack-querier-56ffbd5fdc-nkhbp\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:10.567873 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.567846 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:10.580144 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.580107 4925 generic.go:356] "Generic (PLEG): container finished" podID="19df8c92-3c72-4731-94fc-bcea4c3bf6e3" containerID="18e9100e927820cf8774975e584379b425625db35c0526e3f878798f15acae19" exitCode=0 Oct 30 04:10:10.580299 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.580152 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" Oct 30 04:10:10.580299 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.580185 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" event={"ID":"19df8c92-3c72-4731-94fc-bcea4c3bf6e3","Type":"ContainerDied","Data":"18e9100e927820cf8774975e584379b425625db35c0526e3f878798f15acae19"} Oct 30 04:10:10.580299 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.580262 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-1" event={"ID":"19df8c92-3c72-4731-94fc-bcea4c3bf6e3","Type":"ContainerDied","Data":"341f19274264cbfe990da5420578a47d799fd745e06c5cc9e8d71fddde1799d2"} Oct 30 04:10:10.580299 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.580285 4925 scope.go:122] "RemoveContainer" containerID="18e9100e927820cf8774975e584379b425625db35c0526e3f878798f15acae19" Oct 30 04:10:10.582430 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.582406 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:10.593487 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.593467 4925 scope.go:122] "RemoveContainer" containerID="18e9100e927820cf8774975e584379b425625db35c0526e3f878798f15acae19" Oct 30 04:10:10.593787 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:10:10.593746 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e9100e927820cf8774975e584379b425625db35c0526e3f878798f15acae19\": container with ID starting with 18e9100e927820cf8774975e584379b425625db35c0526e3f878798f15acae19 not found: ID does not exist" containerID="18e9100e927820cf8774975e584379b425625db35c0526e3f878798f15acae19" Oct 30 04:10:10.593867 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.593800 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e9100e927820cf8774975e584379b425625db35c0526e3f878798f15acae19"} err="failed to get container status \"18e9100e927820cf8774975e584379b425625db35c0526e3f878798f15acae19\": rpc error: code = NotFound desc = could not find container \"18e9100e927820cf8774975e584379b425625db35c0526e3f878798f15acae19\": container with ID starting with 18e9100e927820cf8774975e584379b425625db35c0526e3f878798f15acae19 not found: ID does not exist" Oct 30 04:10:10.644365 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.644307 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-config\" (UniqueName: \"kubernetes.io/configmap/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-config\") pod \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " Oct 30 04:10:10.644544 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.644414 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-kube-api-access-4hmn9\" (UniqueName: \"kubernetes.io/projected/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-kube-api-access-4hmn9\") pod \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " Oct 30 04:10:10.644544 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.644452 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-test-loki-shared-secret\") pod \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " Oct 30 04:10:10.646337 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.644817 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-config" pod "19df8c92-3c72-4731-94fc-bcea4c3bf6e3" (UID: "19df8c92-3c72-4731-94fc-bcea4c3bf6e3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:10.646337 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.644869 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-ca-bundle\") pod \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " Oct 30 04:10:10.646337 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.644938 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-index-gateway-grpc\") pod \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " Oct 30 04:10:10.646337 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.645062 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") pod \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " Oct 30 04:10:10.646337 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.645135 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-index-gateway-http\") pod \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\" (UID: \"19df8c92-3c72-4731-94fc-bcea4c3bf6e3\") " Oct 30 04:10:10.646337 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.645453 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:10:10.646337 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.646299 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-ca-bundle" pod "19df8c92-3c72-4731-94fc-bcea4c3bf6e3" (UID: "19df8c92-3c72-4731-94fc-bcea4c3bf6e3"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:10.650342 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.648049 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-test-loki-shared-secret" pod "19df8c92-3c72-4731-94fc-bcea4c3bf6e3" (UID: "19df8c92-3c72-4731-94fc-bcea4c3bf6e3"). InnerVolumeSpecName "test-loki-shared-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:10:10.650424 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.650358 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-kube-api-access-4hmn9" pod "19df8c92-3c72-4731-94fc-bcea4c3bf6e3" (UID: "19df8c92-3c72-4731-94fc-bcea4c3bf6e3"). InnerVolumeSpecName "kube-api-access-4hmn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:10.650808 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.650784 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-index-gateway-grpc" pod "19df8c92-3c72-4731-94fc-bcea4c3bf6e3" (UID: "19df8c92-3c72-4731-94fc-bcea4c3bf6e3"). InnerVolumeSpecName "data-science-lokistack-index-gateway-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:10:10.652283 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.652263 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00" pod "19df8c92-3c72-4731-94fc-bcea4c3bf6e3" (UID: "19df8c92-3c72-4731-94fc-bcea4c3bf6e3"). InnerVolumeSpecName "pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:10:10.654779 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.654177 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-index-gateway-http" pod "19df8c92-3c72-4731-94fc-bcea4c3bf6e3" (UID: "19df8c92-3c72-4731-94fc-bcea4c3bf6e3"). InnerVolumeSpecName "data-science-lokistack-index-gateway-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:10:10.746660 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.746621 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hmn9\" (UniqueName: \"kubernetes.io/projected/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-kube-api-access-4hmn9\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:10:10.746660 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.746654 4925 reconciler_common.go:299] "Volume detached for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-test-loki-shared-secret\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:10:10.746660 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.746669 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-ca-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:10:10.747011 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.746682 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-index-gateway-grpc\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:10:10.747011 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.746719 4925 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") on node \"ip-10-0-139-19.ec2.internal\" " Oct 30 04:10:10.747011 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.746733 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/19df8c92-3c72-4731-94fc-bcea4c3bf6e3-data-science-lokistack-index-gateway-http\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:10:10.760600 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.760577 4925 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00") on node "ip-10-0-139-19.ec2.internal" Oct 30 04:10:10.847621 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.847590 4925 reconciler_common.go:299] "Volume detached for volume \"pvc-87df5c67-d124-46a5-84f1-c086e0b4f0a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06270f2929765bf00\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"csi-8c3a007c96ba81e950cd1bf6462ced09c1187973e7875ac6796d0426158c98a3\"" Oct 30 04:10:10.897339 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.897303 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp"] Oct 30 04:10:10.901032 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:10:10.901008 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3652c6da_ee63_46ca_8d7f_78fd47177fe0.slice/crio-b563a8f0cd02fa83f3a952b7af29c11586575184bf4bbb8033f446b217a6695a WatchSource:0}: Error finding container b563a8f0cd02fa83f3a952b7af29c11586575184bf4bbb8033f446b217a6695a: Status 404 returned error can't find the container with id b563a8f0cd02fa83f3a952b7af29c11586575184bf4bbb8033f446b217a6695a Oct 30 04:10:10.927773 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.927733 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-1"] Oct 30 04:10:10.933597 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:10.933572 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-1"] Oct 30 04:10:11.121061 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.120980 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19df8c92-3c72-4731-94fc-bcea4c3bf6e3" path="/var/lib/kubelet/pods/19df8c92-3c72-4731-94fc-bcea4c3bf6e3/volumes" Oct 30 04:10:11.207856 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.207828 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:10:11.250732 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.250695 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-compactor-http\") pod \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " Oct 30 04:10:11.250928 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.250743 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-test-loki-shared-secret\") pod \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " Oct 30 04:10:11.250928 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.250813 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-compactor-grpc\") pod \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " Oct 30 04:10:11.250928 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.250842 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-ca-bundle\") pod \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " Oct 30 04:10:11.250928 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.250881 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acff1f0b-ceb6-4222-b151-8a77fcce43ee-kube-api-access-55v74\" (UniqueName: \"kubernetes.io/projected/acff1f0b-ceb6-4222-b151-8a77fcce43ee-kube-api-access-55v74\") pod \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " Oct 30 04:10:11.251170 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.251057 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " Oct 30 04:10:11.251170 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.251122 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acff1f0b-ceb6-4222-b151-8a77fcce43ee-config\" (UniqueName: \"kubernetes.io/configmap/acff1f0b-ceb6-4222-b151-8a77fcce43ee-config\") pod \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\" (UID: \"acff1f0b-ceb6-4222-b151-8a77fcce43ee\") " Oct 30 04:10:11.251321 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.251283 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-ca-bundle" pod "acff1f0b-ceb6-4222-b151-8a77fcce43ee" (UID: "acff1f0b-ceb6-4222-b151-8a77fcce43ee"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:11.251430 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.251410 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-ca-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:10:11.251669 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.251643 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acff1f0b-ceb6-4222-b151-8a77fcce43ee-config" pod "acff1f0b-ceb6-4222-b151-8a77fcce43ee" (UID: "acff1f0b-ceb6-4222-b151-8a77fcce43ee"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:11.254407 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.253917 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acff1f0b-ceb6-4222-b151-8a77fcce43ee-kube-api-access-55v74" pod "acff1f0b-ceb6-4222-b151-8a77fcce43ee" (UID: "acff1f0b-ceb6-4222-b151-8a77fcce43ee"). InnerVolumeSpecName "kube-api-access-55v74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:11.254407 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.254305 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-compactor-grpc" pod "acff1f0b-ceb6-4222-b151-8a77fcce43ee" (UID: "acff1f0b-ceb6-4222-b151-8a77fcce43ee"). InnerVolumeSpecName "data-science-lokistack-compactor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:10:11.254824 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.254557 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-test-loki-shared-secret" pod "acff1f0b-ceb6-4222-b151-8a77fcce43ee" (UID: "acff1f0b-ceb6-4222-b151-8a77fcce43ee"). InnerVolumeSpecName "test-loki-shared-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:10:11.255216 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.255002 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-compactor-http" pod "acff1f0b-ceb6-4222-b151-8a77fcce43ee" (UID: "acff1f0b-ceb6-4222-b151-8a77fcce43ee"). InnerVolumeSpecName "data-science-lokistack-compactor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:10:11.256335 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.256299 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e" pod "acff1f0b-ceb6-4222-b151-8a77fcce43ee" (UID: "acff1f0b-ceb6-4222-b151-8a77fcce43ee"). InnerVolumeSpecName "pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:10:11.352200 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.352169 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-compactor-http\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:10:11.352355 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.352211 4925 reconciler_common.go:299] "Volume detached for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-test-loki-shared-secret\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:10:11.352355 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.352222 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/acff1f0b-ceb6-4222-b151-8a77fcce43ee-data-science-lokistack-compactor-grpc\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:10:11.352355 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.352231 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55v74\" (UniqueName: \"kubernetes.io/projected/acff1f0b-ceb6-4222-b151-8a77fcce43ee-kube-api-access-55v74\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:10:11.352355 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.352265 4925 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") on node \"ip-10-0-139-19.ec2.internal\" " Oct 30 04:10:11.352355 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.352278 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acff1f0b-ceb6-4222-b151-8a77fcce43ee-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:10:11.366338 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.366315 4925 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e") on node "ip-10-0-139-19.ec2.internal" Oct 30 04:10:11.453673 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.453596 4925 reconciler_common.go:299] "Volume detached for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"csi-3faa7a4919429bfae9e5fe05bb9f7e440b0f7cbd5797a7f1cb9f75e405a5c758\"" Oct 30 04:10:11.588298 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.588239 4925 generic.go:356] "Generic (PLEG): container finished" podID="acff1f0b-ceb6-4222-b151-8a77fcce43ee" containerID="78c8b09a3cd0e67e1dcd5795bb27757106654acd4718645ae9e044e6907926b9" exitCode=0 Oct 30 04:10:11.588513 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.588327 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"acff1f0b-ceb6-4222-b151-8a77fcce43ee","Type":"ContainerDied","Data":"78c8b09a3cd0e67e1dcd5795bb27757106654acd4718645ae9e044e6907926b9"} Oct 30 04:10:11.588513 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.588372 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"acff1f0b-ceb6-4222-b151-8a77fcce43ee","Type":"ContainerDied","Data":"bbf503be312695a47c9897daa9f1be286d9dc914f09679a0cb802777ab9c12be"} Oct 30 04:10:11.588513 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.588396 4925 scope.go:122] "RemoveContainer" containerID="78c8b09a3cd0e67e1dcd5795bb27757106654acd4718645ae9e044e6907926b9" Oct 30 04:10:11.588513 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.588487 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:10:11.593683 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.593648 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" event={"ID":"3652c6da-ee63-46ca-8d7f-78fd47177fe0","Type":"ContainerStarted","Data":"500ab871406632a74ecde76957e689e81472702e267d0580f3befb12fc134813"} Oct 30 04:10:11.593855 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.593713 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" event={"ID":"3652c6da-ee63-46ca-8d7f-78fd47177fe0","Type":"ContainerStarted","Data":"b563a8f0cd02fa83f3a952b7af29c11586575184bf4bbb8033f446b217a6695a"} Oct 30 04:10:11.593855 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.593830 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:11.601326 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.601309 4925 scope.go:122] "RemoveContainer" containerID="78c8b09a3cd0e67e1dcd5795bb27757106654acd4718645ae9e044e6907926b9" Oct 30 04:10:11.601580 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:10:11.601563 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78c8b09a3cd0e67e1dcd5795bb27757106654acd4718645ae9e044e6907926b9\": container with ID starting with 78c8b09a3cd0e67e1dcd5795bb27757106654acd4718645ae9e044e6907926b9 not found: ID does not exist" containerID="78c8b09a3cd0e67e1dcd5795bb27757106654acd4718645ae9e044e6907926b9" Oct 30 04:10:11.601631 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.601588 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c8b09a3cd0e67e1dcd5795bb27757106654acd4718645ae9e044e6907926b9"} err="failed to get container status \"78c8b09a3cd0e67e1dcd5795bb27757106654acd4718645ae9e044e6907926b9\": rpc error: code = NotFound desc = could not find container \"78c8b09a3cd0e67e1dcd5795bb27757106654acd4718645ae9e044e6907926b9\": container with ID starting with 78c8b09a3cd0e67e1dcd5795bb27757106654acd4718645ae9e044e6907926b9 not found: ID does not exist" Oct 30 04:10:11.643829 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.643797 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 04:10:11.647789 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.647765 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 04:10:11.689240 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.689176 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" podStartSLOduration=1.689157213 podStartE2EDuration="1.689157213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:10:11.688712587 +0000 UTC m=+5372.701990714" watchObservedRunningTime="2026-10-30 04:10:11.689157213 +0000 UTC m=+5372.702435316" Oct 30 04:10:11.704504 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.704412 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p"] Oct 30 04:10:11.704690 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.704648 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p" podUID="2f25405a-9d8f-447f-8549-51c547f87142" containerName="otc-container" containerID="cri-o://9b1fd7237ef21599bf6e777c1ae32321bed78f1668a513f83e381297389cb20c" gracePeriod=30 Oct 30 04:10:11.753954 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.753923 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq"] Oct 30 04:10:11.754360 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.754343 4925 cpu_manager.go:414] "containerMap: removing container" podUID="acff1f0b-ceb6-4222-b151-8a77fcce43ee" containerName="loki-compactor" Oct 30 04:10:11.754407 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.754361 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acff1f0b-ceb6-4222-b151-8a77fcce43ee" containerName="loki-compactor" Oct 30 04:10:11.754407 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.754377 4925 cpu_manager.go:414] "containerMap: removing container" podUID="19df8c92-3c72-4731-94fc-bcea4c3bf6e3" containerName="loki-index-gateway" Oct 30 04:10:11.754407 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.754381 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19df8c92-3c72-4731-94fc-bcea4c3bf6e3" containerName="loki-index-gateway" Oct 30 04:10:11.754501 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.754434 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="acff1f0b-ceb6-4222-b151-8a77fcce43ee" containerName="loki-compactor" Oct 30 04:10:11.754501 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.754443 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="19df8c92-3c72-4731-94fc-bcea4c3bf6e3" containerName="loki-index-gateway" Oct 30 04:10:11.757844 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.757825 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq" Oct 30 04:10:11.777992 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.777964 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq"] Oct 30 04:10:11.812946 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.812915 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-94m9t\"" Oct 30 04:10:11.857156 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.857116 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwj49\" (UniqueName: \"kubernetes.io/projected/3b5f1de5-ab71-473c-9e16-d0531f199ce6-kube-api-access-jwj49\") pod \"data-science-usage-logs-collector-54c8646c45-q8cdq\" (UID: \"3b5f1de5-ab71-473c-9e16-d0531f199ce6\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq" Oct 30 04:10:11.857331 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.857277 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3b5f1de5-ab71-473c-9e16-d0531f199ce6-otc-internal\") pod \"data-science-usage-logs-collector-54c8646c45-q8cdq\" (UID: \"3b5f1de5-ab71-473c-9e16-d0531f199ce6\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq" Oct 30 04:10:11.957790 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.957710 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3b5f1de5-ab71-473c-9e16-d0531f199ce6-otc-internal\") pod \"data-science-usage-logs-collector-54c8646c45-q8cdq\" (UID: \"3b5f1de5-ab71-473c-9e16-d0531f199ce6\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq" Oct 30 04:10:11.957933 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.957810 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwj49\" (UniqueName: \"kubernetes.io/projected/3b5f1de5-ab71-473c-9e16-d0531f199ce6-kube-api-access-jwj49\") pod \"data-science-usage-logs-collector-54c8646c45-q8cdq\" (UID: \"3b5f1de5-ab71-473c-9e16-d0531f199ce6\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq" Oct 30 04:10:11.958456 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.958430 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3b5f1de5-ab71-473c-9e16-d0531f199ce6-otc-internal\") pod \"data-science-usage-logs-collector-54c8646c45-q8cdq\" (UID: \"3b5f1de5-ab71-473c-9e16-d0531f199ce6\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq" Oct 30 04:10:11.960833 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.960817 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p" Oct 30 04:10:11.981398 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:11.981365 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwj49\" (UniqueName: \"kubernetes.io/projected/3b5f1de5-ab71-473c-9e16-d0531f199ce6-kube-api-access-jwj49\") pod \"data-science-usage-logs-collector-54c8646c45-q8cdq\" (UID: \"3b5f1de5-ab71-473c-9e16-d0531f199ce6\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq" Oct 30 04:10:12.058947 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.058915 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f25405a-9d8f-447f-8549-51c547f87142-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2f25405a-9d8f-447f-8549-51c547f87142-otc-internal\") pod \"2f25405a-9d8f-447f-8549-51c547f87142\" (UID: \"2f25405a-9d8f-447f-8549-51c547f87142\") " Oct 30 04:10:12.059156 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.058976 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f25405a-9d8f-447f-8549-51c547f87142-kube-api-access-qpx6s\" (UniqueName: \"kubernetes.io/projected/2f25405a-9d8f-447f-8549-51c547f87142-kube-api-access-qpx6s\") pod \"2f25405a-9d8f-447f-8549-51c547f87142\" (UID: \"2f25405a-9d8f-447f-8549-51c547f87142\") " Oct 30 04:10:12.059323 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.059298 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f25405a-9d8f-447f-8549-51c547f87142-otc-internal" pod "2f25405a-9d8f-447f-8549-51c547f87142" (UID: "2f25405a-9d8f-447f-8549-51c547f87142"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:12.061109 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.061085 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f25405a-9d8f-447f-8549-51c547f87142-kube-api-access-qpx6s" pod "2f25405a-9d8f-447f-8549-51c547f87142" (UID: "2f25405a-9d8f-447f-8549-51c547f87142"). InnerVolumeSpecName "kube-api-access-qpx6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:12.089160 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.089131 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq" Oct 30 04:10:12.160289 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.160251 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2f25405a-9d8f-447f-8549-51c547f87142-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:10:12.160289 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.160287 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpx6s\" (UniqueName: \"kubernetes.io/projected/2f25405a-9d8f-447f-8549-51c547f87142-kube-api-access-qpx6s\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:10:12.387994 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.387961 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq"] Oct 30 04:10:12.389923 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:10:12.389895 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5f1de5_ab71_473c_9e16_d0531f199ce6.slice/crio-826ade324ab860610100f977813d7c7279d469d9889e803f65413b8a4a5cfe0d WatchSource:0}: Error finding container 826ade324ab860610100f977813d7c7279d469d9889e803f65413b8a4a5cfe0d: Status 404 returned error can't find the container with id 826ade324ab860610100f977813d7c7279d469d9889e803f65413b8a4a5cfe0d Oct 30 04:10:12.392223 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.392192 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:10:12.392338 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.392249 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:10:12.601265 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.601232 4925 generic.go:356] "Generic (PLEG): container finished" podID="2f25405a-9d8f-447f-8549-51c547f87142" containerID="9b1fd7237ef21599bf6e777c1ae32321bed78f1668a513f83e381297389cb20c" exitCode=0 Oct 30 04:10:12.601461 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.601284 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p" Oct 30 04:10:12.601461 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.601294 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p" event={"ID":"2f25405a-9d8f-447f-8549-51c547f87142","Type":"ContainerDied","Data":"9b1fd7237ef21599bf6e777c1ae32321bed78f1668a513f83e381297389cb20c"} Oct 30 04:10:12.601461 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.601344 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p" event={"ID":"2f25405a-9d8f-447f-8549-51c547f87142","Type":"ContainerDied","Data":"4d85e558e49f7fdeb2816c34aaeb99eafe320d6382fb7b56c8120ef42124409d"} Oct 30 04:10:12.601461 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.601374 4925 scope.go:122] "RemoveContainer" containerID="9b1fd7237ef21599bf6e777c1ae32321bed78f1668a513f83e381297389cb20c" Oct 30 04:10:12.604299 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.604271 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq" event={"ID":"3b5f1de5-ab71-473c-9e16-d0531f199ce6","Type":"ContainerStarted","Data":"e2b41786c00232509ebbabc6e37744947b96352357a82e1d35c7e15bcdc79895"} Oct 30 04:10:12.604419 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.604328 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq" event={"ID":"3b5f1de5-ab71-473c-9e16-d0531f199ce6","Type":"ContainerStarted","Data":"826ade324ab860610100f977813d7c7279d469d9889e803f65413b8a4a5cfe0d"} Oct 30 04:10:12.611361 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.611342 4925 scope.go:122] "RemoveContainer" containerID="9b1fd7237ef21599bf6e777c1ae32321bed78f1668a513f83e381297389cb20c" Oct 30 04:10:12.611620 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:10:12.611602 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b1fd7237ef21599bf6e777c1ae32321bed78f1668a513f83e381297389cb20c\": container with ID starting with 9b1fd7237ef21599bf6e777c1ae32321bed78f1668a513f83e381297389cb20c not found: ID does not exist" containerID="9b1fd7237ef21599bf6e777c1ae32321bed78f1668a513f83e381297389cb20c" Oct 30 04:10:12.611690 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.611626 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b1fd7237ef21599bf6e777c1ae32321bed78f1668a513f83e381297389cb20c"} err="failed to get container status \"9b1fd7237ef21599bf6e777c1ae32321bed78f1668a513f83e381297389cb20c\": rpc error: code = NotFound desc = could not find container \"9b1fd7237ef21599bf6e777c1ae32321bed78f1668a513f83e381297389cb20c\": container with ID starting with 9b1fd7237ef21599bf6e777c1ae32321bed78f1668a513f83e381297389cb20c not found: ID does not exist" Oct 30 04:10:12.657520 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.657464 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq" podStartSLOduration=1.657451312 podStartE2EDuration="1.657451312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:10:12.656943986 +0000 UTC m=+5373.670222089" watchObservedRunningTime="2026-10-30 04:10:12.657451312 +0000 UTC m=+5373.670729417" Oct 30 04:10:12.698713 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.698682 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p"] Oct 30 04:10:12.707671 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:12.707638 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76867dbf9c-zmb4p"] Oct 30 04:10:13.120028 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:13.119990 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f25405a-9d8f-447f-8549-51c547f87142" path="/var/lib/kubelet/pods/2f25405a-9d8f-447f-8549-51c547f87142/volumes" Oct 30 04:10:13.120424 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:13.120411 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acff1f0b-ceb6-4222-b151-8a77fcce43ee" path="/var/lib/kubelet/pods/acff1f0b-ceb6-4222-b151-8a77fcce43ee/volumes" Oct 30 04:10:13.395899 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:13.395818 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq"] Oct 30 04:10:13.610240 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:13.610211 4925 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq" secret="" err="secret \"data-science-usage-logs-collector-dockercfg-94m9t\" not found" Oct 30 04:10:13.775277 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:10:13.775188 4925 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-usage-logs-collector-6a6878d3: configmap "data-science-usage-logs-collector-6a6878d3" not found Oct 30 04:10:13.775277 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:10:13.775275 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b5f1de5-ab71-473c-9e16-d0531f199ce6-otc-internal podName:3b5f1de5-ab71-473c-9e16-d0531f199ce6 nodeName:}" failed. No retries permitted until 2026-10-30 04:10:14.275253151 +0000 UTC m=+5375.288531237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/3b5f1de5-ab71-473c-9e16-d0531f199ce6-otc-internal") pod "data-science-usage-logs-collector-54c8646c45-q8cdq" (UID: "3b5f1de5-ab71-473c-9e16-d0531f199ce6") : configmap "data-science-usage-logs-collector-6a6878d3" not found Oct 30 04:10:14.280861 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:10:14.280827 4925 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-usage-logs-collector-6a6878d3: configmap "data-science-usage-logs-collector-6a6878d3" not found Oct 30 04:10:14.281028 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:10:14.280902 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b5f1de5-ab71-473c-9e16-d0531f199ce6-otc-internal podName:3b5f1de5-ab71-473c-9e16-d0531f199ce6 nodeName:}" failed. No retries permitted until 2026-10-30 04:10:15.28088711 +0000 UTC m=+5376.294165191 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/3b5f1de5-ab71-473c-9e16-d0531f199ce6-otc-internal") pod "data-science-usage-logs-collector-54c8646c45-q8cdq" (UID: "3b5f1de5-ab71-473c-9e16-d0531f199ce6") : configmap "data-science-usage-logs-collector-6a6878d3" not found Oct 30 04:10:14.614605 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:14.614567 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq" podUID="3b5f1de5-ab71-473c-9e16-d0531f199ce6" containerName="otc-container" containerID="cri-o://e2b41786c00232509ebbabc6e37744947b96352357a82e1d35c7e15bcdc79895" gracePeriod=30 Oct 30 04:10:14.876186 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:14.876128 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq" Oct 30 04:10:14.986650 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:14.986606 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5f1de5-ab71-473c-9e16-d0531f199ce6-kube-api-access-jwj49\" (UniqueName: \"kubernetes.io/projected/3b5f1de5-ab71-473c-9e16-d0531f199ce6-kube-api-access-jwj49\") pod \"3b5f1de5-ab71-473c-9e16-d0531f199ce6\" (UID: \"3b5f1de5-ab71-473c-9e16-d0531f199ce6\") " Oct 30 04:10:14.986856 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:14.986721 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b5f1de5-ab71-473c-9e16-d0531f199ce6-otc-internal\" (UniqueName: \"kubernetes.io/configmap/3b5f1de5-ab71-473c-9e16-d0531f199ce6-otc-internal\") pod \"3b5f1de5-ab71-473c-9e16-d0531f199ce6\" (UID: \"3b5f1de5-ab71-473c-9e16-d0531f199ce6\") " Oct 30 04:10:14.987177 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:14.987146 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5f1de5-ab71-473c-9e16-d0531f199ce6-otc-internal" pod "3b5f1de5-ab71-473c-9e16-d0531f199ce6" (UID: "3b5f1de5-ab71-473c-9e16-d0531f199ce6"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:14.988844 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:14.988823 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5f1de5-ab71-473c-9e16-d0531f199ce6-kube-api-access-jwj49" pod "3b5f1de5-ab71-473c-9e16-d0531f199ce6" (UID: "3b5f1de5-ab71-473c-9e16-d0531f199ce6"). InnerVolumeSpecName "kube-api-access-jwj49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:15.087801 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:15.087733 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwj49\" (UniqueName: \"kubernetes.io/projected/3b5f1de5-ab71-473c-9e16-d0531f199ce6-kube-api-access-jwj49\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:10:15.087801 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:15.087796 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3b5f1de5-ab71-473c-9e16-d0531f199ce6-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:10:15.620961 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:15.620921 4925 generic.go:356] "Generic (PLEG): container finished" podID="3b5f1de5-ab71-473c-9e16-d0531f199ce6" containerID="e2b41786c00232509ebbabc6e37744947b96352357a82e1d35c7e15bcdc79895" exitCode=0 Oct 30 04:10:15.621368 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:15.620973 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq" Oct 30 04:10:15.621368 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:15.620972 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq" event={"ID":"3b5f1de5-ab71-473c-9e16-d0531f199ce6","Type":"ContainerDied","Data":"e2b41786c00232509ebbabc6e37744947b96352357a82e1d35c7e15bcdc79895"} Oct 30 04:10:15.621368 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:15.621116 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq" event={"ID":"3b5f1de5-ab71-473c-9e16-d0531f199ce6","Type":"ContainerDied","Data":"826ade324ab860610100f977813d7c7279d469d9889e803f65413b8a4a5cfe0d"} Oct 30 04:10:15.621368 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:15.621143 4925 scope.go:122] "RemoveContainer" containerID="e2b41786c00232509ebbabc6e37744947b96352357a82e1d35c7e15bcdc79895" Oct 30 04:10:15.630162 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:15.630141 4925 scope.go:122] "RemoveContainer" containerID="e2b41786c00232509ebbabc6e37744947b96352357a82e1d35c7e15bcdc79895" Oct 30 04:10:15.630426 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:10:15.630406 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b41786c00232509ebbabc6e37744947b96352357a82e1d35c7e15bcdc79895\": container with ID starting with e2b41786c00232509ebbabc6e37744947b96352357a82e1d35c7e15bcdc79895 not found: ID does not exist" containerID="e2b41786c00232509ebbabc6e37744947b96352357a82e1d35c7e15bcdc79895" Oct 30 04:10:15.630486 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:15.630438 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b41786c00232509ebbabc6e37744947b96352357a82e1d35c7e15bcdc79895"} err="failed to get container status \"e2b41786c00232509ebbabc6e37744947b96352357a82e1d35c7e15bcdc79895\": rpc error: code = NotFound desc = could not find container \"e2b41786c00232509ebbabc6e37744947b96352357a82e1d35c7e15bcdc79895\": container with ID starting with e2b41786c00232509ebbabc6e37744947b96352357a82e1d35c7e15bcdc79895 not found: ID does not exist" Oct 30 04:10:15.680391 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:15.680362 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq"] Oct 30 04:10:15.681936 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:15.681913 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-q8cdq"] Oct 30 04:10:17.119404 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:17.119368 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b5f1de5-ab71-473c-9e16-d0531f199ce6" path="/var/lib/kubelet/pods/3b5f1de5-ab71-473c-9e16-d0531f199ce6/volumes" Oct 30 04:10:18.161819 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:10:18.161774 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:10:32.610067 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:32.610037 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:10:39.678883 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:39.678741 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 04:10:39.687505 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:39.687482 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 04:10:39.694387 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:39.694368 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 04:10:39.702049 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:39.702031 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 04:10:46.876978 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:46.876936 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 04:10:46.877494 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:46.877463 4925 cpu_manager.go:414] "containerMap: removing container" podUID="3b5f1de5-ab71-473c-9e16-d0531f199ce6" containerName="otc-container" Oct 30 04:10:46.877494 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:46.877483 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b5f1de5-ab71-473c-9e16-d0531f199ce6" containerName="otc-container" Oct 30 04:10:46.877711 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:46.877538 4925 cpu_manager.go:414] "containerMap: removing container" podUID="2f25405a-9d8f-447f-8549-51c547f87142" containerName="otc-container" Oct 30 04:10:46.877711 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:46.877548 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f25405a-9d8f-447f-8549-51c547f87142" containerName="otc-container" Oct 30 04:10:46.877711 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:46.877640 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f25405a-9d8f-447f-8549-51c547f87142" containerName="otc-container" Oct 30 04:10:46.877711 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:46.877655 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b5f1de5-ab71-473c-9e16-d0531f199ce6" containerName="otc-container" Oct 30 04:10:46.882280 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:46.882260 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:10:46.912497 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:46.912473 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-grpc\"" Oct 30 04:10:46.912497 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:46.912486 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-http\"" Oct 30 04:10:46.914489 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:46.914467 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 04:10:47.045538 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:47.045507 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6829d385-4667-4ef5-9026-5fd116e70ae0-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:10:47.045700 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:47.045565 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:10:47.045700 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:47.045591 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:10:47.045700 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:47.045682 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:10:47.045826 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:47.045715 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:10:47.045826 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:47.045744 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfs8j\" (UniqueName: \"kubernetes.io/projected/6829d385-4667-4ef5-9026-5fd116e70ae0-kube-api-access-qfs8j\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:10:47.146719 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:47.146635 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6829d385-4667-4ef5-9026-5fd116e70ae0-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:10:47.146719 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:47.146697 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:10:47.146976 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:47.146851 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:10:47.146976 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:47.146900 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:10:47.146976 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:47.146935 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:10:47.146976 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:47.146963 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfs8j\" (UniqueName: \"kubernetes.io/projected/6829d385-4667-4ef5-9026-5fd116e70ae0-kube-api-access-qfs8j\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:10:47.147633 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:47.147608 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6829d385-4667-4ef5-9026-5fd116e70ae0-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:10:47.147745 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:47.147652 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:10:47.149301 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:47.149284 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:10:47.149422 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:47.149352 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:10:47.149485 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:47.149418 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:10:47.170707 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:10:47.170677 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfs8j\" (UniqueName: \"kubernetes.io/projected/6829d385-4667-4ef5-9026-5fd116e70ae0-kube-api-access-qfs8j\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:10:48.160568 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:10:48.160526 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:11:02.781108 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:02.781062 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:11:02.812338 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:02.812306 4925 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:11:02.882187 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:02.882150 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:11:02.882334 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:02.882279 4925 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:11:02.896957 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:02.896935 4925 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") DevicePath \"csi-09f7dd9c6d02fc70105c8ae46375645a3d310e7e56fb2e91114833c2c076d551\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:11:03.001877 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:03.001850 4925 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d0ba80d454beb69db9470fc45e5a47dc3c954c5997bec8b3d02403b351bf919d/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:11:03.028234 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:03.028209 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:11:03.093810 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:03.093781 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:11:03.395171 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:03.395140 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 04:11:03.396698 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:11:03.396666 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6829d385_4667_4ef5_9026_5fd116e70ae0.slice/crio-74029d541660e99ca52c8a6a01eec7cc0ecca692b7b1bb09bd8f5fdfe2d1098c WatchSource:0}: Error finding container 74029d541660e99ca52c8a6a01eec7cc0ecca692b7b1bb09bd8f5fdfe2d1098c: Status 404 returned error can't find the container with id 74029d541660e99ca52c8a6a01eec7cc0ecca692b7b1bb09bd8f5fdfe2d1098c Oct 30 04:11:03.854326 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:03.854285 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"6829d385-4667-4ef5-9026-5fd116e70ae0","Type":"ContainerStarted","Data":"79370ba5f91fb40431bfce138d78aa1a734edaaca7b08bbffbac3bbdde93fccc"} Oct 30 04:11:03.854803 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:03.854419 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:11:03.854803 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:03.854458 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"6829d385-4667-4ef5-9026-5fd116e70ae0","Type":"ContainerStarted","Data":"74029d541660e99ca52c8a6a01eec7cc0ecca692b7b1bb09bd8f5fdfe2d1098c"} Oct 30 04:11:03.912996 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:03.912947 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podStartSLOduration=17.91293368 podStartE2EDuration="17.91293368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:11:03.912446948 +0000 UTC m=+5424.925725050" watchObservedRunningTime="2026-10-30 04:11:03.91293368 +0000 UTC m=+5424.926211782" Oct 30 04:11:18.161955 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:11:18.161920 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:11:24.864159 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:24.864078 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:11:43.503459 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:43.503424 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 04:11:43.504140 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:43.503643 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="d50eb753-37ae-4aff-94b6-893a98133df6" containerName="loki-ingester" containerID="cri-o://e14cd81acb74a0e435c11727be5d6d9f47a97a9d3b21bd8213de20c88c44a9e7" gracePeriod=600 Oct 30 04:11:44.347460 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:11:44.347429 4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50eb753_37ae_4aff_94b6_893a98133df6.slice/crio-e14cd81acb74a0e435c11727be5d6d9f47a97a9d3b21bd8213de20c88c44a9e7.scope\": RecentStats: unable to find data in memory cache]" Oct 30 04:11:44.559257 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.559233 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:44.649517 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.649481 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-test-loki-shared-secret\") pod \"d50eb753-37ae-4aff-94b6-893a98133df6\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " Oct 30 04:11:44.649517 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.649523 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d50eb753-37ae-4aff-94b6-893a98133df6-config\" (UniqueName: \"kubernetes.io/configmap/d50eb753-37ae-4aff-94b6-893a98133df6-config\") pod \"d50eb753-37ae-4aff-94b6-893a98133df6\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " Oct 30 04:11:44.649809 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.649560 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ca-bundle\") pod \"d50eb753-37ae-4aff-94b6-893a98133df6\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " Oct 30 04:11:44.649809 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.649584 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ingester-grpc\") pod \"d50eb753-37ae-4aff-94b6-893a98133df6\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " Oct 30 04:11:44.649809 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.649700 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"d50eb753-37ae-4aff-94b6-893a98133df6\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " Oct 30 04:11:44.649809 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.649800 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"d50eb753-37ae-4aff-94b6-893a98133df6\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " Oct 30 04:11:44.650044 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.649850 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d50eb753-37ae-4aff-94b6-893a98133df6-kube-api-access-lfzx4\" (UniqueName: \"kubernetes.io/projected/d50eb753-37ae-4aff-94b6-893a98133df6-kube-api-access-lfzx4\") pod \"d50eb753-37ae-4aff-94b6-893a98133df6\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " Oct 30 04:11:44.650044 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.649905 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ingester-http\") pod \"d50eb753-37ae-4aff-94b6-893a98133df6\" (UID: \"d50eb753-37ae-4aff-94b6-893a98133df6\") " Oct 30 04:11:44.650044 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.650011 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50eb753-37ae-4aff-94b6-893a98133df6-config" pod "d50eb753-37ae-4aff-94b6-893a98133df6" (UID: "d50eb753-37ae-4aff-94b6-893a98133df6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:11:44.650044 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.650022 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ca-bundle" pod "d50eb753-37ae-4aff-94b6-893a98133df6" (UID: "d50eb753-37ae-4aff-94b6-893a98133df6"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:11:44.650244 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.650230 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d50eb753-37ae-4aff-94b6-893a98133df6-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:11:44.650303 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.650253 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ca-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:11:44.652254 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.652230 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ingester-grpc" pod "d50eb753-37ae-4aff-94b6-893a98133df6" (UID: "d50eb753-37ae-4aff-94b6-893a98133df6"). InnerVolumeSpecName "data-science-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:11:44.652254 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.652236 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50eb753-37ae-4aff-94b6-893a98133df6-kube-api-access-lfzx4" pod "d50eb753-37ae-4aff-94b6-893a98133df6" (UID: "d50eb753-37ae-4aff-94b6-893a98133df6"). InnerVolumeSpecName "kube-api-access-lfzx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:11:44.652479 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.652370 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-test-loki-shared-secret" pod "d50eb753-37ae-4aff-94b6-893a98133df6" (UID: "d50eb753-37ae-4aff-94b6-893a98133df6"). InnerVolumeSpecName "test-loki-shared-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:11:44.652629 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.652607 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ingester-http" pod "d50eb753-37ae-4aff-94b6-893a98133df6" (UID: "d50eb753-37ae-4aff-94b6-893a98133df6"). InnerVolumeSpecName "data-science-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:11:44.652950 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.652933 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18" pod "d50eb753-37ae-4aff-94b6-893a98133df6" (UID: "d50eb753-37ae-4aff-94b6-893a98133df6"). InnerVolumeSpecName "pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:11:44.653048 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.652985 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f" pod "d50eb753-37ae-4aff-94b6-893a98133df6" (UID: "d50eb753-37ae-4aff-94b6-893a98133df6"). InnerVolumeSpecName "pvc-1477050d-bd59-4ae1-b712-0b73e1306b06". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:11:44.751459 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.751429 4925 reconciler_common.go:299] "Volume detached for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-test-loki-shared-secret\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:11:44.751459 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.751456 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ingester-grpc\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:11:44.751613 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.751481 4925 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") on node \"ip-10-0-139-19.ec2.internal\" " Oct 30 04:11:44.751613 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.751495 4925 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") on node \"ip-10-0-139-19.ec2.internal\" " Oct 30 04:11:44.751613 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.751505 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfzx4\" (UniqueName: \"kubernetes.io/projected/d50eb753-37ae-4aff-94b6-893a98133df6-kube-api-access-lfzx4\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:11:44.751613 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.751514 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/d50eb753-37ae-4aff-94b6-893a98133df6-data-science-lokistack-ingester-http\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:11:44.765441 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.765420 4925 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-1477050d-bd59-4ae1-b712-0b73e1306b06" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f") on node "ip-10-0-139-19.ec2.internal" Oct 30 04:11:44.766850 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.766835 4925 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18") on node "ip-10-0-139-19.ec2.internal" Oct 30 04:11:44.852856 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.852827 4925 reconciler_common.go:299] "Volume detached for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"csi-1cbd6671aaaf13073dd4ac6088fd398781af87903101ff82770e0956bdbc63f1\"" Oct 30 04:11:44.852856 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:44.852855 4925 reconciler_common.go:299] "Volume detached for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"csi-d631f5804e757beb9cee949b1ba08adddfe0f2a6d74da288dfa6a5946ab2fad2\"" Oct 30 04:11:45.060036 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.059949 4925 generic.go:356] "Generic (PLEG): container finished" podID="d50eb753-37ae-4aff-94b6-893a98133df6" containerID="e14cd81acb74a0e435c11727be5d6d9f47a97a9d3b21bd8213de20c88c44a9e7" exitCode=0 Oct 30 04:11:45.060036 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.059997 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:45.060036 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.060015 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"d50eb753-37ae-4aff-94b6-893a98133df6","Type":"ContainerDied","Data":"e14cd81acb74a0e435c11727be5d6d9f47a97a9d3b21bd8213de20c88c44a9e7"} Oct 30 04:11:45.060278 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.060064 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"d50eb753-37ae-4aff-94b6-893a98133df6","Type":"ContainerDied","Data":"822d50c5d6aa83bffe8b3978ded2f46d073620179292792e222426d2deab1264"} Oct 30 04:11:45.060278 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.060092 4925 scope.go:122] "RemoveContainer" containerID="e14cd81acb74a0e435c11727be5d6d9f47a97a9d3b21bd8213de20c88c44a9e7" Oct 30 04:11:45.070290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.070269 4925 scope.go:122] "RemoveContainer" containerID="e14cd81acb74a0e435c11727be5d6d9f47a97a9d3b21bd8213de20c88c44a9e7" Oct 30 04:11:45.070559 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:11:45.070538 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e14cd81acb74a0e435c11727be5d6d9f47a97a9d3b21bd8213de20c88c44a9e7\": container with ID starting with e14cd81acb74a0e435c11727be5d6d9f47a97a9d3b21bd8213de20c88c44a9e7 not found: ID does not exist" containerID="e14cd81acb74a0e435c11727be5d6d9f47a97a9d3b21bd8213de20c88c44a9e7" Oct 30 04:11:45.070623 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.070571 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e14cd81acb74a0e435c11727be5d6d9f47a97a9d3b21bd8213de20c88c44a9e7"} err="failed to get container status \"e14cd81acb74a0e435c11727be5d6d9f47a97a9d3b21bd8213de20c88c44a9e7\": rpc error: code = NotFound desc = could not find container \"e14cd81acb74a0e435c11727be5d6d9f47a97a9d3b21bd8213de20c88c44a9e7\": container with ID starting with e14cd81acb74a0e435c11727be5d6d9f47a97a9d3b21bd8213de20c88c44a9e7 not found: ID does not exist" Oct 30 04:11:45.112583 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.112553 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 04:11:45.125653 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.125628 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 04:11:45.159852 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.159826 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 04:11:45.160212 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.160199 4925 cpu_manager.go:414] "containerMap: removing container" podUID="d50eb753-37ae-4aff-94b6-893a98133df6" containerName="loki-ingester" Oct 30 04:11:45.160255 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.160214 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d50eb753-37ae-4aff-94b6-893a98133df6" containerName="loki-ingester" Oct 30 04:11:45.160320 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.160309 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d50eb753-37ae-4aff-94b6-893a98133df6" containerName="loki-ingester" Oct 30 04:11:45.164741 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.164724 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:45.180080 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.180062 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-grpc\"" Oct 30 04:11:45.180169 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.180134 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-http\"" Oct 30 04:11:45.185660 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.185639 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 04:11:45.357466 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.357436 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5c6c\" (UniqueName: \"kubernetes.io/projected/52925eb5-4cb7-4483-b0fd-a299621be317-kube-api-access-k5c6c\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:45.357609 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.357484 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:45.357609 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.357565 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:45.357688 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.357616 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:45.357688 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.357642 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/52925eb5-4cb7-4483-b0fd-a299621be317-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:45.357799 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.357688 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:45.458124 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.458095 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:45.458308 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.458134 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/52925eb5-4cb7-4483-b0fd-a299621be317-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:45.458308 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.458162 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:45.458308 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.458228 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5c6c\" (UniqueName: \"kubernetes.io/projected/52925eb5-4cb7-4483-b0fd-a299621be317-kube-api-access-k5c6c\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:45.458308 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.458262 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:45.458308 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.458290 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:45.459029 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.458999 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:45.459169 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.459090 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/52925eb5-4cb7-4483-b0fd-a299621be317-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:45.460798 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.460769 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:45.460930 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.460855 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:45.460995 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.460926 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:45.481829 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:45.481805 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5c6c\" (UniqueName: \"kubernetes.io/projected/52925eb5-4cb7-4483-b0fd-a299621be317-kube-api-access-k5c6c\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:47.119465 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:47.119430 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d50eb753-37ae-4aff-94b6-893a98133df6" path="/var/lib/kubelet/pods/d50eb753-37ae-4aff-94b6-893a98133df6/volumes" Oct 30 04:11:47.274869 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:47.274831 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:47.275041 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:47.274881 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:47.318030 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:47.317996 4925 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:47.318173 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:47.318047 4925 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:47.375503 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:47.375427 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:47.375503 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:47.375471 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:47.375665 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:47.375549 4925 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:47.375665 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:47.375564 4925 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:47.390507 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:47.390483 4925 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") DevicePath \"csi-1cbd6671aaaf13073dd4ac6088fd398781af87903101ff82770e0956bdbc63f1\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:47.390606 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:47.390484 4925 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") DevicePath \"csi-d631f5804e757beb9cee949b1ba08adddfe0f2a6d74da288dfa6a5946ab2fad2\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:47.493349 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:47.493311 4925 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/ad17c5ee2a7e4881911e4c7c70dd9b4982223d2ab904c958abdf2443e32ecc28/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:47.499939 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:47.499909 4925 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4b631ef058d3f818390cfd4c55b829173556e3e1b26e1e76626d152426c8d48c/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:47.524112 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:47.524088 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:47.525508 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:47.525490 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:47.575990 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:47.575960 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:47.888734 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:47.888711 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 04:11:47.890583 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:11:47.890542 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52925eb5_4cb7_4483_b0fd_a299621be317.slice/crio-135b8305b7e7c954ccae95dd26ae48b58aa925e05079b5c5fed2afefdd36a5b1 WatchSource:0}: Error finding container 135b8305b7e7c954ccae95dd26ae48b58aa925e05079b5c5fed2afefdd36a5b1: Status 404 returned error can't find the container with id 135b8305b7e7c954ccae95dd26ae48b58aa925e05079b5c5fed2afefdd36a5b1 Oct 30 04:11:48.085441 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:48.085405 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"52925eb5-4cb7-4483-b0fd-a299621be317","Type":"ContainerStarted","Data":"1a2718279f43b5d0ed5417ac66e4ae98d88e6c238fa77555bad3495108fcb7bb"} Oct 30 04:11:48.085668 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:48.085458 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"52925eb5-4cb7-4483-b0fd-a299621be317","Type":"ContainerStarted","Data":"135b8305b7e7c954ccae95dd26ae48b58aa925e05079b5c5fed2afefdd36a5b1"} Oct 30 04:11:48.086235 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:48.086208 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:11:48.130158 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:11:48.130114 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podStartSLOduration=3.130101289 podStartE2EDuration="3.130101289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:11:48.129296956 +0000 UTC m=+5469.142575059" watchObservedRunningTime="2026-10-30 04:11:48.130101289 +0000 UTC m=+5469.143379392" Oct 30 04:11:48.161364 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:11:48.161286 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:12:10.100953 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:10.100915 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="52925eb5-4cb7-4483-b0fd-a299621be317" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:12:18.160905 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:12:18.160862 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:12:20.099510 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:20.099473 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="52925eb5-4cb7-4483-b0fd-a299621be317" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:12:30.099014 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:30.098984 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:12:48.161735 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:12:48.161696 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:12:53.655316 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:53.655281 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz"] Oct 30 04:12:53.659910 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:53.659891 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz" Oct 30 04:12:53.690244 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:53.690212 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz"] Oct 30 04:12:53.745193 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:53.745144 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-6a6878d3\"" Oct 30 04:12:53.745193 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:53.745152 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-rxgsr\"" Oct 30 04:12:53.853092 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:53.853059 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b5ee714d-f669-4c48-838f-f3fb3d7f3a55-otc-internal\") pod \"data-science-usage-logs-collector-54c8646c45-lpmzz\" (UID: \"b5ee714d-f669-4c48-838f-f3fb3d7f3a55\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz" Oct 30 04:12:53.853279 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:53.853101 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7wgq\" (UniqueName: \"kubernetes.io/projected/b5ee714d-f669-4c48-838f-f3fb3d7f3a55-kube-api-access-q7wgq\") pod \"data-science-usage-logs-collector-54c8646c45-lpmzz\" (UID: \"b5ee714d-f669-4c48-838f-f3fb3d7f3a55\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz" Oct 30 04:12:53.954538 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:53.954447 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7wgq\" (UniqueName: \"kubernetes.io/projected/b5ee714d-f669-4c48-838f-f3fb3d7f3a55-kube-api-access-q7wgq\") pod \"data-science-usage-logs-collector-54c8646c45-lpmzz\" (UID: \"b5ee714d-f669-4c48-838f-f3fb3d7f3a55\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz" Oct 30 04:12:53.954692 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:53.954546 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b5ee714d-f669-4c48-838f-f3fb3d7f3a55-otc-internal\") pod \"data-science-usage-logs-collector-54c8646c45-lpmzz\" (UID: \"b5ee714d-f669-4c48-838f-f3fb3d7f3a55\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz" Oct 30 04:12:53.955162 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:53.955142 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b5ee714d-f669-4c48-838f-f3fb3d7f3a55-otc-internal\") pod \"data-science-usage-logs-collector-54c8646c45-lpmzz\" (UID: \"b5ee714d-f669-4c48-838f-f3fb3d7f3a55\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz" Oct 30 04:12:53.978638 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:53.978611 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7wgq\" (UniqueName: \"kubernetes.io/projected/b5ee714d-f669-4c48-838f-f3fb3d7f3a55-kube-api-access-q7wgq\") pod \"data-science-usage-logs-collector-54c8646c45-lpmzz\" (UID: \"b5ee714d-f669-4c48-838f-f3fb3d7f3a55\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz" Oct 30 04:12:54.273285 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:54.273202 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz" Oct 30 04:12:54.573383 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:54.573356 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz"] Oct 30 04:12:54.575453 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:12:54.575420 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ee714d_f669_4c48_838f_f3fb3d7f3a55.slice/crio-3022001e33079c79106918b574d501aac1d4f01c24266d50af9e1768a3cafbac WatchSource:0}: Error finding container 3022001e33079c79106918b574d501aac1d4f01c24266d50af9e1768a3cafbac: Status 404 returned error can't find the container with id 3022001e33079c79106918b574d501aac1d4f01c24266d50af9e1768a3cafbac Oct 30 04:12:54.577571 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:54.577543 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:12:54.577667 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:54.577606 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:12:55.404060 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:55.404022 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz" event={"ID":"b5ee714d-f669-4c48-838f-f3fb3d7f3a55","Type":"ContainerStarted","Data":"1818214dccbe9037bc8aa292df3a55291f66eee543a6d0c3cd94256c60379112"} Oct 30 04:12:55.404497 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:55.404102 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz" event={"ID":"b5ee714d-f669-4c48-838f-f3fb3d7f3a55","Type":"ContainerStarted","Data":"3022001e33079c79106918b574d501aac1d4f01c24266d50af9e1768a3cafbac"} Oct 30 04:12:55.447707 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:55.447656 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz" podStartSLOduration=2.447642878 podStartE2EDuration="2.447642878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:12:55.44641523 +0000 UTC m=+5536.459693335" watchObservedRunningTime="2026-10-30 04:12:55.447642878 +0000 UTC m=+5536.460920980" Oct 30 04:12:57.572173 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.572137 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 04:12:57.572642 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.572394 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="52925eb5-4cb7-4483-b0fd-a299621be317" containerName="loki-ingester" containerID="cri-o://1a2718279f43b5d0ed5417ac66e4ae98d88e6c238fa77555bad3495108fcb7bb" gracePeriod=600 Oct 30 04:12:57.591982 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.591945 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 04:12:57.592270 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.592241 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podUID="6829d385-4667-4ef5-9026-5fd116e70ae0" containerName="loki-index-gateway" containerID="cri-o://79370ba5f91fb40431bfce138d78aa1a734edaaca7b08bbffbac3bbdde93fccc" gracePeriod=30 Oct 30 04:12:57.634522 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.634490 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx"] Oct 30 04:12:57.634739 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.634713 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" podUID="27e236f1-2d56-4aa3-be69-d081172c27b1" containerName="opa" containerID="cri-o://762463fe068d096d95db3eab689023d784a6d75359a5a2506df939e8905ebab5" gracePeriod=30 Oct 30 04:12:57.634910 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.634865 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" podUID="27e236f1-2d56-4aa3-be69-d081172c27b1" containerName="gateway" containerID="cri-o://16f3797807ad44cec9d51d77b811fd32684228f1a3a16c5307b785793f0aeb3b" gracePeriod=30 Oct 30 04:12:57.639767 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.639728 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp"] Oct 30 04:12:57.639970 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.639944 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" podUID="3652c6da-ee63-46ca-8d7f-78fd47177fe0" containerName="loki-querier" containerID="cri-o://500ab871406632a74ecde76957e689e81472702e267d0580f3befb12fc134813" gracePeriod=30 Oct 30 04:12:57.658081 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.658050 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m"] Oct 30 04:12:57.658308 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.658276 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" podUID="9d91fb0f-d274-4d79-a481-4608ca9d108c" containerName="loki-query-frontend" containerID="cri-o://1bda0812aeb9ff07c98f3da316750c1b1b7f1be794272bd52ed5b3e5f5d768cd" gracePeriod=30 Oct 30 04:12:57.776597 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.776569 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz"] Oct 30 04:12:57.776810 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.776787 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz" podUID="b5ee714d-f669-4c48-838f-f3fb3d7f3a55" containerName="otc-container" containerID="cri-o://1818214dccbe9037bc8aa292df3a55291f66eee543a6d0c3cd94256c60379112" gracePeriod=30 Oct 30 04:12:57.988223 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.988187 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:12:57.989037 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.989009 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-ca-bundle\") pod \"9d91fb0f-d274-4d79-a481-4608ca9d108c\" (UID: \"9d91fb0f-d274-4d79-a481-4608ca9d108c\") " Oct 30 04:12:57.989123 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.989072 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d91fb0f-d274-4d79-a481-4608ca9d108c-kube-api-access-mtcws\" (UniqueName: \"kubernetes.io/projected/9d91fb0f-d274-4d79-a481-4608ca9d108c-kube-api-access-mtcws\") pod \"9d91fb0f-d274-4d79-a481-4608ca9d108c\" (UID: \"9d91fb0f-d274-4d79-a481-4608ca9d108c\") " Oct 30 04:12:57.989178 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.989118 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-query-frontend-http\") pod \"9d91fb0f-d274-4d79-a481-4608ca9d108c\" (UID: \"9d91fb0f-d274-4d79-a481-4608ca9d108c\") " Oct 30 04:12:57.989178 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.989164 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-query-frontend-grpc\") pod \"9d91fb0f-d274-4d79-a481-4608ca9d108c\" (UID: \"9d91fb0f-d274-4d79-a481-4608ca9d108c\") " Oct 30 04:12:57.989252 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.989198 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d91fb0f-d274-4d79-a481-4608ca9d108c-config\" (UniqueName: \"kubernetes.io/configmap/9d91fb0f-d274-4d79-a481-4608ca9d108c-config\") pod \"9d91fb0f-d274-4d79-a481-4608ca9d108c\" (UID: \"9d91fb0f-d274-4d79-a481-4608ca9d108c\") " Oct 30 04:12:57.990480 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.990444 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-ca-bundle" pod "9d91fb0f-d274-4d79-a481-4608ca9d108c" (UID: "9d91fb0f-d274-4d79-a481-4608ca9d108c"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:12:57.990670 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.990638 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d91fb0f-d274-4d79-a481-4608ca9d108c-config" pod "9d91fb0f-d274-4d79-a481-4608ca9d108c" (UID: "9d91fb0f-d274-4d79-a481-4608ca9d108c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:12:57.992465 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.992426 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d91fb0f-d274-4d79-a481-4608ca9d108c-kube-api-access-mtcws" pod "9d91fb0f-d274-4d79-a481-4608ca9d108c" (UID: "9d91fb0f-d274-4d79-a481-4608ca9d108c"). InnerVolumeSpecName "kube-api-access-mtcws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:12:57.993081 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.993055 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:12:57.993259 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.993234 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-query-frontend-grpc" pod "9d91fb0f-d274-4d79-a481-4608ca9d108c" (UID: "9d91fb0f-d274-4d79-a481-4608ca9d108c"). InnerVolumeSpecName "data-science-lokistack-query-frontend-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:57.993701 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:57.993680 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-query-frontend-http" pod "9d91fb0f-d274-4d79-a481-4608ca9d108c" (UID: "9d91fb0f-d274-4d79-a481-4608ca9d108c"). InnerVolumeSpecName "data-science-lokistack-query-frontend-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:58.041010 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.040980 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz" Oct 30 04:12:58.089880 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.089804 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"6829d385-4667-4ef5-9026-5fd116e70ae0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " Oct 30 04:12:58.089880 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.089872 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-ca-bundle\") pod \"6829d385-4667-4ef5-9026-5fd116e70ae0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " Oct 30 04:12:58.090102 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.089899 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6829d385-4667-4ef5-9026-5fd116e70ae0-config\" (UniqueName: \"kubernetes.io/configmap/6829d385-4667-4ef5-9026-5fd116e70ae0-config\") pod \"6829d385-4667-4ef5-9026-5fd116e70ae0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " Oct 30 04:12:58.090102 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.089921 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-index-gateway-http\") pod \"6829d385-4667-4ef5-9026-5fd116e70ae0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " Oct 30 04:12:58.090102 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.089970 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5ee714d-f669-4c48-838f-f3fb3d7f3a55-kube-api-access-q7wgq\" (UniqueName: \"kubernetes.io/projected/b5ee714d-f669-4c48-838f-f3fb3d7f3a55-kube-api-access-q7wgq\") pod \"b5ee714d-f669-4c48-838f-f3fb3d7f3a55\" (UID: \"b5ee714d-f669-4c48-838f-f3fb3d7f3a55\") " Oct 30 04:12:58.090102 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.090000 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-test-loki-lifecycle-secret\") pod \"6829d385-4667-4ef5-9026-5fd116e70ae0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " Oct 30 04:12:58.090102 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.090051 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-index-gateway-grpc\") pod \"6829d385-4667-4ef5-9026-5fd116e70ae0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " Oct 30 04:12:58.090102 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.090102 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6829d385-4667-4ef5-9026-5fd116e70ae0-kube-api-access-qfs8j\" (UniqueName: \"kubernetes.io/projected/6829d385-4667-4ef5-9026-5fd116e70ae0-kube-api-access-qfs8j\") pod \"6829d385-4667-4ef5-9026-5fd116e70ae0\" (UID: \"6829d385-4667-4ef5-9026-5fd116e70ae0\") " Oct 30 04:12:58.090408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.090146 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5ee714d-f669-4c48-838f-f3fb3d7f3a55-otc-internal\" (UniqueName: \"kubernetes.io/configmap/b5ee714d-f669-4c48-838f-f3fb3d7f3a55-otc-internal\") pod \"b5ee714d-f669-4c48-838f-f3fb3d7f3a55\" (UID: \"b5ee714d-f669-4c48-838f-f3fb3d7f3a55\") " Oct 30 04:12:58.090506 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.090483 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-ca-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.090568 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.090510 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtcws\" (UniqueName: \"kubernetes.io/projected/9d91fb0f-d274-4d79-a481-4608ca9d108c-kube-api-access-mtcws\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.090568 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.090525 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-query-frontend-http\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.090568 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.090538 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/9d91fb0f-d274-4d79-a481-4608ca9d108c-data-science-lokistack-query-frontend-grpc\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.090568 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.090552 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d91fb0f-d274-4d79-a481-4608ca9d108c-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.090972 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.090944 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6829d385-4667-4ef5-9026-5fd116e70ae0-config" pod "6829d385-4667-4ef5-9026-5fd116e70ae0" (UID: "6829d385-4667-4ef5-9026-5fd116e70ae0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:12:58.091139 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.091033 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5ee714d-f669-4c48-838f-f3fb3d7f3a55-otc-internal" pod "b5ee714d-f669-4c48-838f-f3fb3d7f3a55" (UID: "b5ee714d-f669-4c48-838f-f3fb3d7f3a55"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:12:58.091213 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.091136 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-ca-bundle" pod "6829d385-4667-4ef5-9026-5fd116e70ae0" (UID: "6829d385-4667-4ef5-9026-5fd116e70ae0"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:12:58.093413 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.093378 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ee714d-f669-4c48-838f-f3fb3d7f3a55-kube-api-access-q7wgq" pod "b5ee714d-f669-4c48-838f-f3fb3d7f3a55" (UID: "b5ee714d-f669-4c48-838f-f3fb3d7f3a55"). InnerVolumeSpecName "kube-api-access-q7wgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:12:58.093413 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.093392 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6829d385-4667-4ef5-9026-5fd116e70ae0-kube-api-access-qfs8j" pod "6829d385-4667-4ef5-9026-5fd116e70ae0" (UID: "6829d385-4667-4ef5-9026-5fd116e70ae0"). InnerVolumeSpecName "kube-api-access-qfs8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:12:58.093571 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.093440 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-index-gateway-grpc" pod "6829d385-4667-4ef5-9026-5fd116e70ae0" (UID: "6829d385-4667-4ef5-9026-5fd116e70ae0"). InnerVolumeSpecName "data-science-lokistack-index-gateway-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:58.093571 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.093469 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-index-gateway-http" pod "6829d385-4667-4ef5-9026-5fd116e70ae0" (UID: "6829d385-4667-4ef5-9026-5fd116e70ae0"). InnerVolumeSpecName "data-science-lokistack-index-gateway-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:58.093571 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.093475 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a" pod "6829d385-4667-4ef5-9026-5fd116e70ae0" (UID: "6829d385-4667-4ef5-9026-5fd116e70ae0"). InnerVolumeSpecName "pvc-536701e3-6ae5-4168-ad78-e30a293330df". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:12:58.094621 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.094602 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-test-loki-lifecycle-secret" pod "6829d385-4667-4ef5-9026-5fd116e70ae0" (UID: "6829d385-4667-4ef5-9026-5fd116e70ae0"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:58.137568 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.137542 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:12:58.191159 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.191129 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-test-loki-lifecycle-secret\") pod \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " Oct 30 04:12:58.191327 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.191170 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3652c6da-ee63-46ca-8d7f-78fd47177fe0-config\" (UniqueName: \"kubernetes.io/configmap/3652c6da-ee63-46ca-8d7f-78fd47177fe0-config\") pod \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " Oct 30 04:12:58.191327 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.191214 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-querier-grpc\") pod \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " Oct 30 04:12:58.191327 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.191313 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-ca-bundle\") pod \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " Oct 30 04:12:58.191466 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.191368 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3652c6da-ee63-46ca-8d7f-78fd47177fe0-kube-api-access-kgpdl\" (UniqueName: \"kubernetes.io/projected/3652c6da-ee63-46ca-8d7f-78fd47177fe0-kube-api-access-kgpdl\") pod \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " Oct 30 04:12:58.191466 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.191412 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-querier-http\") pod \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\" (UID: \"3652c6da-ee63-46ca-8d7f-78fd47177fe0\") " Oct 30 04:12:58.191690 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.191639 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-ca-bundle" pod "3652c6da-ee63-46ca-8d7f-78fd47177fe0" (UID: "3652c6da-ee63-46ca-8d7f-78fd47177fe0"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:12:58.191690 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.191648 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3652c6da-ee63-46ca-8d7f-78fd47177fe0-config" pod "3652c6da-ee63-46ca-8d7f-78fd47177fe0" (UID: "3652c6da-ee63-46ca-8d7f-78fd47177fe0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:12:58.191878 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.191786 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b5ee714d-f669-4c48-838f-f3fb3d7f3a55-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.191878 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.191818 4925 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") on node \"ip-10-0-139-19.ec2.internal\" " Oct 30 04:12:58.191878 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.191834 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-ca-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.191878 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.191852 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6829d385-4667-4ef5-9026-5fd116e70ae0-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.191878 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.191867 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-index-gateway-http\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.192138 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.191884 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7wgq\" (UniqueName: \"kubernetes.io/projected/b5ee714d-f669-4c48-838f-f3fb3d7f3a55-kube-api-access-q7wgq\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.192138 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.191898 4925 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-test-loki-lifecycle-secret\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.192138 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.191914 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/6829d385-4667-4ef5-9026-5fd116e70ae0-data-science-lokistack-index-gateway-grpc\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.192138 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.191930 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfs8j\" (UniqueName: \"kubernetes.io/projected/6829d385-4667-4ef5-9026-5fd116e70ae0-kube-api-access-qfs8j\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.193577 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.193554 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3652c6da-ee63-46ca-8d7f-78fd47177fe0-kube-api-access-kgpdl" pod "3652c6da-ee63-46ca-8d7f-78fd47177fe0" (UID: "3652c6da-ee63-46ca-8d7f-78fd47177fe0"). InnerVolumeSpecName "kube-api-access-kgpdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:12:58.194095 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.194073 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-querier-grpc" pod "3652c6da-ee63-46ca-8d7f-78fd47177fe0" (UID: "3652c6da-ee63-46ca-8d7f-78fd47177fe0"). InnerVolumeSpecName "data-science-lokistack-querier-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:58.194167 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.194127 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-test-loki-lifecycle-secret" pod "3652c6da-ee63-46ca-8d7f-78fd47177fe0" (UID: "3652c6da-ee63-46ca-8d7f-78fd47177fe0"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:58.194565 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.194551 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-querier-http" pod "3652c6da-ee63-46ca-8d7f-78fd47177fe0" (UID: "3652c6da-ee63-46ca-8d7f-78fd47177fe0"). InnerVolumeSpecName "data-science-lokistack-querier-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:58.202208 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.202191 4925 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-536701e3-6ae5-4168-ad78-e30a293330df" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a") on node "ip-10-0-139-19.ec2.internal" Oct 30 04:12:58.292962 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.292928 4925 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-test-loki-lifecycle-secret\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.292962 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.292955 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3652c6da-ee63-46ca-8d7f-78fd47177fe0-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.292962 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.292968 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-querier-grpc\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.293201 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.292977 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-ca-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.293201 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.292987 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgpdl\" (UniqueName: \"kubernetes.io/projected/3652c6da-ee63-46ca-8d7f-78fd47177fe0-kube-api-access-kgpdl\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.293201 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.292996 4925 reconciler_common.go:299] "Volume detached for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"csi-09f7dd9c6d02fc70105c8ae46375645a3d310e7e56fb2e91114833c2c076d551\"" Oct 30 04:12:58.293201 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.293006 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/3652c6da-ee63-46ca-8d7f-78fd47177fe0-data-science-lokistack-querier-http\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.419537 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.419443 4925 generic.go:356] "Generic (PLEG): container finished" podID="6829d385-4667-4ef5-9026-5fd116e70ae0" containerID="79370ba5f91fb40431bfce138d78aa1a734edaaca7b08bbffbac3bbdde93fccc" exitCode=0 Oct 30 04:12:58.419537 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.419489 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 04:12:58.419537 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.419526 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"6829d385-4667-4ef5-9026-5fd116e70ae0","Type":"ContainerDied","Data":"79370ba5f91fb40431bfce138d78aa1a734edaaca7b08bbffbac3bbdde93fccc"} Oct 30 04:12:58.419859 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.419575 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"6829d385-4667-4ef5-9026-5fd116e70ae0","Type":"ContainerDied","Data":"74029d541660e99ca52c8a6a01eec7cc0ecca692b7b1bb09bd8f5fdfe2d1098c"} Oct 30 04:12:58.419859 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.419600 4925 scope.go:122] "RemoveContainer" containerID="79370ba5f91fb40431bfce138d78aa1a734edaaca7b08bbffbac3bbdde93fccc" Oct 30 04:12:58.421050 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.421026 4925 generic.go:356] "Generic (PLEG): container finished" podID="9d91fb0f-d274-4d79-a481-4608ca9d108c" containerID="1bda0812aeb9ff07c98f3da316750c1b1b7f1be794272bd52ed5b3e5f5d768cd" exitCode=0 Oct 30 04:12:58.421185 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.421076 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" Oct 30 04:12:58.421185 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.421135 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" event={"ID":"9d91fb0f-d274-4d79-a481-4608ca9d108c","Type":"ContainerDied","Data":"1bda0812aeb9ff07c98f3da316750c1b1b7f1be794272bd52ed5b3e5f5d768cd"} Oct 30 04:12:58.421185 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.421172 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m" event={"ID":"9d91fb0f-d274-4d79-a481-4608ca9d108c","Type":"ContainerDied","Data":"622b4b63c0469aab78a1551706069d39cc6acb098c947d3633e701cfd7d976af"} Oct 30 04:12:58.423581 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.423555 4925 generic.go:356] "Generic (PLEG): container finished" podID="27e236f1-2d56-4aa3-be69-d081172c27b1" containerID="762463fe068d096d95db3eab689023d784a6d75359a5a2506df939e8905ebab5" exitCode=0 Oct 30 04:12:58.423699 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.423618 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" event={"ID":"27e236f1-2d56-4aa3-be69-d081172c27b1","Type":"ContainerDied","Data":"762463fe068d096d95db3eab689023d784a6d75359a5a2506df939e8905ebab5"} Oct 30 04:12:58.425122 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.425103 4925 generic.go:356] "Generic (PLEG): container finished" podID="3652c6da-ee63-46ca-8d7f-78fd47177fe0" containerID="500ab871406632a74ecde76957e689e81472702e267d0580f3befb12fc134813" exitCode=0 Oct 30 04:12:58.425244 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.425155 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" Oct 30 04:12:58.425244 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.425127 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" event={"ID":"3652c6da-ee63-46ca-8d7f-78fd47177fe0","Type":"ContainerDied","Data":"500ab871406632a74ecde76957e689e81472702e267d0580f3befb12fc134813"} Oct 30 04:12:58.425244 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.425204 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp" event={"ID":"3652c6da-ee63-46ca-8d7f-78fd47177fe0","Type":"ContainerDied","Data":"b563a8f0cd02fa83f3a952b7af29c11586575184bf4bbb8033f446b217a6695a"} Oct 30 04:12:58.426529 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.426508 4925 generic.go:356] "Generic (PLEG): container finished" podID="b5ee714d-f669-4c48-838f-f3fb3d7f3a55" containerID="1818214dccbe9037bc8aa292df3a55291f66eee543a6d0c3cd94256c60379112" exitCode=0 Oct 30 04:12:58.426616 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.426554 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz" event={"ID":"b5ee714d-f669-4c48-838f-f3fb3d7f3a55","Type":"ContainerDied","Data":"1818214dccbe9037bc8aa292df3a55291f66eee543a6d0c3cd94256c60379112"} Oct 30 04:12:58.426616 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.426589 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz" event={"ID":"b5ee714d-f669-4c48-838f-f3fb3d7f3a55","Type":"ContainerDied","Data":"3022001e33079c79106918b574d501aac1d4f01c24266d50af9e1768a3cafbac"} Oct 30 04:12:58.426742 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.426647 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz" Oct 30 04:12:58.431013 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.430994 4925 scope.go:122] "RemoveContainer" containerID="79370ba5f91fb40431bfce138d78aa1a734edaaca7b08bbffbac3bbdde93fccc" Oct 30 04:12:58.431338 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:12:58.431301 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79370ba5f91fb40431bfce138d78aa1a734edaaca7b08bbffbac3bbdde93fccc\": container with ID starting with 79370ba5f91fb40431bfce138d78aa1a734edaaca7b08bbffbac3bbdde93fccc not found: ID does not exist" containerID="79370ba5f91fb40431bfce138d78aa1a734edaaca7b08bbffbac3bbdde93fccc" Oct 30 04:12:58.431399 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.431335 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79370ba5f91fb40431bfce138d78aa1a734edaaca7b08bbffbac3bbdde93fccc"} err="failed to get container status \"79370ba5f91fb40431bfce138d78aa1a734edaaca7b08bbffbac3bbdde93fccc\": rpc error: code = NotFound desc = could not find container \"79370ba5f91fb40431bfce138d78aa1a734edaaca7b08bbffbac3bbdde93fccc\": container with ID starting with 79370ba5f91fb40431bfce138d78aa1a734edaaca7b08bbffbac3bbdde93fccc not found: ID does not exist" Oct 30 04:12:58.431399 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.431357 4925 scope.go:122] "RemoveContainer" containerID="1bda0812aeb9ff07c98f3da316750c1b1b7f1be794272bd52ed5b3e5f5d768cd" Oct 30 04:12:58.440642 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.440621 4925 scope.go:122] "RemoveContainer" containerID="1bda0812aeb9ff07c98f3da316750c1b1b7f1be794272bd52ed5b3e5f5d768cd" Oct 30 04:12:58.440929 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:12:58.440911 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bda0812aeb9ff07c98f3da316750c1b1b7f1be794272bd52ed5b3e5f5d768cd\": container with ID starting with 1bda0812aeb9ff07c98f3da316750c1b1b7f1be794272bd52ed5b3e5f5d768cd not found: ID does not exist" containerID="1bda0812aeb9ff07c98f3da316750c1b1b7f1be794272bd52ed5b3e5f5d768cd" Oct 30 04:12:58.441004 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.440934 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bda0812aeb9ff07c98f3da316750c1b1b7f1be794272bd52ed5b3e5f5d768cd"} err="failed to get container status \"1bda0812aeb9ff07c98f3da316750c1b1b7f1be794272bd52ed5b3e5f5d768cd\": rpc error: code = NotFound desc = could not find container \"1bda0812aeb9ff07c98f3da316750c1b1b7f1be794272bd52ed5b3e5f5d768cd\": container with ID starting with 1bda0812aeb9ff07c98f3da316750c1b1b7f1be794272bd52ed5b3e5f5d768cd not found: ID does not exist" Oct 30 04:12:58.441004 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.440949 4925 scope.go:122] "RemoveContainer" containerID="500ab871406632a74ecde76957e689e81472702e267d0580f3befb12fc134813" Oct 30 04:12:58.448829 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.448811 4925 scope.go:122] "RemoveContainer" containerID="500ab871406632a74ecde76957e689e81472702e267d0580f3befb12fc134813" Oct 30 04:12:58.449070 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:12:58.449052 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"500ab871406632a74ecde76957e689e81472702e267d0580f3befb12fc134813\": container with ID starting with 500ab871406632a74ecde76957e689e81472702e267d0580f3befb12fc134813 not found: ID does not exist" containerID="500ab871406632a74ecde76957e689e81472702e267d0580f3befb12fc134813" Oct 30 04:12:58.449117 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.449077 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"500ab871406632a74ecde76957e689e81472702e267d0580f3befb12fc134813"} err="failed to get container status \"500ab871406632a74ecde76957e689e81472702e267d0580f3befb12fc134813\": rpc error: code = NotFound desc = could not find container \"500ab871406632a74ecde76957e689e81472702e267d0580f3befb12fc134813\": container with ID starting with 500ab871406632a74ecde76957e689e81472702e267d0580f3befb12fc134813 not found: ID does not exist" Oct 30 04:12:58.449117 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.449093 4925 scope.go:122] "RemoveContainer" containerID="1818214dccbe9037bc8aa292df3a55291f66eee543a6d0c3cd94256c60379112" Oct 30 04:12:58.456652 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.456632 4925 scope.go:122] "RemoveContainer" containerID="1818214dccbe9037bc8aa292df3a55291f66eee543a6d0c3cd94256c60379112" Oct 30 04:12:58.456954 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:12:58.456934 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1818214dccbe9037bc8aa292df3a55291f66eee543a6d0c3cd94256c60379112\": container with ID starting with 1818214dccbe9037bc8aa292df3a55291f66eee543a6d0c3cd94256c60379112 not found: ID does not exist" containerID="1818214dccbe9037bc8aa292df3a55291f66eee543a6d0c3cd94256c60379112" Oct 30 04:12:58.457026 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.456964 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1818214dccbe9037bc8aa292df3a55291f66eee543a6d0c3cd94256c60379112"} err="failed to get container status \"1818214dccbe9037bc8aa292df3a55291f66eee543a6d0c3cd94256c60379112\": rpc error: code = NotFound desc = could not find container \"1818214dccbe9037bc8aa292df3a55291f66eee543a6d0c3cd94256c60379112\": container with ID starting with 1818214dccbe9037bc8aa292df3a55291f66eee543a6d0c3cd94256c60379112 not found: ID does not exist" Oct 30 04:12:58.504921 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.504890 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 04:12:58.507377 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.507352 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 04:12:58.559993 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.559449 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz"] Oct 30 04:12:58.564027 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.563996 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-lpmzz"] Oct 30 04:12:58.601488 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.601453 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m"] Oct 30 04:12:58.608776 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.608731 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-7wg8m"] Oct 30 04:12:58.642007 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.641980 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp"] Oct 30 04:12:58.645908 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.645878 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-nkhbp"] Oct 30 04:12:58.882526 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.882503 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:12:58.896648 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.896625 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-ca-bundle\") pod \"27e236f1-2d56-4aa3-be69-d081172c27b1\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " Oct 30 04:12:58.896817 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.896662 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-tls-secret\" (UniqueName: \"kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-tls-secret\") pod \"27e236f1-2d56-4aa3-be69-d081172c27b1\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " Oct 30 04:12:58.896817 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.896687 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27e236f1-2d56-4aa3-be69-d081172c27b1-tenants\" (UniqueName: \"kubernetes.io/secret/27e236f1-2d56-4aa3-be69-d081172c27b1-tenants\") pod \"27e236f1-2d56-4aa3-be69-d081172c27b1\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " Oct 30 04:12:58.896817 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.896706 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-lokistack-gateway\") pod \"27e236f1-2d56-4aa3-be69-d081172c27b1\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " Oct 30 04:12:58.896817 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.896746 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-gateway-ca-bundle\") pod \"27e236f1-2d56-4aa3-be69-d081172c27b1\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " Oct 30 04:12:58.896817 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.896787 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-rbac\" (UniqueName: \"kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-rbac\") pod \"27e236f1-2d56-4aa3-be69-d081172c27b1\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " Oct 30 04:12:58.897127 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.896858 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-kube-api-access-9x2mm\" (UniqueName: \"kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-kube-api-access-9x2mm\") pod \"27e236f1-2d56-4aa3-be69-d081172c27b1\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " Oct 30 04:12:58.897127 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.896902 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-gateway-client-http\") pod \"27e236f1-2d56-4aa3-be69-d081172c27b1\" (UID: \"27e236f1-2d56-4aa3-be69-d081172c27b1\") " Oct 30 04:12:58.897127 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.897025 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-ca-bundle" pod "27e236f1-2d56-4aa3-be69-d081172c27b1" (UID: "27e236f1-2d56-4aa3-be69-d081172c27b1"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:12:58.897293 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.897169 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-ca-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.899266 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.899240 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-tls-secret" pod "27e236f1-2d56-4aa3-be69-d081172c27b1" (UID: "27e236f1-2d56-4aa3-be69-d081172c27b1"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:12:58.899496 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.899469 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-gateway-ca-bundle" pod "27e236f1-2d56-4aa3-be69-d081172c27b1" (UID: "27e236f1-2d56-4aa3-be69-d081172c27b1"). InnerVolumeSpecName "data-science-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:12:58.899705 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.899673 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-gateway-client-http" pod "27e236f1-2d56-4aa3-be69-d081172c27b1" (UID: "27e236f1-2d56-4aa3-be69-d081172c27b1"). InnerVolumeSpecName "data-science-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:58.899832 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.899785 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-kube-api-access-9x2mm" pod "27e236f1-2d56-4aa3-be69-d081172c27b1" (UID: "27e236f1-2d56-4aa3-be69-d081172c27b1"). InnerVolumeSpecName "kube-api-access-9x2mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:12:58.902121 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.902099 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-lokistack-gateway" pod "27e236f1-2d56-4aa3-be69-d081172c27b1" (UID: "27e236f1-2d56-4aa3-be69-d081172c27b1"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:12:58.902226 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.902194 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-rbac" pod "27e236f1-2d56-4aa3-be69-d081172c27b1" (UID: "27e236f1-2d56-4aa3-be69-d081172c27b1"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:12:58.903498 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.903479 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27e236f1-2d56-4aa3-be69-d081172c27b1-tenants" pod "27e236f1-2d56-4aa3-be69-d081172c27b1" (UID: "27e236f1-2d56-4aa3-be69-d081172c27b1"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:58.998400 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.998311 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x2mm\" (UniqueName: \"kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-kube-api-access-9x2mm\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.998400 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.998345 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-gateway-client-http\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.998400 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.998360 4925 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-tls-secret\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.998400 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.998373 4925 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/27e236f1-2d56-4aa3-be69-d081172c27b1-tenants\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.998400 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.998385 4925 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-lokistack-gateway\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.998400 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.998398 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/27e236f1-2d56-4aa3-be69-d081172c27b1-data-science-lokistack-gateway-ca-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:58.998694 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:58.998410 4925 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/27e236f1-2d56-4aa3-be69-d081172c27b1-rbac\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:12:59.119489 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:59.119455 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3652c6da-ee63-46ca-8d7f-78fd47177fe0" path="/var/lib/kubelet/pods/3652c6da-ee63-46ca-8d7f-78fd47177fe0/volumes" Oct 30 04:12:59.119896 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:59.119883 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6829d385-4667-4ef5-9026-5fd116e70ae0" path="/var/lib/kubelet/pods/6829d385-4667-4ef5-9026-5fd116e70ae0/volumes" Oct 30 04:12:59.120232 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:59.120221 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d91fb0f-d274-4d79-a481-4608ca9d108c" path="/var/lib/kubelet/pods/9d91fb0f-d274-4d79-a481-4608ca9d108c/volumes" Oct 30 04:12:59.120855 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:59.120841 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5ee714d-f669-4c48-838f-f3fb3d7f3a55" path="/var/lib/kubelet/pods/b5ee714d-f669-4c48-838f-f3fb3d7f3a55/volumes" Oct 30 04:12:59.434992 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:59.434955 4925 generic.go:356] "Generic (PLEG): container finished" podID="27e236f1-2d56-4aa3-be69-d081172c27b1" containerID="16f3797807ad44cec9d51d77b811fd32684228f1a3a16c5307b785793f0aeb3b" exitCode=0 Oct 30 04:12:59.435199 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:59.435007 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" Oct 30 04:12:59.435199 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:59.435028 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" event={"ID":"27e236f1-2d56-4aa3-be69-d081172c27b1","Type":"ContainerDied","Data":"16f3797807ad44cec9d51d77b811fd32684228f1a3a16c5307b785793f0aeb3b"} Oct 30 04:12:59.435199 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:59.435077 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx" event={"ID":"27e236f1-2d56-4aa3-be69-d081172c27b1","Type":"ContainerDied","Data":"674e36df7c2791e2a92000cc2bfe1422952f937edc873d96ed47cab220bb6c93"} Oct 30 04:12:59.435199 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:59.435109 4925 scope.go:122] "RemoveContainer" containerID="762463fe068d096d95db3eab689023d784a6d75359a5a2506df939e8905ebab5" Oct 30 04:12:59.443411 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:59.443384 4925 scope.go:122] "RemoveContainer" containerID="16f3797807ad44cec9d51d77b811fd32684228f1a3a16c5307b785793f0aeb3b" Oct 30 04:12:59.451124 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:59.451109 4925 scope.go:122] "RemoveContainer" containerID="762463fe068d096d95db3eab689023d784a6d75359a5a2506df939e8905ebab5" Oct 30 04:12:59.451334 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:12:59.451316 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"762463fe068d096d95db3eab689023d784a6d75359a5a2506df939e8905ebab5\": container with ID starting with 762463fe068d096d95db3eab689023d784a6d75359a5a2506df939e8905ebab5 not found: ID does not exist" containerID="762463fe068d096d95db3eab689023d784a6d75359a5a2506df939e8905ebab5" Oct 30 04:12:59.451390 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:59.451340 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762463fe068d096d95db3eab689023d784a6d75359a5a2506df939e8905ebab5"} err="failed to get container status \"762463fe068d096d95db3eab689023d784a6d75359a5a2506df939e8905ebab5\": rpc error: code = NotFound desc = could not find container \"762463fe068d096d95db3eab689023d784a6d75359a5a2506df939e8905ebab5\": container with ID starting with 762463fe068d096d95db3eab689023d784a6d75359a5a2506df939e8905ebab5 not found: ID does not exist" Oct 30 04:12:59.451390 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:59.451359 4925 scope.go:122] "RemoveContainer" containerID="16f3797807ad44cec9d51d77b811fd32684228f1a3a16c5307b785793f0aeb3b" Oct 30 04:12:59.451559 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:12:59.451543 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f3797807ad44cec9d51d77b811fd32684228f1a3a16c5307b785793f0aeb3b\": container with ID starting with 16f3797807ad44cec9d51d77b811fd32684228f1a3a16c5307b785793f0aeb3b not found: ID does not exist" containerID="16f3797807ad44cec9d51d77b811fd32684228f1a3a16c5307b785793f0aeb3b" Oct 30 04:12:59.451595 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:59.451563 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f3797807ad44cec9d51d77b811fd32684228f1a3a16c5307b785793f0aeb3b"} err="failed to get container status \"16f3797807ad44cec9d51d77b811fd32684228f1a3a16c5307b785793f0aeb3b\": rpc error: code = NotFound desc = could not find container \"16f3797807ad44cec9d51d77b811fd32684228f1a3a16c5307b785793f0aeb3b\": container with ID starting with 16f3797807ad44cec9d51d77b811fd32684228f1a3a16c5307b785793f0aeb3b not found: ID does not exist" Oct 30 04:12:59.488109 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:59.488080 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx"] Oct 30 04:12:59.494446 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:12:59.494424 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-m4qpx"] Oct 30 04:13:00.099792 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:00.099734 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="52925eb5-4cb7-4483-b0fd-a299621be317" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:13:01.119350 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:01.119315 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27e236f1-2d56-4aa3-be69-d081172c27b1" path="/var/lib/kubelet/pods/27e236f1-2d56-4aa3-be69-d081172c27b1/volumes" Oct 30 04:13:03.507918 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.507883 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 04:13:03.508357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.508195 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6829d385-4667-4ef5-9026-5fd116e70ae0" containerName="loki-index-gateway" Oct 30 04:13:03.508357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.508206 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6829d385-4667-4ef5-9026-5fd116e70ae0" containerName="loki-index-gateway" Oct 30 04:13:03.508357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.508214 4925 cpu_manager.go:414] "containerMap: removing container" podUID="b5ee714d-f669-4c48-838f-f3fb3d7f3a55" containerName="otc-container" Oct 30 04:13:03.508357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.508219 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5ee714d-f669-4c48-838f-f3fb3d7f3a55" containerName="otc-container" Oct 30 04:13:03.508357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.508228 4925 cpu_manager.go:414] "containerMap: removing container" podUID="9d91fb0f-d274-4d79-a481-4608ca9d108c" containerName="loki-query-frontend" Oct 30 04:13:03.508357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.508234 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d91fb0f-d274-4d79-a481-4608ca9d108c" containerName="loki-query-frontend" Oct 30 04:13:03.508357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.508245 4925 cpu_manager.go:414] "containerMap: removing container" podUID="3652c6da-ee63-46ca-8d7f-78fd47177fe0" containerName="loki-querier" Oct 30 04:13:03.508357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.508250 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3652c6da-ee63-46ca-8d7f-78fd47177fe0" containerName="loki-querier" Oct 30 04:13:03.508357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.508260 4925 cpu_manager.go:414] "containerMap: removing container" podUID="27e236f1-2d56-4aa3-be69-d081172c27b1" containerName="gateway" Oct 30 04:13:03.508357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.508264 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e236f1-2d56-4aa3-be69-d081172c27b1" containerName="gateway" Oct 30 04:13:03.508357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.508271 4925 cpu_manager.go:414] "containerMap: removing container" podUID="27e236f1-2d56-4aa3-be69-d081172c27b1" containerName="opa" Oct 30 04:13:03.508357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.508276 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e236f1-2d56-4aa3-be69-d081172c27b1" containerName="opa" Oct 30 04:13:03.508357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.508329 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6829d385-4667-4ef5-9026-5fd116e70ae0" containerName="loki-index-gateway" Oct 30 04:13:03.508357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.508335 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5ee714d-f669-4c48-838f-f3fb3d7f3a55" containerName="otc-container" Oct 30 04:13:03.508357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.508340 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d91fb0f-d274-4d79-a481-4608ca9d108c" containerName="loki-query-frontend" Oct 30 04:13:03.508357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.508347 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="3652c6da-ee63-46ca-8d7f-78fd47177fe0" containerName="loki-querier" Oct 30 04:13:03.508357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.508353 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="27e236f1-2d56-4aa3-be69-d081172c27b1" containerName="gateway" Oct 30 04:13:03.508357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.508358 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="27e236f1-2d56-4aa3-be69-d081172c27b1" containerName="opa" Oct 30 04:13:03.520556 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.520530 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:03.537552 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.537528 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 04:13:03.557941 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.557917 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-dockercfg-vft9z\"" Oct 30 04:13:03.558122 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.558106 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-compactor-http\"" Oct 30 04:13:03.558173 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.558109 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-compactor-grpc\"" Oct 30 04:13:03.583740 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.583715 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7"] Oct 30 04:13:03.587489 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.587467 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:13:03.614775 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.614730 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7"] Oct 30 04:13:03.640232 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.640206 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-test-loki-lifecycle-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:03.640359 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.640237 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8q55\" (UniqueName: \"kubernetes.io/projected/11b21fd1-a5a2-473d-94fa-cc22f744c736-kube-api-access-q8q55\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:03.640359 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.640266 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:03.640359 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.640334 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:03.640536 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.640474 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:03.640536 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.640526 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11b21fd1-a5a2-473d-94fa-cc22f744c736-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:03.654426 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.654385 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-query-frontend-http\"" Oct 30 04:13:03.654426 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.654411 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-query-frontend-grpc\"" Oct 30 04:13:03.741232 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.741196 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:03.741454 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.741248 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-f746fc66d-mrjz7\" (UID: \"6f6daa03-27bc-4159-b250-d845ee3e9625\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:13:03.741454 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.741285 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2hx4\" (UniqueName: \"kubernetes.io/projected/6f6daa03-27bc-4159-b250-d845ee3e9625-kube-api-access-g2hx4\") pod \"data-science-lokistack-query-frontend-f746fc66d-mrjz7\" (UID: \"6f6daa03-27bc-4159-b250-d845ee3e9625\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:13:03.741454 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.741326 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11b21fd1-a5a2-473d-94fa-cc22f744c736-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:03.741676 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.741550 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-f746fc66d-mrjz7\" (UID: \"6f6daa03-27bc-4159-b250-d845ee3e9625\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:13:03.741676 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.741614 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-f746fc66d-mrjz7\" (UID: \"6f6daa03-27bc-4159-b250-d845ee3e9625\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:13:03.741794 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.741713 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-test-loki-lifecycle-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:03.741858 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.741796 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8q55\" (UniqueName: \"kubernetes.io/projected/11b21fd1-a5a2-473d-94fa-cc22f744c736-kube-api-access-q8q55\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:03.741858 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.741847 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:03.741969 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.741872 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:03.741969 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.741901 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f6daa03-27bc-4159-b250-d845ee3e9625-config\") pod \"data-science-lokistack-query-frontend-f746fc66d-mrjz7\" (UID: \"6f6daa03-27bc-4159-b250-d845ee3e9625\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:13:03.742119 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.742082 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:03.742262 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.742242 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11b21fd1-a5a2-473d-94fa-cc22f744c736-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:03.744226 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.744204 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:03.744298 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.744245 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:03.744446 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.744430 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-test-loki-lifecycle-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:03.787785 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.787700 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8q55\" (UniqueName: \"kubernetes.io/projected/11b21fd1-a5a2-473d-94fa-cc22f744c736-kube-api-access-q8q55\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:03.842721 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.842692 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f6daa03-27bc-4159-b250-d845ee3e9625-config\") pod \"data-science-lokistack-query-frontend-f746fc66d-mrjz7\" (UID: \"6f6daa03-27bc-4159-b250-d845ee3e9625\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:13:03.842895 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.842726 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-f746fc66d-mrjz7\" (UID: \"6f6daa03-27bc-4159-b250-d845ee3e9625\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:13:03.842895 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.842769 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2hx4\" (UniqueName: \"kubernetes.io/projected/6f6daa03-27bc-4159-b250-d845ee3e9625-kube-api-access-g2hx4\") pod \"data-science-lokistack-query-frontend-f746fc66d-mrjz7\" (UID: \"6f6daa03-27bc-4159-b250-d845ee3e9625\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:13:03.842895 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.842816 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-f746fc66d-mrjz7\" (UID: \"6f6daa03-27bc-4159-b250-d845ee3e9625\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:13:03.842895 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.842842 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-f746fc66d-mrjz7\" (UID: \"6f6daa03-27bc-4159-b250-d845ee3e9625\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:13:03.843437 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.843416 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-f746fc66d-mrjz7\" (UID: \"6f6daa03-27bc-4159-b250-d845ee3e9625\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:13:03.843546 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.843418 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f6daa03-27bc-4159-b250-d845ee3e9625-config\") pod \"data-science-lokistack-query-frontend-f746fc66d-mrjz7\" (UID: \"6f6daa03-27bc-4159-b250-d845ee3e9625\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:13:03.845061 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.845034 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-f746fc66d-mrjz7\" (UID: \"6f6daa03-27bc-4159-b250-d845ee3e9625\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:13:03.845201 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.845091 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-f746fc66d-mrjz7\" (UID: \"6f6daa03-27bc-4159-b250-d845ee3e9625\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:13:03.868431 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.868411 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2hx4\" (UniqueName: \"kubernetes.io/projected/6f6daa03-27bc-4159-b250-d845ee3e9625-kube-api-access-g2hx4\") pod \"data-science-lokistack-query-frontend-f746fc66d-mrjz7\" (UID: \"6f6daa03-27bc-4159-b250-d845ee3e9625\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:13:03.899864 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.899837 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:13:03.952533 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.952495 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n"] Oct 30 04:13:03.957209 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.956718 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:03.987783 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.985325 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\"" Oct 30 04:13:03.987783 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.985383 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-ca-bundle\"" Oct 30 04:13:03.990195 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:03.990168 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n"] Oct 30 04:13:04.044525 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.044452 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/da4ac54e-413f-44a1-90cd-9be5dc4e133b-tenants\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.044525 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.044499 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-tls-secret\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.044705 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.044604 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.044705 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.044626 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.044705 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.044645 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vphm2\" (UniqueName: \"kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-kube-api-access-vphm2\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.044705 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.044669 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-lokistack-gateway\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.044854 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.044804 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-rbac\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.044854 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.044840 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.045575 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.045546 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-http\"" Oct 30 04:13:04.045713 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.045669 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\"" Oct 30 04:13:04.045822 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.045804 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-client-http\"" Oct 30 04:13:04.045869 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.045807 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-dockercfg-v9896\"" Oct 30 04:13:04.145195 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.145145 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/da4ac54e-413f-44a1-90cd-9be5dc4e133b-tenants\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.145342 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.145202 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-tls-secret\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.145342 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.145246 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.145342 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.145265 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.145342 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.145283 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vphm2\" (UniqueName: \"kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-kube-api-access-vphm2\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.145342 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.145302 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-lokistack-gateway\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.145342 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.145322 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-rbac\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.145692 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.145353 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.146196 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.146169 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-lokistack-gateway\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.146323 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.146213 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.146323 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.146244 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-rbac\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.147718 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.147693 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/da4ac54e-413f-44a1-90cd-9be5dc4e133b-tenants\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.147819 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.147718 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-tls-secret\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.148039 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.148023 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.148117 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.148100 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.169079 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.169058 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vphm2\" (UniqueName: \"kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-kube-api-access-vphm2\") pod \"data-science-lokistack-gateway-557958b888-gf65n\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.207935 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.207894 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7"] Oct 30 04:13:04.209522 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:13:04.209498 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6daa03_27bc_4159_b250_d845ee3e9625.slice/crio-aefd39e496ea1c3f61cd6b786acc9097eb5c2b2fe6e9cc288d444ab8049f39d5 WatchSource:0}: Error finding container aefd39e496ea1c3f61cd6b786acc9097eb5c2b2fe6e9cc288d444ab8049f39d5: Status 404 returned error can't find the container with id aefd39e496ea1c3f61cd6b786acc9097eb5c2b2fe6e9cc288d444ab8049f39d5 Oct 30 04:13:04.277879 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.277855 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:04.463805 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.463766 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" event={"ID":"6f6daa03-27bc-4159-b250-d845ee3e9625","Type":"ContainerStarted","Data":"40cf26ab808a5cfa4469619071c53cb7647e9c14132ceebbd4a425793bed1358"} Oct 30 04:13:04.463969 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.463830 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:13:04.463969 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.463849 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" event={"ID":"6f6daa03-27bc-4159-b250-d845ee3e9625","Type":"ContainerStarted","Data":"aefd39e496ea1c3f61cd6b786acc9097eb5c2b2fe6e9cc288d444ab8049f39d5"} Oct 30 04:13:04.505562 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.505514 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" podStartSLOduration=1.5055005879999999 podStartE2EDuration="1.505500588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:04.504842237 +0000 UTC m=+5545.518120339" watchObservedRunningTime="2026-10-30 04:13:04.505500588 +0000 UTC m=+5545.518778691" Oct 30 04:13:04.595002 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:04.594926 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n"] Oct 30 04:13:04.598208 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:13:04.598178 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4ac54e_413f_44a1_90cd_9be5dc4e133b.slice/crio-62030aece98259667e080d2e2c2e88a9a2dab75f3eecdbedd55783b65ec39ac6 WatchSource:0}: Error finding container 62030aece98259667e080d2e2c2e88a9a2dab75f3eecdbedd55783b65ec39ac6: Status 404 returned error can't find the container with id 62030aece98259667e080d2e2c2e88a9a2dab75f3eecdbedd55783b65ec39ac6 Oct 30 04:13:05.470302 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:05.470266 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" event={"ID":"da4ac54e-413f-44a1-90cd-9be5dc4e133b","Type":"ContainerStarted","Data":"83c7f4bd540e9b1ee5045bb2fe64d8d1c60dc300c6494bb888880bda690cc717"} Oct 30 04:13:05.470536 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:05.470405 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" event={"ID":"da4ac54e-413f-44a1-90cd-9be5dc4e133b","Type":"ContainerStarted","Data":"93b5024cf705990b462f86463a273d164a7fb610dca1e79e068d6bb0c84c5694"} Oct 30 04:13:05.470536 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:05.470438 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" event={"ID":"da4ac54e-413f-44a1-90cd-9be5dc4e133b","Type":"ContainerStarted","Data":"62030aece98259667e080d2e2c2e88a9a2dab75f3eecdbedd55783b65ec39ac6"} Oct 30 04:13:05.470789 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:05.470764 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:05.470900 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:05.470813 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:05.476555 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:05.476532 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:05.479731 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:05.479714 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:13:05.515903 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:05.515845 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" podStartSLOduration=2.515830442 podStartE2EDuration="2.515830442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:05.515038804 +0000 UTC m=+5546.528316906" watchObservedRunningTime="2026-10-30 04:13:05.515830442 +0000 UTC m=+5546.529108544" Oct 30 04:13:10.099588 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:10.099545 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="52925eb5-4cb7-4483-b0fd-a299621be317" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:13:18.161120 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:13:18.161076 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:13:18.324655 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.324629 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:18.365597 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.365531 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52925eb5-4cb7-4483-b0fd-a299621be317-kube-api-access-k5c6c\" (UniqueName: \"kubernetes.io/projected/52925eb5-4cb7-4483-b0fd-a299621be317-kube-api-access-k5c6c\") pod \"52925eb5-4cb7-4483-b0fd-a299621be317\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " Oct 30 04:13:18.365597 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.365567 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ingester-grpc\") pod \"52925eb5-4cb7-4483-b0fd-a299621be317\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " Oct 30 04:13:18.365781 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.365670 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"52925eb5-4cb7-4483-b0fd-a299621be317\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " Oct 30 04:13:18.365781 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.365737 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"52925eb5-4cb7-4483-b0fd-a299621be317\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " Oct 30 04:13:18.365865 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.365794 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-test-loki-lifecycle-secret\") pod \"52925eb5-4cb7-4483-b0fd-a299621be317\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " Oct 30 04:13:18.365910 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.365860 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52925eb5-4cb7-4483-b0fd-a299621be317-config\" (UniqueName: \"kubernetes.io/configmap/52925eb5-4cb7-4483-b0fd-a299621be317-config\") pod \"52925eb5-4cb7-4483-b0fd-a299621be317\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " Oct 30 04:13:18.365910 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.365892 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ingester-http\") pod \"52925eb5-4cb7-4483-b0fd-a299621be317\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " Oct 30 04:13:18.366011 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.365927 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ca-bundle\") pod \"52925eb5-4cb7-4483-b0fd-a299621be317\" (UID: \"52925eb5-4cb7-4483-b0fd-a299621be317\") " Oct 30 04:13:18.366578 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.366514 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52925eb5-4cb7-4483-b0fd-a299621be317-config" pod "52925eb5-4cb7-4483-b0fd-a299621be317" (UID: "52925eb5-4cb7-4483-b0fd-a299621be317"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:13:18.366578 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.366557 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ca-bundle" pod "52925eb5-4cb7-4483-b0fd-a299621be317" (UID: "52925eb5-4cb7-4483-b0fd-a299621be317"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:13:18.368104 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.368069 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52925eb5-4cb7-4483-b0fd-a299621be317-kube-api-access-k5c6c" pod "52925eb5-4cb7-4483-b0fd-a299621be317" (UID: "52925eb5-4cb7-4483-b0fd-a299621be317"). InnerVolumeSpecName "kube-api-access-k5c6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:13:18.368246 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.368221 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ingester-grpc" pod "52925eb5-4cb7-4483-b0fd-a299621be317" (UID: "52925eb5-4cb7-4483-b0fd-a299621be317"). InnerVolumeSpecName "data-science-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:13:18.368464 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.368375 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-test-loki-lifecycle-secret" pod "52925eb5-4cb7-4483-b0fd-a299621be317" (UID: "52925eb5-4cb7-4483-b0fd-a299621be317"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:13:18.368464 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.368437 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ingester-http" pod "52925eb5-4cb7-4483-b0fd-a299621be317" (UID: "52925eb5-4cb7-4483-b0fd-a299621be317"). InnerVolumeSpecName "data-science-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:13:18.368959 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.368936 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f" pod "52925eb5-4cb7-4483-b0fd-a299621be317" (UID: "52925eb5-4cb7-4483-b0fd-a299621be317"). InnerVolumeSpecName "pvc-1477050d-bd59-4ae1-b712-0b73e1306b06". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:13:18.369522 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.369498 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18" pod "52925eb5-4cb7-4483-b0fd-a299621be317" (UID: "52925eb5-4cb7-4483-b0fd-a299621be317"). InnerVolumeSpecName "pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:13:18.466874 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.466840 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/52925eb5-4cb7-4483-b0fd-a299621be317-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:13:18.466874 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.466871 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ingester-http\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:13:18.467070 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.466882 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ca-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:13:18.467070 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.466892 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5c6c\" (UniqueName: \"kubernetes.io/projected/52925eb5-4cb7-4483-b0fd-a299621be317-kube-api-access-k5c6c\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:13:18.467070 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.466901 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-data-science-lokistack-ingester-grpc\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:13:18.467070 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.466927 4925 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") on node \"ip-10-0-139-19.ec2.internal\" " Oct 30 04:13:18.467070 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.466943 4925 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") on node \"ip-10-0-139-19.ec2.internal\" " Oct 30 04:13:18.467070 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.466956 4925 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/52925eb5-4cb7-4483-b0fd-a299621be317-test-loki-lifecycle-secret\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:13:18.476356 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.476335 4925 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18") on node "ip-10-0-139-19.ec2.internal" Oct 30 04:13:18.477036 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.477021 4925 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-1477050d-bd59-4ae1-b712-0b73e1306b06" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f") on node "ip-10-0-139-19.ec2.internal" Oct 30 04:13:18.534237 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.534201 4925 generic.go:356] "Generic (PLEG): container finished" podID="52925eb5-4cb7-4483-b0fd-a299621be317" containerID="1a2718279f43b5d0ed5417ac66e4ae98d88e6c238fa77555bad3495108fcb7bb" exitCode=0 Oct 30 04:13:18.534375 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.534257 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:18.534375 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.534276 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"52925eb5-4cb7-4483-b0fd-a299621be317","Type":"ContainerDied","Data":"1a2718279f43b5d0ed5417ac66e4ae98d88e6c238fa77555bad3495108fcb7bb"} Oct 30 04:13:18.534375 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.534326 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"52925eb5-4cb7-4483-b0fd-a299621be317","Type":"ContainerDied","Data":"135b8305b7e7c954ccae95dd26ae48b58aa925e05079b5c5fed2afefdd36a5b1"} Oct 30 04:13:18.534375 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.534346 4925 scope.go:122] "RemoveContainer" containerID="1a2718279f43b5d0ed5417ac66e4ae98d88e6c238fa77555bad3495108fcb7bb" Oct 30 04:13:18.543473 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.543458 4925 scope.go:122] "RemoveContainer" containerID="1a2718279f43b5d0ed5417ac66e4ae98d88e6c238fa77555bad3495108fcb7bb" Oct 30 04:13:18.543718 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:13:18.543701 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a2718279f43b5d0ed5417ac66e4ae98d88e6c238fa77555bad3495108fcb7bb\": container with ID starting with 1a2718279f43b5d0ed5417ac66e4ae98d88e6c238fa77555bad3495108fcb7bb not found: ID does not exist" containerID="1a2718279f43b5d0ed5417ac66e4ae98d88e6c238fa77555bad3495108fcb7bb" Oct 30 04:13:18.543809 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.543723 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a2718279f43b5d0ed5417ac66e4ae98d88e6c238fa77555bad3495108fcb7bb"} err="failed to get container status \"1a2718279f43b5d0ed5417ac66e4ae98d88e6c238fa77555bad3495108fcb7bb\": rpc error: code = NotFound desc = could not find container \"1a2718279f43b5d0ed5417ac66e4ae98d88e6c238fa77555bad3495108fcb7bb\": container with ID starting with 1a2718279f43b5d0ed5417ac66e4ae98d88e6c238fa77555bad3495108fcb7bb not found: ID does not exist" Oct 30 04:13:18.568236 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.568201 4925 reconciler_common.go:299] "Volume detached for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"csi-d631f5804e757beb9cee949b1ba08adddfe0f2a6d74da288dfa6a5946ab2fad2\"" Oct 30 04:13:18.568236 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.568234 4925 reconciler_common.go:299] "Volume detached for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"csi-1cbd6671aaaf13073dd4ac6088fd398781af87903101ff82770e0956bdbc63f1\"" Oct 30 04:13:18.586968 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.586944 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 04:13:18.590807 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:18.590787 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 04:13:19.119828 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:19.119796 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52925eb5-4cb7-4483-b0fd-a299621be317" path="/var/lib/kubelet/pods/52925eb5-4cb7-4483-b0fd-a299621be317/volumes" Oct 30 04:13:19.983170 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:19.983131 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:20.013898 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:20.013863 4925 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:20.083926 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:20.083885 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:20.084097 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:20.084021 4925 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:20.098642 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:20.098617 4925 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") DevicePath \"csi-3faa7a4919429bfae9e5fe05bb9f7e440b0f7cbd5797a7f1cb9f75e405a5c758\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:20.201549 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:20.201519 4925 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2a84ccb0e632fc97a562bedb1c2138c4534d44455a54f3f6cb9692580fc3a7df/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:20.230239 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:20.230208 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:20.331925 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:20.331893 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:20.654104 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:20.654080 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 04:13:20.656996 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:13:20.656963 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b21fd1_a5a2_473d_94fa_cc22f744c736.slice/crio-49b00ff263bc0561d6320b05817577e6f5c13f22e0cc8a4860483beeed17c7d7 WatchSource:0}: Error finding container 49b00ff263bc0561d6320b05817577e6f5c13f22e0cc8a4860483beeed17c7d7: Status 404 returned error can't find the container with id 49b00ff263bc0561d6320b05817577e6f5c13f22e0cc8a4860483beeed17c7d7 Oct 30 04:13:21.552837 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:21.552791 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"11b21fd1-a5a2-473d-94fa-cc22f744c736","Type":"ContainerStarted","Data":"1bdadd9dcc64718e42a9161efa5e25a423216c2d47d7603e3a85713cb876b0aa"} Oct 30 04:13:21.553253 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:21.552853 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"11b21fd1-a5a2-473d-94fa-cc22f744c736","Type":"ContainerStarted","Data":"49b00ff263bc0561d6320b05817577e6f5c13f22e0cc8a4860483beeed17c7d7"} Oct 30 04:13:21.553253 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:21.553015 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:21.606096 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:21.606047 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" podStartSLOduration=18.606033897 podStartE2EDuration="18.606033897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:21.596705633 +0000 UTC m=+5562.609983736" watchObservedRunningTime="2026-10-30 04:13:21.606033897 +0000 UTC m=+5562.619311994" Oct 30 04:13:24.002456 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.002420 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 04:13:24.002817 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.002766 4925 cpu_manager.go:414] "containerMap: removing container" podUID="52925eb5-4cb7-4483-b0fd-a299621be317" containerName="loki-ingester" Oct 30 04:13:24.002817 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.002780 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52925eb5-4cb7-4483-b0fd-a299621be317" containerName="loki-ingester" Oct 30 04:13:24.002904 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.002862 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="52925eb5-4cb7-4483-b0fd-a299621be317" containerName="loki-ingester" Oct 30 04:13:24.006040 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.006018 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:24.033614 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.033542 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-http\"" Oct 30 04:13:24.033614 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.033594 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-grpc\"" Oct 30 04:13:24.035493 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.035474 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 04:13:24.132837 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.132804 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:24.132987 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.132848 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:24.132987 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.132918 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:24.133088 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.133043 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:24.133271 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.133249 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:24.133321 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.133293 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ch5s\" (UniqueName: \"kubernetes.io/projected/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-kube-api-access-4ch5s\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:24.234188 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.234143 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:24.234349 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.234221 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:24.234349 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.234244 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:24.234349 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.234267 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ch5s\" (UniqueName: \"kubernetes.io/projected/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-kube-api-access-4ch5s\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:24.234349 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.234290 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:24.234349 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.234335 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:24.235107 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.235080 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:24.235461 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.235187 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:24.237241 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.237213 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:24.237361 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.237279 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:24.237361 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.237296 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:24.258415 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:24.258341 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ch5s\" (UniqueName: \"kubernetes.io/projected/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-kube-api-access-4ch5s\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:25.476597 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:25.476564 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:13:29.381946 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:29.381898 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:29.411948 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:29.411912 4925 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:29.483038 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:29.483002 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:29.483207 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:29.483144 4925 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:29.502970 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:29.502944 4925 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") DevicePath \"csi-1cbd6671aaaf13073dd4ac6088fd398781af87903101ff82770e0956bdbc63f1\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:29.607466 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:29.607430 4925 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/ad17c5ee2a7e4881911e4c7c70dd9b4982223d2ab904c958abdf2443e32ecc28/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:29.638398 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:29.638327 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:32.107778 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:32.107716 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:32.138317 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:32.138279 4925 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:32.209346 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:32.209301 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:32.209566 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:32.209454 4925 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:32.225191 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:32.225159 4925 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") DevicePath \"csi-d631f5804e757beb9cee949b1ba08adddfe0f2a6d74da288dfa6a5946ab2fad2\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:32.337415 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:32.337379 4925 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4b631ef058d3f818390cfd4c55b829173556e3e1b26e1e76626d152426c8d48c/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:32.363928 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:32.363851 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"data-science-lokistack-ingester-0\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:32.418356 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:32.418316 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:32.731991 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:32.731964 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 04:13:32.733836 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:13:32.733806 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7de17d_0b8c_44ed_a9ca_e6472acf8e07.slice/crio-91027fc84cc64c3a4087f52b2a9d57525f255bd0a15f64d85118aff67a9bcfc1 WatchSource:0}: Error finding container 91027fc84cc64c3a4087f52b2a9d57525f255bd0a15f64d85118aff67a9bcfc1: Status 404 returned error can't find the container with id 91027fc84cc64c3a4087f52b2a9d57525f255bd0a15f64d85118aff67a9bcfc1 Oct 30 04:13:33.614259 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:33.614221 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07","Type":"ContainerStarted","Data":"827cbbdd3e82b8a2d3d1cd540a1c43b6855065c06520c214b444ecb46c71db17"} Oct 30 04:13:33.614661 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:33.614273 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07","Type":"ContainerStarted","Data":"91027fc84cc64c3a4087f52b2a9d57525f255bd0a15f64d85118aff67a9bcfc1"} Oct 30 04:13:33.614661 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:33.614327 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:13:33.658485 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:33.658432 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podStartSLOduration=10.658418216 podStartE2EDuration="10.658418216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:33.657953448 +0000 UTC m=+5574.671231556" watchObservedRunningTime="2026-10-30 04:13:33.658418216 +0000 UTC m=+5574.671696309" Oct 30 04:13:42.564530 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:42.564500 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:13:48.162176 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:13:48.162130 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:13:54.632144 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:13:54.632109 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="4c7de17d-0b8c-44ed-a9ca-e6472acf8e07" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:14:04.631499 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:14:04.631459 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="4c7de17d-0b8c-44ed-a9ca-e6472acf8e07" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:14:14.630774 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:14:14.630717 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:14:18.162532 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:14:18.162488 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:14:48.161782 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:14:48.161689 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:15:18.161123 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:15:18.161082 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:15:39.709496 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:15:39.709374 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 04:15:39.718148 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:15:39.718126 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 04:15:39.727118 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:15:39.727096 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 04:15:39.735421 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:15:39.735401 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 04:15:48.162052 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:15:48.162016 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:16:12.297878 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:12.297843 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6"] Oct 30 04:16:12.301392 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:12.301373 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6" Oct 30 04:16:12.317293 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:12.317260 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-j5fn8\"" Oct 30 04:16:12.329091 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:12.329056 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-6a6878d3\"" Oct 30 04:16:12.332966 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:12.332932 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6"] Oct 30 04:16:12.358985 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:12.358949 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ms4h\" (UniqueName: \"kubernetes.io/projected/64625ae8-1054-488c-b8c5-672c13d6ba05-kube-api-access-6ms4h\") pod \"data-science-usage-logs-collector-5d66b649d7-zkfx6\" (UID: \"64625ae8-1054-488c-b8c5-672c13d6ba05\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6" Oct 30 04:16:12.359181 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:12.359019 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/64625ae8-1054-488c-b8c5-672c13d6ba05-otc-internal\") pod \"data-science-usage-logs-collector-5d66b649d7-zkfx6\" (UID: \"64625ae8-1054-488c-b8c5-672c13d6ba05\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6" Oct 30 04:16:12.459721 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:12.459684 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/64625ae8-1054-488c-b8c5-672c13d6ba05-otc-internal\") pod \"data-science-usage-logs-collector-5d66b649d7-zkfx6\" (UID: \"64625ae8-1054-488c-b8c5-672c13d6ba05\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6" Oct 30 04:16:12.459942 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:12.459799 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ms4h\" (UniqueName: \"kubernetes.io/projected/64625ae8-1054-488c-b8c5-672c13d6ba05-kube-api-access-6ms4h\") pod \"data-science-usage-logs-collector-5d66b649d7-zkfx6\" (UID: \"64625ae8-1054-488c-b8c5-672c13d6ba05\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6" Oct 30 04:16:12.460354 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:12.460331 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/64625ae8-1054-488c-b8c5-672c13d6ba05-otc-internal\") pod \"data-science-usage-logs-collector-5d66b649d7-zkfx6\" (UID: \"64625ae8-1054-488c-b8c5-672c13d6ba05\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6" Oct 30 04:16:12.483711 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:12.483688 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ms4h\" (UniqueName: \"kubernetes.io/projected/64625ae8-1054-488c-b8c5-672c13d6ba05-kube-api-access-6ms4h\") pod \"data-science-usage-logs-collector-5d66b649d7-zkfx6\" (UID: \"64625ae8-1054-488c-b8c5-672c13d6ba05\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6" Oct 30 04:16:12.621518 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:12.621482 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6" Oct 30 04:16:12.921198 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:12.921169 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6"] Oct 30 04:16:12.924068 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:16:12.924039 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64625ae8_1054_488c_b8c5_672c13d6ba05.slice/crio-5ad14ab24b2dcf55de55d006de4eb6e40f65bbba62d5bf370848ccc626ac1c57 WatchSource:0}: Error finding container 5ad14ab24b2dcf55de55d006de4eb6e40f65bbba62d5bf370848ccc626ac1c57: Status 404 returned error can't find the container with id 5ad14ab24b2dcf55de55d006de4eb6e40f65bbba62d5bf370848ccc626ac1c57 Oct 30 04:16:12.926226 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:12.926197 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:16:12.926303 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:12.926252 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:16:13.407288 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:13.407249 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6" event={"ID":"64625ae8-1054-488c-b8c5-672c13d6ba05","Type":"ContainerStarted","Data":"893cd681e0c19c22cdad17a5762af492b7fb4391d8eeb44a95f7776467667ebe"} Oct 30 04:16:13.407693 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:13.407315 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6" event={"ID":"64625ae8-1054-488c-b8c5-672c13d6ba05","Type":"ContainerStarted","Data":"5ad14ab24b2dcf55de55d006de4eb6e40f65bbba62d5bf370848ccc626ac1c57"} Oct 30 04:16:13.454278 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:13.454214 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6" podStartSLOduration=1.454199786 podStartE2EDuration="1.454199786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:13.45211124 +0000 UTC m=+5734.465389343" watchObservedRunningTime="2026-10-30 04:16:13.454199786 +0000 UTC m=+5734.467477889" Oct 30 04:16:16.177219 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.177176 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 04:16:16.177746 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.177431 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" podUID="11b21fd1-a5a2-473d-94fa-cc22f744c736" containerName="loki-compactor" containerID="cri-o://1bdadd9dcc64718e42a9161efa5e25a423216c2d47d7603e3a85713cb876b0aa" gracePeriod=30 Oct 30 04:16:16.206668 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.206633 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 04:16:16.207064 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.207027 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="4c7de17d-0b8c-44ed-a9ca-e6472acf8e07" containerName="loki-ingester" containerID="cri-o://827cbbdd3e82b8a2d3d1cd540a1c43b6855065c06520c214b444ecb46c71db17" gracePeriod=600 Oct 30 04:16:16.237267 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.237227 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7"] Oct 30 04:16:16.237495 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.237471 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" podUID="6f6daa03-27bc-4159-b250-d845ee3e9625" containerName="loki-query-frontend" containerID="cri-o://40cf26ab808a5cfa4469619071c53cb7647e9c14132ceebbd4a425793bed1358" gracePeriod=30 Oct 30 04:16:16.240713 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.240657 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n"] Oct 30 04:16:16.241499 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.241097 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" podUID="da4ac54e-413f-44a1-90cd-9be5dc4e133b" containerName="gateway" containerID="cri-o://93b5024cf705990b462f86463a273d164a7fb610dca1e79e068d6bb0c84c5694" gracePeriod=30 Oct 30 04:16:16.241654 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.241628 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" podUID="da4ac54e-413f-44a1-90cd-9be5dc4e133b" containerName="opa" containerID="cri-o://83c7f4bd540e9b1ee5045bb2fe64d8d1c60dc300c6494bb888880bda690cc717" gracePeriod=30 Oct 30 04:16:16.384444 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.384398 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6"] Oct 30 04:16:16.384664 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.384636 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6" podUID="64625ae8-1054-488c-b8c5-672c13d6ba05" containerName="otc-container" containerID="cri-o://893cd681e0c19c22cdad17a5762af492b7fb4391d8eeb44a95f7776467667ebe" gracePeriod=30 Oct 30 04:16:16.423926 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.423733 4925 generic.go:356] "Generic (PLEG): container finished" podID="6f6daa03-27bc-4159-b250-d845ee3e9625" containerID="40cf26ab808a5cfa4469619071c53cb7647e9c14132ceebbd4a425793bed1358" exitCode=0 Oct 30 04:16:16.423926 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.423849 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" event={"ID":"6f6daa03-27bc-4159-b250-d845ee3e9625","Type":"ContainerDied","Data":"40cf26ab808a5cfa4469619071c53cb7647e9c14132ceebbd4a425793bed1358"} Oct 30 04:16:16.426509 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.426477 4925 generic.go:356] "Generic (PLEG): container finished" podID="da4ac54e-413f-44a1-90cd-9be5dc4e133b" containerID="83c7f4bd540e9b1ee5045bb2fe64d8d1c60dc300c6494bb888880bda690cc717" exitCode=0 Oct 30 04:16:16.426664 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.426557 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" event={"ID":"da4ac54e-413f-44a1-90cd-9be5dc4e133b","Type":"ContainerDied","Data":"83c7f4bd540e9b1ee5045bb2fe64d8d1c60dc300c6494bb888880bda690cc717"} Oct 30 04:16:16.549582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.549555 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:16:16.591390 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.591342 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f6daa03-27bc-4159-b250-d845ee3e9625-kube-api-access-g2hx4\" (UniqueName: \"kubernetes.io/projected/6f6daa03-27bc-4159-b250-d845ee3e9625-kube-api-access-g2hx4\") pod \"6f6daa03-27bc-4159-b250-d845ee3e9625\" (UID: \"6f6daa03-27bc-4159-b250-d845ee3e9625\") " Oct 30 04:16:16.592247 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.591421 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-query-frontend-http\") pod \"6f6daa03-27bc-4159-b250-d845ee3e9625\" (UID: \"6f6daa03-27bc-4159-b250-d845ee3e9625\") " Oct 30 04:16:16.592247 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.591458 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-query-frontend-grpc\") pod \"6f6daa03-27bc-4159-b250-d845ee3e9625\" (UID: \"6f6daa03-27bc-4159-b250-d845ee3e9625\") " Oct 30 04:16:16.592247 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.591492 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f6daa03-27bc-4159-b250-d845ee3e9625-config\" (UniqueName: \"kubernetes.io/configmap/6f6daa03-27bc-4159-b250-d845ee3e9625-config\") pod \"6f6daa03-27bc-4159-b250-d845ee3e9625\" (UID: \"6f6daa03-27bc-4159-b250-d845ee3e9625\") " Oct 30 04:16:16.592247 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.591532 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-ca-bundle\") pod \"6f6daa03-27bc-4159-b250-d845ee3e9625\" (UID: \"6f6daa03-27bc-4159-b250-d845ee3e9625\") " Oct 30 04:16:16.592785 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.592719 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f6daa03-27bc-4159-b250-d845ee3e9625-config" pod "6f6daa03-27bc-4159-b250-d845ee3e9625" (UID: "6f6daa03-27bc-4159-b250-d845ee3e9625"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:16.594576 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.592964 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-ca-bundle" pod "6f6daa03-27bc-4159-b250-d845ee3e9625" (UID: "6f6daa03-27bc-4159-b250-d845ee3e9625"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:16.594576 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.592996 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f6daa03-27bc-4159-b250-d845ee3e9625-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.595153 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.595127 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-query-frontend-http" pod "6f6daa03-27bc-4159-b250-d845ee3e9625" (UID: "6f6daa03-27bc-4159-b250-d845ee3e9625"). InnerVolumeSpecName "data-science-lokistack-query-frontend-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:16.595269 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.595216 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f6daa03-27bc-4159-b250-d845ee3e9625-kube-api-access-g2hx4" pod "6f6daa03-27bc-4159-b250-d845ee3e9625" (UID: "6f6daa03-27bc-4159-b250-d845ee3e9625"). InnerVolumeSpecName "kube-api-access-g2hx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:16:16.596059 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.596035 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-query-frontend-grpc" pod "6f6daa03-27bc-4159-b250-d845ee3e9625" (UID: "6f6daa03-27bc-4159-b250-d845ee3e9625"). InnerVolumeSpecName "data-science-lokistack-query-frontend-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:16.650797 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.650744 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6" Oct 30 04:16:16.694355 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.694271 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64625ae8-1054-488c-b8c5-672c13d6ba05-kube-api-access-6ms4h\" (UniqueName: \"kubernetes.io/projected/64625ae8-1054-488c-b8c5-672c13d6ba05-kube-api-access-6ms4h\") pod \"64625ae8-1054-488c-b8c5-672c13d6ba05\" (UID: \"64625ae8-1054-488c-b8c5-672c13d6ba05\") " Oct 30 04:16:16.694355 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.694326 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64625ae8-1054-488c-b8c5-672c13d6ba05-otc-internal\" (UniqueName: \"kubernetes.io/configmap/64625ae8-1054-488c-b8c5-672c13d6ba05-otc-internal\") pod \"64625ae8-1054-488c-b8c5-672c13d6ba05\" (UID: \"64625ae8-1054-488c-b8c5-672c13d6ba05\") " Oct 30 04:16:16.694579 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.694565 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-ca-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.694638 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.694584 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2hx4\" (UniqueName: \"kubernetes.io/projected/6f6daa03-27bc-4159-b250-d845ee3e9625-kube-api-access-g2hx4\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.694638 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.694601 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-query-frontend-http\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.694638 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.694616 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/6f6daa03-27bc-4159-b250-d845ee3e9625-data-science-lokistack-query-frontend-grpc\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.694826 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.694703 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64625ae8-1054-488c-b8c5-672c13d6ba05-otc-internal" pod "64625ae8-1054-488c-b8c5-672c13d6ba05" (UID: "64625ae8-1054-488c-b8c5-672c13d6ba05"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:16.696501 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.696475 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64625ae8-1054-488c-b8c5-672c13d6ba05-kube-api-access-6ms4h" pod "64625ae8-1054-488c-b8c5-672c13d6ba05" (UID: "64625ae8-1054-488c-b8c5-672c13d6ba05"). InnerVolumeSpecName "kube-api-access-6ms4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:16:16.795877 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.795845 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/64625ae8-1054-488c-b8c5-672c13d6ba05-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:16.795877 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:16.795873 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ms4h\" (UniqueName: \"kubernetes.io/projected/64625ae8-1054-488c-b8c5-672c13d6ba05-kube-api-access-6ms4h\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.178061 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.178036 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:16:17.198967 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.198934 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-kube-api-access-4ch5s\" (UniqueName: \"kubernetes.io/projected/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-kube-api-access-4ch5s\") pod \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " Oct 30 04:16:17.199151 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.199058 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") pod \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " Oct 30 04:16:17.199151 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.199112 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") pod \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " Oct 30 04:16:17.199277 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.199187 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ingester-grpc\") pod \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " Oct 30 04:16:17.199277 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.199239 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ca-bundle\") pod \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " Oct 30 04:16:17.199389 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.199278 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ingester-http\") pod \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " Oct 30 04:16:17.199389 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.199333 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-test-loki-lifecycle-secret\") pod \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " Oct 30 04:16:17.199389 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.199365 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-config\" (UniqueName: \"kubernetes.io/configmap/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-config\") pod \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\" (UID: \"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07\") " Oct 30 04:16:17.200077 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.200045 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ca-bundle" pod "4c7de17d-0b8c-44ed-a9ca-e6472acf8e07" (UID: "4c7de17d-0b8c-44ed-a9ca-e6472acf8e07"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:17.202506 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.200264 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-config" pod "4c7de17d-0b8c-44ed-a9ca-e6472acf8e07" (UID: "4c7de17d-0b8c-44ed-a9ca-e6472acf8e07"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:17.202506 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.201958 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ingester-grpc" pod "4c7de17d-0b8c-44ed-a9ca-e6472acf8e07" (UID: "4c7de17d-0b8c-44ed-a9ca-e6472acf8e07"). InnerVolumeSpecName "data-science-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:17.202506 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.202412 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-test-loki-lifecycle-secret" pod "4c7de17d-0b8c-44ed-a9ca-e6472acf8e07" (UID: "4c7de17d-0b8c-44ed-a9ca-e6472acf8e07"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:17.203061 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.203040 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f" pod "4c7de17d-0b8c-44ed-a9ca-e6472acf8e07" (UID: "4c7de17d-0b8c-44ed-a9ca-e6472acf8e07"). InnerVolumeSpecName "pvc-1477050d-bd59-4ae1-b712-0b73e1306b06". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:16:17.203173 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.203087 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18" pod "4c7de17d-0b8c-44ed-a9ca-e6472acf8e07" (UID: "4c7de17d-0b8c-44ed-a9ca-e6472acf8e07"). InnerVolumeSpecName "pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:16:17.203173 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.203099 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-kube-api-access-4ch5s" pod "4c7de17d-0b8c-44ed-a9ca-e6472acf8e07" (UID: "4c7de17d-0b8c-44ed-a9ca-e6472acf8e07"). InnerVolumeSpecName "kube-api-access-4ch5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:16:17.204234 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.204193 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ingester-http" pod "4c7de17d-0b8c-44ed-a9ca-e6472acf8e07" (UID: "4c7de17d-0b8c-44ed-a9ca-e6472acf8e07"). InnerVolumeSpecName "data-science-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:17.300065 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.299981 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ingester-http\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.300065 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.300014 4925 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-test-loki-lifecycle-secret\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.300065 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.300028 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.300065 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.300039 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ch5s\" (UniqueName: \"kubernetes.io/projected/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-kube-api-access-4ch5s\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.300065 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.300067 4925 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") on node \"ip-10-0-139-19.ec2.internal\" " Oct 30 04:16:17.300359 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.300080 4925 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") on node \"ip-10-0-139-19.ec2.internal\" " Oct 30 04:16:17.300359 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.300089 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ingester-grpc\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.300359 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.300098 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07-data-science-lokistack-ca-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.310902 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.309951 4925 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18") on node "ip-10-0-139-19.ec2.internal" Oct 30 04:16:17.313385 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.313364 4925 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-1477050d-bd59-4ae1-b712-0b73e1306b06" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f") on node "ip-10-0-139-19.ec2.internal" Oct 30 04:16:17.401010 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.400971 4925 reconciler_common.go:299] "Volume detached for volume \"pvc-1477050d-bd59-4ae1-b712-0b73e1306b06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03b565559276d935f\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"csi-1cbd6671aaaf13073dd4ac6088fd398781af87903101ff82770e0956bdbc63f1\"" Oct 30 04:16:17.401159 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.401014 4925 reconciler_common.go:299] "Volume detached for volume \"pvc-30276bc3-ed22-4300-b413-3a80f5fc7ec4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05059212a43cd4e18\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"csi-d631f5804e757beb9cee949b1ba08adddfe0f2a6d74da288dfa6a5946ab2fad2\"" Oct 30 04:16:17.433950 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.433916 4925 generic.go:356] "Generic (PLEG): container finished" podID="4c7de17d-0b8c-44ed-a9ca-e6472acf8e07" containerID="827cbbdd3e82b8a2d3d1cd540a1c43b6855065c06520c214b444ecb46c71db17" exitCode=0 Oct 30 04:16:17.434114 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.433975 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 30 04:16:17.434114 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.434060 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07","Type":"ContainerDied","Data":"827cbbdd3e82b8a2d3d1cd540a1c43b6855065c06520c214b444ecb46c71db17"} Oct 30 04:16:17.434217 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.434136 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"4c7de17d-0b8c-44ed-a9ca-e6472acf8e07","Type":"ContainerDied","Data":"91027fc84cc64c3a4087f52b2a9d57525f255bd0a15f64d85118aff67a9bcfc1"} Oct 30 04:16:17.434217 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.434164 4925 scope.go:122] "RemoveContainer" containerID="827cbbdd3e82b8a2d3d1cd540a1c43b6855065c06520c214b444ecb46c71db17" Oct 30 04:16:17.436338 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.436313 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" event={"ID":"6f6daa03-27bc-4159-b250-d845ee3e9625","Type":"ContainerDied","Data":"aefd39e496ea1c3f61cd6b786acc9097eb5c2b2fe6e9cc288d444ab8049f39d5"} Oct 30 04:16:17.436451 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.436434 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7" Oct 30 04:16:17.441564 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.441537 4925 generic.go:356] "Generic (PLEG): container finished" podID="da4ac54e-413f-44a1-90cd-9be5dc4e133b" containerID="93b5024cf705990b462f86463a273d164a7fb610dca1e79e068d6bb0c84c5694" exitCode=0 Oct 30 04:16:17.441699 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.441607 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" event={"ID":"da4ac54e-413f-44a1-90cd-9be5dc4e133b","Type":"ContainerDied","Data":"93b5024cf705990b462f86463a273d164a7fb610dca1e79e068d6bb0c84c5694"} Oct 30 04:16:17.444277 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.444255 4925 generic.go:356] "Generic (PLEG): container finished" podID="64625ae8-1054-488c-b8c5-672c13d6ba05" containerID="893cd681e0c19c22cdad17a5762af492b7fb4391d8eeb44a95f7776467667ebe" exitCode=0 Oct 30 04:16:17.444397 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.444304 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6" event={"ID":"64625ae8-1054-488c-b8c5-672c13d6ba05","Type":"ContainerDied","Data":"893cd681e0c19c22cdad17a5762af492b7fb4391d8eeb44a95f7776467667ebe"} Oct 30 04:16:17.444397 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.444310 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6" Oct 30 04:16:17.444397 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.444341 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6" event={"ID":"64625ae8-1054-488c-b8c5-672c13d6ba05","Type":"ContainerDied","Data":"5ad14ab24b2dcf55de55d006de4eb6e40f65bbba62d5bf370848ccc626ac1c57"} Oct 30 04:16:17.486706 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.486675 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7"] Oct 30 04:16:17.491011 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.490982 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-f746fc66d-mrjz7"] Oct 30 04:16:17.526325 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.525637 4925 scope.go:122] "RemoveContainer" containerID="827cbbdd3e82b8a2d3d1cd540a1c43b6855065c06520c214b444ecb46c71db17" Oct 30 04:16:17.526325 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:16:17.526049 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"827cbbdd3e82b8a2d3d1cd540a1c43b6855065c06520c214b444ecb46c71db17\": container with ID starting with 827cbbdd3e82b8a2d3d1cd540a1c43b6855065c06520c214b444ecb46c71db17 not found: ID does not exist" containerID="827cbbdd3e82b8a2d3d1cd540a1c43b6855065c06520c214b444ecb46c71db17" Oct 30 04:16:17.526325 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.526083 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"827cbbdd3e82b8a2d3d1cd540a1c43b6855065c06520c214b444ecb46c71db17"} err="failed to get container status \"827cbbdd3e82b8a2d3d1cd540a1c43b6855065c06520c214b444ecb46c71db17\": rpc error: code = NotFound desc = could not find container \"827cbbdd3e82b8a2d3d1cd540a1c43b6855065c06520c214b444ecb46c71db17\": container with ID starting with 827cbbdd3e82b8a2d3d1cd540a1c43b6855065c06520c214b444ecb46c71db17 not found: ID does not exist" Oct 30 04:16:17.526325 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.526101 4925 scope.go:122] "RemoveContainer" containerID="40cf26ab808a5cfa4469619071c53cb7647e9c14132ceebbd4a425793bed1358" Oct 30 04:16:17.532107 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.532085 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:16:17.535230 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.535212 4925 scope.go:122] "RemoveContainer" containerID="893cd681e0c19c22cdad17a5762af492b7fb4391d8eeb44a95f7776467667ebe" Oct 30 04:16:17.543175 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.543158 4925 scope.go:122] "RemoveContainer" containerID="893cd681e0c19c22cdad17a5762af492b7fb4391d8eeb44a95f7776467667ebe" Oct 30 04:16:17.543448 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:16:17.543430 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"893cd681e0c19c22cdad17a5762af492b7fb4391d8eeb44a95f7776467667ebe\": container with ID starting with 893cd681e0c19c22cdad17a5762af492b7fb4391d8eeb44a95f7776467667ebe not found: ID does not exist" containerID="893cd681e0c19c22cdad17a5762af492b7fb4391d8eeb44a95f7776467667ebe" Oct 30 04:16:17.543518 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.543456 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"893cd681e0c19c22cdad17a5762af492b7fb4391d8eeb44a95f7776467667ebe"} err="failed to get container status \"893cd681e0c19c22cdad17a5762af492b7fb4391d8eeb44a95f7776467667ebe\": rpc error: code = NotFound desc = could not find container \"893cd681e0c19c22cdad17a5762af492b7fb4391d8eeb44a95f7776467667ebe\": container with ID starting with 893cd681e0c19c22cdad17a5762af492b7fb4391d8eeb44a95f7776467667ebe not found: ID does not exist" Oct 30 04:16:17.562592 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.562519 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6"] Oct 30 04:16:17.567923 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.567888 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-zkfx6"] Oct 30 04:16:17.602055 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.602021 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-ca-bundle\") pod \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " Oct 30 04:16:17.602055 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.602062 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-rbac\" (UniqueName: \"kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-rbac\") pod \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " Oct 30 04:16:17.602298 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.602106 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da4ac54e-413f-44a1-90cd-9be5dc4e133b-tenants\" (UniqueName: \"kubernetes.io/secret/da4ac54e-413f-44a1-90cd-9be5dc4e133b-tenants\") pod \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " Oct 30 04:16:17.602298 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.602177 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-tls-secret\" (UniqueName: \"kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-tls-secret\") pod \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " Oct 30 04:16:17.602298 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.602213 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-lokistack-gateway\") pod \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " Oct 30 04:16:17.602298 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.602246 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-kube-api-access-vphm2\" (UniqueName: \"kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-kube-api-access-vphm2\") pod \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " Oct 30 04:16:17.602298 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.602286 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-gateway-ca-bundle\") pod \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " Oct 30 04:16:17.602997 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.602332 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-gateway-client-http\") pod \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\" (UID: \"da4ac54e-413f-44a1-90cd-9be5dc4e133b\") " Oct 30 04:16:17.603057 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.603040 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-ca-bundle" pod "da4ac54e-413f-44a1-90cd-9be5dc4e133b" (UID: "da4ac54e-413f-44a1-90cd-9be5dc4e133b"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:17.605035 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.605007 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 04:16:17.605896 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.605815 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-tls-secret" pod "da4ac54e-413f-44a1-90cd-9be5dc4e133b" (UID: "da4ac54e-413f-44a1-90cd-9be5dc4e133b"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:16:17.606309 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.606278 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-gateway-client-http" pod "da4ac54e-413f-44a1-90cd-9be5dc4e133b" (UID: "da4ac54e-413f-44a1-90cd-9be5dc4e133b"). InnerVolumeSpecName "data-science-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:17.606604 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.606581 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-gateway-ca-bundle" pod "da4ac54e-413f-44a1-90cd-9be5dc4e133b" (UID: "da4ac54e-413f-44a1-90cd-9be5dc4e133b"). InnerVolumeSpecName "data-science-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:16:17.607100 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.607073 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-kube-api-access-vphm2" pod "da4ac54e-413f-44a1-90cd-9be5dc4e133b" (UID: "da4ac54e-413f-44a1-90cd-9be5dc4e133b"). InnerVolumeSpecName "kube-api-access-vphm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:16:17.608256 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.608232 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 30 04:16:17.609377 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.609356 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-rbac" pod "da4ac54e-413f-44a1-90cd-9be5dc4e133b" (UID: "da4ac54e-413f-44a1-90cd-9be5dc4e133b"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:17.610575 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.610550 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-lokistack-gateway" pod "da4ac54e-413f-44a1-90cd-9be5dc4e133b" (UID: "da4ac54e-413f-44a1-90cd-9be5dc4e133b"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:17.611835 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.611730 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da4ac54e-413f-44a1-90cd-9be5dc4e133b-tenants" pod "da4ac54e-413f-44a1-90cd-9be5dc4e133b" (UID: "da4ac54e-413f-44a1-90cd-9be5dc4e133b"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:17.615994 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.615975 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:16:17.703584 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.703553 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b21fd1-a5a2-473d-94fa-cc22f744c736-kube-api-access-q8q55\" (UniqueName: \"kubernetes.io/projected/11b21fd1-a5a2-473d-94fa-cc22f744c736-kube-api-access-q8q55\") pod \"11b21fd1-a5a2-473d-94fa-cc22f744c736\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " Oct 30 04:16:17.703791 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.703596 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-compactor-http\") pod \"11b21fd1-a5a2-473d-94fa-cc22f744c736\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " Oct 30 04:16:17.703791 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.703712 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"11b21fd1-a5a2-473d-94fa-cc22f744c736\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " Oct 30 04:16:17.703791 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.703783 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-compactor-grpc\") pod \"11b21fd1-a5a2-473d-94fa-cc22f744c736\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " Oct 30 04:16:17.704349 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.703840 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-test-loki-lifecycle-secret\") pod \"11b21fd1-a5a2-473d-94fa-cc22f744c736\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " Oct 30 04:16:17.704349 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.703880 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-ca-bundle\") pod \"11b21fd1-a5a2-473d-94fa-cc22f744c736\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " Oct 30 04:16:17.704349 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.703988 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11b21fd1-a5a2-473d-94fa-cc22f744c736-config\" (UniqueName: \"kubernetes.io/configmap/11b21fd1-a5a2-473d-94fa-cc22f744c736-config\") pod \"11b21fd1-a5a2-473d-94fa-cc22f744c736\" (UID: \"11b21fd1-a5a2-473d-94fa-cc22f744c736\") " Oct 30 04:16:17.704349 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.704202 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-ca-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.704349 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.704222 4925 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-rbac\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.704349 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.704235 4925 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/da4ac54e-413f-44a1-90cd-9be5dc4e133b-tenants\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.704349 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.704247 4925 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-tls-secret\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.704349 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.704260 4925 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/da4ac54e-413f-44a1-90cd-9be5dc4e133b-lokistack-gateway\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.704349 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.704273 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vphm2\" (UniqueName: \"kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-kube-api-access-vphm2\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.704349 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.704286 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-gateway-ca-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.704349 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.704302 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/da4ac54e-413f-44a1-90cd-9be5dc4e133b-data-science-lokistack-gateway-client-http\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.705494 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.704441 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-ca-bundle" pod "11b21fd1-a5a2-473d-94fa-cc22f744c736" (UID: "11b21fd1-a5a2-473d-94fa-cc22f744c736"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:17.705494 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.704638 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b21fd1-a5a2-473d-94fa-cc22f744c736-config" pod "11b21fd1-a5a2-473d-94fa-cc22f744c736" (UID: "11b21fd1-a5a2-473d-94fa-cc22f744c736"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:17.706401 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.706371 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b21fd1-a5a2-473d-94fa-cc22f744c736-kube-api-access-q8q55" pod "11b21fd1-a5a2-473d-94fa-cc22f744c736" (UID: "11b21fd1-a5a2-473d-94fa-cc22f744c736"). InnerVolumeSpecName "kube-api-access-q8q55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:16:17.706492 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.706432 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-compactor-http" pod "11b21fd1-a5a2-473d-94fa-cc22f744c736" (UID: "11b21fd1-a5a2-473d-94fa-cc22f744c736"). InnerVolumeSpecName "data-science-lokistack-compactor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:17.706492 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.706464 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-test-loki-lifecycle-secret" pod "11b21fd1-a5a2-473d-94fa-cc22f744c736" (UID: "11b21fd1-a5a2-473d-94fa-cc22f744c736"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:17.706696 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.706677 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-compactor-grpc" pod "11b21fd1-a5a2-473d-94fa-cc22f744c736" (UID: "11b21fd1-a5a2-473d-94fa-cc22f744c736"). InnerVolumeSpecName "data-science-lokistack-compactor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:17.707057 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.707037 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e" pod "11b21fd1-a5a2-473d-94fa-cc22f744c736" (UID: "11b21fd1-a5a2-473d-94fa-cc22f744c736"). InnerVolumeSpecName "pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 04:16:17.804904 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.804869 4925 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-test-loki-lifecycle-secret\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.804904 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.804903 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-ca-bundle\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.805196 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.804921 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11b21fd1-a5a2-473d-94fa-cc22f744c736-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.805196 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.804935 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8q55\" (UniqueName: \"kubernetes.io/projected/11b21fd1-a5a2-473d-94fa-cc22f744c736-kube-api-access-q8q55\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.805196 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.804949 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-compactor-http\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.805196 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.804987 4925 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") on node \"ip-10-0-139-19.ec2.internal\" " Oct 30 04:16:17.805196 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.805001 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/11b21fd1-a5a2-473d-94fa-cc22f744c736-data-science-lokistack-compactor-grpc\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:16:17.814077 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.814026 4925 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e") on node "ip-10-0-139-19.ec2.internal" Oct 30 04:16:17.905519 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:17.905484 4925 reconciler_common.go:299] "Volume detached for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"csi-3faa7a4919429bfae9e5fe05bb9f7e440b0f7cbd5797a7f1cb9f75e405a5c758\"" Oct 30 04:16:18.161986 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:16:18.161950 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:16:18.453335 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:18.453257 4925 generic.go:356] "Generic (PLEG): container finished" podID="11b21fd1-a5a2-473d-94fa-cc22f744c736" containerID="1bdadd9dcc64718e42a9161efa5e25a423216c2d47d7603e3a85713cb876b0aa" exitCode=0 Oct 30 04:16:18.453335 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:18.453306 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 04:16:18.453923 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:18.453337 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"11b21fd1-a5a2-473d-94fa-cc22f744c736","Type":"ContainerDied","Data":"1bdadd9dcc64718e42a9161efa5e25a423216c2d47d7603e3a85713cb876b0aa"} Oct 30 04:16:18.453923 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:18.453392 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"11b21fd1-a5a2-473d-94fa-cc22f744c736","Type":"ContainerDied","Data":"49b00ff263bc0561d6320b05817577e6f5c13f22e0cc8a4860483beeed17c7d7"} Oct 30 04:16:18.453923 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:18.453422 4925 scope.go:122] "RemoveContainer" containerID="1bdadd9dcc64718e42a9161efa5e25a423216c2d47d7603e3a85713cb876b0aa" Oct 30 04:16:18.455539 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:18.455517 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" Oct 30 04:16:18.455626 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:18.455569 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n" event={"ID":"da4ac54e-413f-44a1-90cd-9be5dc4e133b","Type":"ContainerDied","Data":"62030aece98259667e080d2e2c2e88a9a2dab75f3eecdbedd55783b65ec39ac6"} Oct 30 04:16:18.462993 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:18.462976 4925 scope.go:122] "RemoveContainer" containerID="1bdadd9dcc64718e42a9161efa5e25a423216c2d47d7603e3a85713cb876b0aa" Oct 30 04:16:18.463232 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:16:18.463214 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bdadd9dcc64718e42a9161efa5e25a423216c2d47d7603e3a85713cb876b0aa\": container with ID starting with 1bdadd9dcc64718e42a9161efa5e25a423216c2d47d7603e3a85713cb876b0aa not found: ID does not exist" containerID="1bdadd9dcc64718e42a9161efa5e25a423216c2d47d7603e3a85713cb876b0aa" Oct 30 04:16:18.463304 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:18.463237 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bdadd9dcc64718e42a9161efa5e25a423216c2d47d7603e3a85713cb876b0aa"} err="failed to get container status \"1bdadd9dcc64718e42a9161efa5e25a423216c2d47d7603e3a85713cb876b0aa\": rpc error: code = NotFound desc = could not find container \"1bdadd9dcc64718e42a9161efa5e25a423216c2d47d7603e3a85713cb876b0aa\": container with ID starting with 1bdadd9dcc64718e42a9161efa5e25a423216c2d47d7603e3a85713cb876b0aa not found: ID does not exist" Oct 30 04:16:18.463304 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:18.463252 4925 scope.go:122] "RemoveContainer" containerID="83c7f4bd540e9b1ee5045bb2fe64d8d1c60dc300c6494bb888880bda690cc717" Oct 30 04:16:18.471066 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:18.471049 4925 scope.go:122] "RemoveContainer" containerID="93b5024cf705990b462f86463a273d164a7fb610dca1e79e068d6bb0c84c5694" Oct 30 04:16:18.505161 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:18.505128 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 04:16:18.517868 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:18.517824 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 04:16:18.544283 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:18.544251 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n"] Oct 30 04:16:18.548706 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:18.548681 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-gf65n"] Oct 30 04:16:19.119449 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:19.119416 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11b21fd1-a5a2-473d-94fa-cc22f744c736" path="/var/lib/kubelet/pods/11b21fd1-a5a2-473d-94fa-cc22f744c736/volumes" Oct 30 04:16:19.119846 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:19.119831 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c7de17d-0b8c-44ed-a9ca-e6472acf8e07" path="/var/lib/kubelet/pods/4c7de17d-0b8c-44ed-a9ca-e6472acf8e07/volumes" Oct 30 04:16:19.120155 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:19.120143 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64625ae8-1054-488c-b8c5-672c13d6ba05" path="/var/lib/kubelet/pods/64625ae8-1054-488c-b8c5-672c13d6ba05/volumes" Oct 30 04:16:19.120799 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:19.120785 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f6daa03-27bc-4159-b250-d845ee3e9625" path="/var/lib/kubelet/pods/6f6daa03-27bc-4159-b250-d845ee3e9625/volumes" Oct 30 04:16:19.121139 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:19.121127 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4ac54e-413f-44a1-90cd-9be5dc4e133b" path="/var/lib/kubelet/pods/da4ac54e-413f-44a1-90cd-9be5dc4e133b/volumes" Oct 30 04:16:48.161963 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:16:48.161923 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:16:49.980037 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:49.980004 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk"] Oct 30 04:16:49.980425 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:49.980317 4925 cpu_manager.go:414] "containerMap: removing container" podUID="64625ae8-1054-488c-b8c5-672c13d6ba05" containerName="otc-container" Oct 30 04:16:49.980425 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:49.980328 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64625ae8-1054-488c-b8c5-672c13d6ba05" containerName="otc-container" Oct 30 04:16:49.980425 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:49.980342 4925 cpu_manager.go:414] "containerMap: removing container" podUID="6f6daa03-27bc-4159-b250-d845ee3e9625" containerName="loki-query-frontend" Oct 30 04:16:49.980425 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:49.980349 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f6daa03-27bc-4159-b250-d845ee3e9625" containerName="loki-query-frontend" Oct 30 04:16:49.980425 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:49.980358 4925 cpu_manager.go:414] "containerMap: removing container" podUID="da4ac54e-413f-44a1-90cd-9be5dc4e133b" containerName="gateway" Oct 30 04:16:49.980425 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:49.980364 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4ac54e-413f-44a1-90cd-9be5dc4e133b" containerName="gateway" Oct 30 04:16:49.980425 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:49.980373 4925 cpu_manager.go:414] "containerMap: removing container" podUID="11b21fd1-a5a2-473d-94fa-cc22f744c736" containerName="loki-compactor" Oct 30 04:16:49.980425 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:49.980378 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b21fd1-a5a2-473d-94fa-cc22f744c736" containerName="loki-compactor" Oct 30 04:16:49.980425 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:49.980394 4925 cpu_manager.go:414] "containerMap: removing container" podUID="4c7de17d-0b8c-44ed-a9ca-e6472acf8e07" containerName="loki-ingester" Oct 30 04:16:49.980425 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:49.980398 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c7de17d-0b8c-44ed-a9ca-e6472acf8e07" containerName="loki-ingester" Oct 30 04:16:49.980425 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:49.980406 4925 cpu_manager.go:414] "containerMap: removing container" podUID="da4ac54e-413f-44a1-90cd-9be5dc4e133b" containerName="opa" Oct 30 04:16:49.980425 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:49.980411 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4ac54e-413f-44a1-90cd-9be5dc4e133b" containerName="opa" Oct 30 04:16:49.980824 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:49.980464 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c7de17d-0b8c-44ed-a9ca-e6472acf8e07" containerName="loki-ingester" Oct 30 04:16:49.980824 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:49.980472 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4ac54e-413f-44a1-90cd-9be5dc4e133b" containerName="opa" Oct 30 04:16:49.980824 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:49.980480 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="64625ae8-1054-488c-b8c5-672c13d6ba05" containerName="otc-container" Oct 30 04:16:49.980824 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:49.980486 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f6daa03-27bc-4159-b250-d845ee3e9625" containerName="loki-query-frontend" Oct 30 04:16:49.980824 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:49.980493 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4ac54e-413f-44a1-90cd-9be5dc4e133b" containerName="gateway" Oct 30 04:16:49.980824 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:49.980498 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b21fd1-a5a2-473d-94fa-cc22f744c736" containerName="loki-compactor" Oct 30 04:16:49.985282 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:49.985264 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:16:50.000850 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.000824 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 04:16:50.000850 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.000824 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 04:16:50.013426 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.013393 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 04:16:50.013597 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.013433 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 04:16:50.013597 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.013490 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-wkbdk\"" Oct 30 04:16:50.013739 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.013618 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 04:16:50.013841 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.013738 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 04:16:50.015356 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.015330 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk"] Oct 30 04:16:50.027470 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.027437 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9"] Oct 30 04:16:50.030994 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.030972 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" Oct 30 04:16:50.048198 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.048168 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9"] Oct 30 04:16:50.070394 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.070341 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/34e141d9-381b-43ed-8a09-869b3dc16e0a-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ks8pk\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:16:50.070628 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.070470 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cltwb\" (UniqueName: \"kubernetes.io/projected/34e141d9-381b-43ed-8a09-869b3dc16e0a-kube-api-access-cltwb\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ks8pk\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:16:50.070628 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.070613 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ks8pk\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:16:50.070785 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.070652 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ks8pk\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:16:50.070785 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.070716 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ks8pk\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:16:50.073793 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.073767 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv"] Oct 30 04:16:50.078152 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.078136 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" Oct 30 04:16:50.097550 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.097518 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv"] Oct 30 04:16:50.100981 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.100946 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 04:16:50.100981 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.100960 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 04:16:50.101290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.101000 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 04:16:50.101290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.101258 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 04:16:50.101290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.101272 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-pfbxj\"" Oct 30 04:16:50.101454 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.101304 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 04:16:50.101454 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.101412 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-c7qqk\"" Oct 30 04:16:50.118746 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.118713 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p"] Oct 30 04:16:50.122183 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.122160 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" Oct 30 04:16:50.143493 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.143465 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p"] Oct 30 04:16:50.172011 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.171972 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmqnb\" (UniqueName: \"kubernetes.io/projected/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-kube-api-access-vmqnb\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-f8hb9\" (UID: \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" Oct 30 04:16:50.172011 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.172012 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/34e141d9-381b-43ed-8a09-869b3dc16e0a-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ks8pk\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:16:50.172275 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.172144 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cltwb\" (UniqueName: \"kubernetes.io/projected/34e141d9-381b-43ed-8a09-869b3dc16e0a-kube-api-access-cltwb\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ks8pk\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:16:50.172330 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.172276 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f46907db-6f62-4d68-835b-3bf31401eb66-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-65lfv\" (UID: \"f46907db-6f62-4d68-835b-3bf31401eb66\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" Oct 30 04:16:50.172512 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.172494 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-f8hb9\" (UID: \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" Oct 30 04:16:50.172581 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.172516 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-f8hb9\" (UID: \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" Oct 30 04:16:50.172620 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.172605 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ks8pk\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:16:50.172654 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.172627 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ks8pk\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:16:50.172654 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.172645 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ks8pk\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:16:50.172729 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.172662 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-f8hb9\" (UID: \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" Oct 30 04:16:50.172729 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.172700 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-f8hb9\" (UID: \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" Oct 30 04:16:50.172875 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.172741 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f46907db-6f62-4d68-835b-3bf31401eb66-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-65lfv\" (UID: \"f46907db-6f62-4d68-835b-3bf31401eb66\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" Oct 30 04:16:50.172875 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.172811 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/34e141d9-381b-43ed-8a09-869b3dc16e0a-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ks8pk\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:16:50.172875 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:16:50.172822 4925 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 04:16:50.172875 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.172842 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xkp4\" (UniqueName: \"kubernetes.io/projected/f46907db-6f62-4d68-835b-3bf31401eb66-kube-api-access-9xkp4\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-65lfv\" (UID: \"f46907db-6f62-4d68-835b-3bf31401eb66\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" Oct 30 04:16:50.173054 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:16:50.172898 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-prometheus-client-cert podName:34e141d9-381b-43ed-8a09-869b3dc16e0a nodeName:}" failed. No retries permitted until 2026-10-30 04:16:50.672878246 +0000 UTC m=+5771.686156353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" (UID: "34e141d9-381b-43ed-8a09-869b3dc16e0a") : secret "prometheus-operated-tls" not found Oct 30 04:16:50.175036 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.175012 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ks8pk\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:16:50.175136 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.175120 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ks8pk\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:16:50.197472 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.197445 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cltwb\" (UniqueName: \"kubernetes.io/projected/34e141d9-381b-43ed-8a09-869b3dc16e0a-kube-api-access-cltwb\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ks8pk\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:16:50.200937 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.200910 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 04:16:50.201069 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.201002 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 04:16:50.201161 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.201148 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-qnwvs\"" Oct 30 04:16:50.274454 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.274357 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmqnb\" (UniqueName: \"kubernetes.io/projected/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-kube-api-access-vmqnb\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-f8hb9\" (UID: \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" Oct 30 04:16:50.274454 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.274423 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f46907db-6f62-4d68-835b-3bf31401eb66-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-65lfv\" (UID: \"f46907db-6f62-4d68-835b-3bf31401eb66\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" Oct 30 04:16:50.274713 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.274466 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-f8hb9\" (UID: \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" Oct 30 04:16:50.274713 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.274489 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-f8hb9\" (UID: \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" Oct 30 04:16:50.274713 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.274538 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/504573d1-3937-4775-a6d5-24320706eb55-config\") pod \"korrel8r-5cbcc89b8f-c7h2p\" (UID: \"504573d1-3937-4775-a6d5-24320706eb55\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" Oct 30 04:16:50.274713 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.274578 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-f8hb9\" (UID: \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" Oct 30 04:16:50.274944 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.274792 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-f8hb9\" (UID: \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" Oct 30 04:16:50.274944 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.274842 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f46907db-6f62-4d68-835b-3bf31401eb66-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-65lfv\" (UID: \"f46907db-6f62-4d68-835b-3bf31401eb66\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" Oct 30 04:16:50.274944 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.274872 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/504573d1-3937-4775-a6d5-24320706eb55-tls\") pod \"korrel8r-5cbcc89b8f-c7h2p\" (UID: \"504573d1-3937-4775-a6d5-24320706eb55\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" Oct 30 04:16:50.274944 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.274902 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sblcm\" (UniqueName: \"kubernetes.io/projected/504573d1-3937-4775-a6d5-24320706eb55-kube-api-access-sblcm\") pod \"korrel8r-5cbcc89b8f-c7h2p\" (UID: \"504573d1-3937-4775-a6d5-24320706eb55\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" Oct 30 04:16:50.274944 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.274934 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xkp4\" (UniqueName: \"kubernetes.io/projected/f46907db-6f62-4d68-835b-3bf31401eb66-kube-api-access-9xkp4\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-65lfv\" (UID: \"f46907db-6f62-4d68-835b-3bf31401eb66\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" Oct 30 04:16:50.275363 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.275342 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-f8hb9\" (UID: \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" Oct 30 04:16:50.275466 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.275391 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-f8hb9\" (UID: \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" Oct 30 04:16:50.275466 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.275423 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-f8hb9\" (UID: \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" Oct 30 04:16:50.275603 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.275578 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f46907db-6f62-4d68-835b-3bf31401eb66-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-65lfv\" (UID: \"f46907db-6f62-4d68-835b-3bf31401eb66\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" Oct 30 04:16:50.277169 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.277146 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-f8hb9\" (UID: \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" Oct 30 04:16:50.277265 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.277190 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f46907db-6f62-4d68-835b-3bf31401eb66-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-65lfv\" (UID: \"f46907db-6f62-4d68-835b-3bf31401eb66\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" Oct 30 04:16:50.301223 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.301193 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmqnb\" (UniqueName: \"kubernetes.io/projected/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-kube-api-access-vmqnb\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-f8hb9\" (UID: \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" Oct 30 04:16:50.301361 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.301230 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xkp4\" (UniqueName: \"kubernetes.io/projected/f46907db-6f62-4d68-835b-3bf31401eb66-kube-api-access-9xkp4\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-65lfv\" (UID: \"f46907db-6f62-4d68-835b-3bf31401eb66\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" Oct 30 04:16:50.346716 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.346681 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" Oct 30 04:16:50.375431 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.375392 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/504573d1-3937-4775-a6d5-24320706eb55-config\") pod \"korrel8r-5cbcc89b8f-c7h2p\" (UID: \"504573d1-3937-4775-a6d5-24320706eb55\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" Oct 30 04:16:50.375598 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.375454 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/504573d1-3937-4775-a6d5-24320706eb55-tls\") pod \"korrel8r-5cbcc89b8f-c7h2p\" (UID: \"504573d1-3937-4775-a6d5-24320706eb55\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" Oct 30 04:16:50.375598 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.375472 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sblcm\" (UniqueName: \"kubernetes.io/projected/504573d1-3937-4775-a6d5-24320706eb55-kube-api-access-sblcm\") pod \"korrel8r-5cbcc89b8f-c7h2p\" (UID: \"504573d1-3937-4775-a6d5-24320706eb55\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" Oct 30 04:16:50.376250 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.376226 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/504573d1-3937-4775-a6d5-24320706eb55-config\") pod \"korrel8r-5cbcc89b8f-c7h2p\" (UID: \"504573d1-3937-4775-a6d5-24320706eb55\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" Oct 30 04:16:50.377868 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.377850 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/504573d1-3937-4775-a6d5-24320706eb55-tls\") pod \"korrel8r-5cbcc89b8f-c7h2p\" (UID: \"504573d1-3937-4775-a6d5-24320706eb55\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" Oct 30 04:16:50.393794 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.393762 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" Oct 30 04:16:50.402841 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.402815 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sblcm\" (UniqueName: \"kubernetes.io/projected/504573d1-3937-4775-a6d5-24320706eb55-kube-api-access-sblcm\") pod \"korrel8r-5cbcc89b8f-c7h2p\" (UID: \"504573d1-3937-4775-a6d5-24320706eb55\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" Oct 30 04:16:50.434903 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.434869 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" Oct 30 04:16:50.653798 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.653267 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9"] Oct 30 04:16:50.657993 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:16:50.657949 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1762351e_ba22_436d_aed8_7fe7a6e5cc4a.slice/crio-15a6735c08fb497d59340ce04cb7de7a7289cc10f3a23be908b241b76bd89ff4 WatchSource:0}: Error finding container 15a6735c08fb497d59340ce04cb7de7a7289cc10f3a23be908b241b76bd89ff4: Status 404 returned error can't find the container with id 15a6735c08fb497d59340ce04cb7de7a7289cc10f3a23be908b241b76bd89ff4 Oct 30 04:16:50.678635 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.678602 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ks8pk\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:16:50.681165 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.681134 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-ks8pk\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:16:50.707517 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.706061 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv"] Oct 30 04:16:50.710923 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:16:50.710898 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46907db_6f62_4d68_835b_3bf31401eb66.slice/crio-d6bae1bef6d0ce0095a1cba6c732dcae39f16b376acc0e7de63196379bb5a03a WatchSource:0}: Error finding container d6bae1bef6d0ce0095a1cba6c732dcae39f16b376acc0e7de63196379bb5a03a: Status 404 returned error can't find the container with id d6bae1bef6d0ce0095a1cba6c732dcae39f16b376acc0e7de63196379bb5a03a Oct 30 04:16:50.751690 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.751662 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p"] Oct 30 04:16:50.754880 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:16:50.754855 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504573d1_3937_4775_a6d5_24320706eb55.slice/crio-4e72461d43701a4fc3d204354bbd7c37a1bffea6abbe3dda2be9e29033653eec WatchSource:0}: Error finding container 4e72461d43701a4fc3d204354bbd7c37a1bffea6abbe3dda2be9e29033653eec: Status 404 returned error can't find the container with id 4e72461d43701a4fc3d204354bbd7c37a1bffea6abbe3dda2be9e29033653eec Oct 30 04:16:50.898254 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:50.898216 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:16:51.208545 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:51.208516 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk"] Oct 30 04:16:51.211227 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:16:51.211186 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e141d9_381b_43ed_8a09_869b3dc16e0a.slice/crio-58bce57734608b99182810b6310807cb198c9312576a132829157f579c2eda35 WatchSource:0}: Error finding container 58bce57734608b99182810b6310807cb198c9312576a132829157f579c2eda35: Status 404 returned error can't find the container with id 58bce57734608b99182810b6310807cb198c9312576a132829157f579c2eda35 Oct 30 04:16:51.614195 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:51.614145 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" event={"ID":"504573d1-3937-4775-a6d5-24320706eb55","Type":"ContainerStarted","Data":"f3109690ebd68068ff5617203f9fb1d4d919e094d459278534f55f1cd7ee2cee"} Oct 30 04:16:51.614455 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:51.614221 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" Oct 30 04:16:51.614455 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:51.614247 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" event={"ID":"504573d1-3937-4775-a6d5-24320706eb55","Type":"ContainerStarted","Data":"4e72461d43701a4fc3d204354bbd7c37a1bffea6abbe3dda2be9e29033653eec"} Oct 30 04:16:51.616972 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:51.616937 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" event={"ID":"f46907db-6f62-4d68-835b-3bf31401eb66","Type":"ContainerStarted","Data":"a9c36577dadf33db32c5c29e6a9753a081c474048b29c93ec56c74b69f04d993"} Oct 30 04:16:51.617082 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:51.617003 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" event={"ID":"f46907db-6f62-4d68-835b-3bf31401eb66","Type":"ContainerStarted","Data":"f57c9a3da461cec50631b1ea47fc952d94b4a1a4b7a479f8a4e34f1c7dd17aab"} Oct 30 04:16:51.617082 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:51.617034 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" event={"ID":"f46907db-6f62-4d68-835b-3bf31401eb66","Type":"ContainerStarted","Data":"d6bae1bef6d0ce0095a1cba6c732dcae39f16b376acc0e7de63196379bb5a03a"} Oct 30 04:16:51.619369 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:51.619208 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" event={"ID":"1762351e-ba22-436d-aed8-7fe7a6e5cc4a","Type":"ContainerStarted","Data":"ea15d210e8822d275465d38867d7f86c518e56b94aeff10f0764bebf0c1c6c65"} Oct 30 04:16:51.619369 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:51.619270 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" event={"ID":"1762351e-ba22-436d-aed8-7fe7a6e5cc4a","Type":"ContainerStarted","Data":"635c0539792283f91524def775fceecd2aa47a77e68af45d3b39c88073fefc26"} Oct 30 04:16:51.619369 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:51.619302 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" event={"ID":"1762351e-ba22-436d-aed8-7fe7a6e5cc4a","Type":"ContainerStarted","Data":"15a6735c08fb497d59340ce04cb7de7a7289cc10f3a23be908b241b76bd89ff4"} Oct 30 04:16:51.621066 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:51.621037 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" event={"ID":"34e141d9-381b-43ed-8a09-869b3dc16e0a","Type":"ContainerStarted","Data":"58bce57734608b99182810b6310807cb198c9312576a132829157f579c2eda35"} Oct 30 04:16:51.683474 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:51.682558 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" podStartSLOduration=1.682538572 podStartE2EDuration="1.682538572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:51.661203244 +0000 UTC m=+5772.674481349" watchObservedRunningTime="2026-10-30 04:16:51.682538572 +0000 UTC m=+5772.695816677" Oct 30 04:16:51.738535 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:51.738385 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" podStartSLOduration=1.738370446 podStartE2EDuration="1.738370446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:51.737157157 +0000 UTC m=+5772.750435261" watchObservedRunningTime="2026-10-30 04:16:51.738370446 +0000 UTC m=+5772.751648549" Oct 30 04:16:51.785135 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:51.784656 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" podStartSLOduration=2.784637037 podStartE2EDuration="2.784637037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:51.78293044 +0000 UTC m=+5772.796208545" watchObservedRunningTime="2026-10-30 04:16:51.784637037 +0000 UTC m=+5772.797915141" Oct 30 04:16:52.626992 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:52.626884 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" event={"ID":"34e141d9-381b-43ed-8a09-869b3dc16e0a","Type":"ContainerStarted","Data":"a070a7c2b6e0b55034cc0e20ce4ceb43b7ede343bb51cbc90c0e59ab380c7399"} Oct 30 04:16:52.626992 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:52.626969 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:16:52.672639 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:52.672588 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" podStartSLOduration=3.672575812 podStartE2EDuration="3.672575812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:52.67143905 +0000 UTC m=+5773.684717153" watchObservedRunningTime="2026-10-30 04:16:52.672575812 +0000 UTC m=+5773.685853915" Oct 30 04:16:55.328907 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.328870 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:16:55.332803 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.332782 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.363238 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.363207 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:16:55.418662 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.418625 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 04:16:55.419016 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.418995 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 04:16:55.419016 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.418995 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:16:55.419169 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.419032 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 04:16:55.419169 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.419092 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-5vht9\"" Oct 30 04:16:55.420432 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.420411 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.420487 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.420446 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12802aaa-666e-41ee-9f1c-25700d52148c-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.420538 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.420525 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.420574 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.420560 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.420622 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.420584 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12802aaa-666e-41ee-9f1c-25700d52148c-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.420622 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.420601 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/12802aaa-666e-41ee-9f1c-25700d52148c-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.420723 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.420702 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8zgf\" (UniqueName: \"kubernetes.io/projected/12802aaa-666e-41ee-9f1c-25700d52148c-kube-api-access-c8zgf\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.521141 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.521096 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.521141 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.521143 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12802aaa-666e-41ee-9f1c-25700d52148c-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.521395 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.521161 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/12802aaa-666e-41ee-9f1c-25700d52148c-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.521395 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.521364 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8zgf\" (UniqueName: \"kubernetes.io/projected/12802aaa-666e-41ee-9f1c-25700d52148c-kube-api-access-c8zgf\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.521521 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.521463 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.521521 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.521500 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12802aaa-666e-41ee-9f1c-25700d52148c-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.521631 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.521573 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.521691 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.521661 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/12802aaa-666e-41ee-9f1c-25700d52148c-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.523734 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.523680 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.523924 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.523907 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.524033 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.524013 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12802aaa-666e-41ee-9f1c-25700d52148c-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.524071 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.524037 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.524125 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.524108 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12802aaa-666e-41ee-9f1c-25700d52148c-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.544957 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.544932 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8zgf\" (UniqueName: \"kubernetes.io/projected/12802aaa-666e-41ee-9f1c-25700d52148c-kube-api-access-c8zgf\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.652510 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.652420 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:16:55.987494 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:55.987384 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:16:55.995887 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:16:55.995854 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12802aaa_666e_41ee_9f1c_25700d52148c.slice/crio-9c52d3768d22812e3cc6c1fe09737055c6769a6aeda0f87777f2a2aeb262a1c9 WatchSource:0}: Error finding container 9c52d3768d22812e3cc6c1fe09737055c6769a6aeda0f87777f2a2aeb262a1c9: Status 404 returned error can't find the container with id 9c52d3768d22812e3cc6c1fe09737055c6769a6aeda0f87777f2a2aeb262a1c9 Oct 30 04:16:56.649115 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:56.649072 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"12802aaa-666e-41ee-9f1c-25700d52148c","Type":"ContainerStarted","Data":"4325235cb55a43cc9c9bf88578aa1b582601c21d756ab36e197cdb564197ae8d"} Oct 30 04:16:56.649511 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:56.649138 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"12802aaa-666e-41ee-9f1c-25700d52148c","Type":"ContainerStarted","Data":"9c52d3768d22812e3cc6c1fe09737055c6769a6aeda0f87777f2a2aeb262a1c9"} Oct 30 04:16:58.634484 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:16:58.634450 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:17:00.669528 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:00.669493 4925 generic.go:356] "Generic (PLEG): container finished" podID="12802aaa-666e-41ee-9f1c-25700d52148c" containerID="4325235cb55a43cc9c9bf88578aa1b582601c21d756ab36e197cdb564197ae8d" exitCode=0 Oct 30 04:17:00.670054 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:00.669547 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"12802aaa-666e-41ee-9f1c-25700d52148c","Type":"ContainerDied","Data":"4325235cb55a43cc9c9bf88578aa1b582601c21d756ab36e197cdb564197ae8d"} Oct 30 04:17:01.677144 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:01.677096 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"12802aaa-666e-41ee-9f1c-25700d52148c","Type":"ContainerStarted","Data":"72930da9ed4e0eefb874b4db3a6555333a94f021e10114909cd172cb83bedbdf"} Oct 30 04:17:01.677555 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:01.677172 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"12802aaa-666e-41ee-9f1c-25700d52148c","Type":"ContainerStarted","Data":"22313c2ac466d574abfb1af42c01cdc3297eb67095ac581062bb58cc6f12a6d3"} Oct 30 04:17:01.677555 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:01.677371 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:17:01.753134 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:01.753059 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.068003896 podStartE2EDuration="6.753040364s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.685036468s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:01.738452767 +0000 UTC m=+5782.751730884" watchObservedRunningTime="2026-10-30 04:17:01.753040364 +0000 UTC m=+5782.766318468" Oct 30 04:17:02.629814 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:02.629776 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" Oct 30 04:17:07.684072 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:07.684041 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:17:18.161709 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:17:18.161671 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:17:30.344906 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.344828 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:17:30.348357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.348339 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:30.375962 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.375936 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:17:30.376141 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.376103 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:17:30.376191 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.376138 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:17:30.376232 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.376190 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-m9xwj\"" Oct 30 04:17:30.388572 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.388543 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:17:30.390397 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.390371 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:17:30.416627 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.416601 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzthc\" (UniqueName: \"kubernetes.io/projected/ec756d88-4fcf-47c3-81c3-e536a9e2725b-kube-api-access-pzthc\") pod \"data-science-collector-collector-1\" (UID: \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:30.416775 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.416646 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:30.416775 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.416668 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:30.416775 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.416737 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:30.416913 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.416861 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ec756d88-4fcf-47c3-81c3-e536a9e2725b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:30.517287 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.517248 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ec756d88-4fcf-47c3-81c3-e536a9e2725b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:30.517479 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.517306 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzthc\" (UniqueName: \"kubernetes.io/projected/ec756d88-4fcf-47c3-81c3-e536a9e2725b-kube-api-access-pzthc\") pod \"data-science-collector-collector-1\" (UID: \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:30.517479 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.517341 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:30.517479 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.517365 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:30.517479 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.517388 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:30.518080 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.518053 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ec756d88-4fcf-47c3-81c3-e536a9e2725b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:30.519802 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.519778 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:30.519900 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.519830 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:30.519946 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.519935 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:30.540876 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.540855 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzthc\" (UniqueName: \"kubernetes.io/projected/ec756d88-4fcf-47c3-81c3-e536a9e2725b-kube-api-access-pzthc\") pod \"data-science-collector-collector-1\" (UID: \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:30.662395 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.662302 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:30.965344 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.965315 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:17:30.968060 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:17:30.968037 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec756d88_4fcf_47c3_81c3_e536a9e2725b.slice/crio-77fb91108e1c3e8e20068303cd83baf7c82ff977b7f86821d888afc68b287a63 WatchSource:0}: Error finding container 77fb91108e1c3e8e20068303cd83baf7c82ff977b7f86821d888afc68b287a63: Status 404 returned error can't find the container with id 77fb91108e1c3e8e20068303cd83baf7c82ff977b7f86821d888afc68b287a63 Oct 30 04:17:30.970229 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.970193 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:17:30.970334 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:30.970264 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:17:31.819220 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:31.819179 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ec756d88-4fcf-47c3-81c3-e536a9e2725b","Type":"ContainerStarted","Data":"d5c7c79c83f5f3211517b87bfd53b52e3e422807ae2c71c0f20a78cf4407ca16"} Oct 30 04:17:31.819617 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:31.819247 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ec756d88-4fcf-47c3-81c3-e536a9e2725b","Type":"ContainerStarted","Data":"77fb91108e1c3e8e20068303cd83baf7c82ff977b7f86821d888afc68b287a63"} Oct 30 04:17:31.865233 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:31.865181 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.8651678280000001 podStartE2EDuration="1.865167828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:31.863630629 +0000 UTC m=+5812.876908731" watchObservedRunningTime="2026-10-30 04:17:31.865167828 +0000 UTC m=+5812.878445928" Oct 30 04:17:35.433675 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.433639 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:17:35.434182 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.433885 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="ec756d88-4fcf-47c3-81c3-e536a9e2725b" containerName="otc-container" containerID="cri-o://d5c7c79c83f5f3211517b87bfd53b52e3e422807ae2c71c0f20a78cf4407ca16" gracePeriod=30 Oct 30 04:17:35.688066 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.688004 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:35.766628 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.766595 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-monitoring-tls-certs\") pod \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\" (UID: \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\") " Oct 30 04:17:35.766851 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.766636 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec756d88-4fcf-47c3-81c3-e536a9e2725b-kube-api-access-pzthc\" (UniqueName: \"kubernetes.io/projected/ec756d88-4fcf-47c3-81c3-e536a9e2725b-kube-api-access-pzthc\") pod \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\" (UID: \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\") " Oct 30 04:17:35.766851 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.766666 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-tls-certs\") pod \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\" (UID: \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\") " Oct 30 04:17:35.766851 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.766695 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec756d88-4fcf-47c3-81c3-e536a9e2725b-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ec756d88-4fcf-47c3-81c3-e536a9e2725b-otc-internal\") pod \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\" (UID: \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\") " Oct 30 04:17:35.766851 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.766720 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-data-science-collector-ta-client-cert\") pod \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\" (UID: \"ec756d88-4fcf-47c3-81c3-e536a9e2725b\") " Oct 30 04:17:35.767126 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.767105 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec756d88-4fcf-47c3-81c3-e536a9e2725b-otc-internal" pod "ec756d88-4fcf-47c3-81c3-e536a9e2725b" (UID: "ec756d88-4fcf-47c3-81c3-e536a9e2725b"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:35.768943 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.768909 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-monitoring-tls-certs" pod "ec756d88-4fcf-47c3-81c3-e536a9e2725b" (UID: "ec756d88-4fcf-47c3-81c3-e536a9e2725b"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:17:35.769058 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.768984 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-data-science-collector-ta-client-cert" pod "ec756d88-4fcf-47c3-81c3-e536a9e2725b" (UID: "ec756d88-4fcf-47c3-81c3-e536a9e2725b"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:17:35.769058 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.768996 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec756d88-4fcf-47c3-81c3-e536a9e2725b-kube-api-access-pzthc" pod "ec756d88-4fcf-47c3-81c3-e536a9e2725b" (UID: "ec756d88-4fcf-47c3-81c3-e536a9e2725b"). InnerVolumeSpecName "kube-api-access-pzthc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:35.769142 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.769084 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-tls-certs" pod "ec756d88-4fcf-47c3-81c3-e536a9e2725b" (UID: "ec756d88-4fcf-47c3-81c3-e536a9e2725b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:17:35.839876 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.839839 4925 generic.go:356] "Generic (PLEG): container finished" podID="ec756d88-4fcf-47c3-81c3-e536a9e2725b" containerID="d5c7c79c83f5f3211517b87bfd53b52e3e422807ae2c71c0f20a78cf4407ca16" exitCode=0 Oct 30 04:17:35.840081 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.839899 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:35.840081 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.839901 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ec756d88-4fcf-47c3-81c3-e536a9e2725b","Type":"ContainerDied","Data":"d5c7c79c83f5f3211517b87bfd53b52e3e422807ae2c71c0f20a78cf4407ca16"} Oct 30 04:17:35.840081 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.840019 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ec756d88-4fcf-47c3-81c3-e536a9e2725b","Type":"ContainerDied","Data":"77fb91108e1c3e8e20068303cd83baf7c82ff977b7f86821d888afc68b287a63"} Oct 30 04:17:35.840081 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.840041 4925 scope.go:122] "RemoveContainer" containerID="d5c7c79c83f5f3211517b87bfd53b52e3e422807ae2c71c0f20a78cf4407ca16" Oct 30 04:17:35.850066 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.850042 4925 scope.go:122] "RemoveContainer" containerID="d5c7c79c83f5f3211517b87bfd53b52e3e422807ae2c71c0f20a78cf4407ca16" Oct 30 04:17:35.850347 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:17:35.850317 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c7c79c83f5f3211517b87bfd53b52e3e422807ae2c71c0f20a78cf4407ca16\": container with ID starting with d5c7c79c83f5f3211517b87bfd53b52e3e422807ae2c71c0f20a78cf4407ca16 not found: ID does not exist" containerID="d5c7c79c83f5f3211517b87bfd53b52e3e422807ae2c71c0f20a78cf4407ca16" Oct 30 04:17:35.850420 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.850348 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c7c79c83f5f3211517b87bfd53b52e3e422807ae2c71c0f20a78cf4407ca16"} err="failed to get container status \"d5c7c79c83f5f3211517b87bfd53b52e3e422807ae2c71c0f20a78cf4407ca16\": rpc error: code = NotFound desc = could not find container \"d5c7c79c83f5f3211517b87bfd53b52e3e422807ae2c71c0f20a78cf4407ca16\": container with ID starting with d5c7c79c83f5f3211517b87bfd53b52e3e422807ae2c71c0f20a78cf4407ca16 not found: ID does not exist" Oct 30 04:17:35.867899 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.867868 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzthc\" (UniqueName: \"kubernetes.io/projected/ec756d88-4fcf-47c3-81c3-e536a9e2725b-kube-api-access-pzthc\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:17:35.867899 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.867899 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:17:35.868037 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.867910 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ec756d88-4fcf-47c3-81c3-e536a9e2725b-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:17:35.868037 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.867919 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:17:35.868037 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.867929 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec756d88-4fcf-47c3-81c3-e536a9e2725b-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:17:35.892412 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.892377 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:17:35.895985 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.895959 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:17:35.934974 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.934939 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:17:35.935346 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.935331 4925 cpu_manager.go:414] "containerMap: removing container" podUID="ec756d88-4fcf-47c3-81c3-e536a9e2725b" containerName="otc-container" Oct 30 04:17:35.935394 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.935348 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec756d88-4fcf-47c3-81c3-e536a9e2725b" containerName="otc-container" Oct 30 04:17:35.935429 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.935416 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec756d88-4fcf-47c3-81c3-e536a9e2725b" containerName="otc-container" Oct 30 04:17:35.940008 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.939952 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:35.958137 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.958063 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:17:35.958312 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.958140 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-m9xwj\"" Oct 30 04:17:35.958312 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.958214 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:17:35.958429 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.958316 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:17:35.958429 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.958339 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:17:35.960796 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:35.960731 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:17:36.069858 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.069797 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"664a4b10-dd85-4d60-8521-71a51d9d3728\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:36.070055 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.069938 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"664a4b10-dd85-4d60-8521-71a51d9d3728\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:36.070055 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.069976 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvsvx\" (UniqueName: \"kubernetes.io/projected/664a4b10-dd85-4d60-8521-71a51d9d3728-kube-api-access-nvsvx\") pod \"data-science-collector-collector-1\" (UID: \"664a4b10-dd85-4d60-8521-71a51d9d3728\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:36.070156 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.070057 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/664a4b10-dd85-4d60-8521-71a51d9d3728-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"664a4b10-dd85-4d60-8521-71a51d9d3728\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:36.070156 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.070085 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"664a4b10-dd85-4d60-8521-71a51d9d3728\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:36.171132 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.171090 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"664a4b10-dd85-4d60-8521-71a51d9d3728\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:36.171132 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.171134 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvsvx\" (UniqueName: \"kubernetes.io/projected/664a4b10-dd85-4d60-8521-71a51d9d3728-kube-api-access-nvsvx\") pod \"data-science-collector-collector-1\" (UID: \"664a4b10-dd85-4d60-8521-71a51d9d3728\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:36.171435 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.171154 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/664a4b10-dd85-4d60-8521-71a51d9d3728-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"664a4b10-dd85-4d60-8521-71a51d9d3728\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:36.171435 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.171176 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"664a4b10-dd85-4d60-8521-71a51d9d3728\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:36.171435 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.171282 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"664a4b10-dd85-4d60-8521-71a51d9d3728\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:36.172194 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.172155 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/664a4b10-dd85-4d60-8521-71a51d9d3728-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"664a4b10-dd85-4d60-8521-71a51d9d3728\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:36.173843 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.173819 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"664a4b10-dd85-4d60-8521-71a51d9d3728\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:36.173933 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.173879 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"664a4b10-dd85-4d60-8521-71a51d9d3728\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:36.174012 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.173996 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"664a4b10-dd85-4d60-8521-71a51d9d3728\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:36.195134 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.195084 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvsvx\" (UniqueName: \"kubernetes.io/projected/664a4b10-dd85-4d60-8521-71a51d9d3728-kube-api-access-nvsvx\") pod \"data-science-collector-collector-1\" (UID: \"664a4b10-dd85-4d60-8521-71a51d9d3728\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:36.252770 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.252724 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:17:36.552711 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.552684 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:17:36.554908 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:17:36.554874 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664a4b10_dd85_4d60_8521_71a51d9d3728.slice/crio-a1471d12c141cbb25f2019751f7d964d87e10241278dce44922404bc27240643 WatchSource:0}: Error finding container a1471d12c141cbb25f2019751f7d964d87e10241278dce44922404bc27240643: Status 404 returned error can't find the container with id a1471d12c141cbb25f2019751f7d964d87e10241278dce44922404bc27240643 Oct 30 04:17:36.557171 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.557137 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:17:36.557288 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.557206 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:17:36.853847 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.853802 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"664a4b10-dd85-4d60-8521-71a51d9d3728","Type":"ContainerStarted","Data":"17ad5a952ef9430ffaf7425d998c43a7ca16a812e6ac60ae05c52e841d253a85"} Oct 30 04:17:36.854022 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.853900 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"664a4b10-dd85-4d60-8521-71a51d9d3728","Type":"ContainerStarted","Data":"a1471d12c141cbb25f2019751f7d964d87e10241278dce44922404bc27240643"} Oct 30 04:17:36.896566 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:36.896519 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.896507541 podStartE2EDuration="1.896507541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:36.896307867 +0000 UTC m=+5817.909585982" watchObservedRunningTime="2026-10-30 04:17:36.896507541 +0000 UTC m=+5817.909785643" Oct 30 04:17:37.119991 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:17:37.119910 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec756d88-4fcf-47c3-81c3-e536a9e2725b" path="/var/lib/kubelet/pods/ec756d88-4fcf-47c3-81c3-e536a9e2725b/volumes" Oct 30 04:17:48.161877 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:17:48.161842 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:18:01.596405 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:01.596370 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:18:01.597004 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:01.596563 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="664a4b10-dd85-4d60-8521-71a51d9d3728" containerName="otc-container" containerID="cri-o://17ad5a952ef9430ffaf7425d998c43a7ca16a812e6ac60ae05c52e841d253a85" gracePeriod=30 Oct 30 04:18:01.855323 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:01.855252 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:01.979920 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:01.979889 4925 generic.go:356] "Generic (PLEG): container finished" podID="664a4b10-dd85-4d60-8521-71a51d9d3728" containerID="17ad5a952ef9430ffaf7425d998c43a7ca16a812e6ac60ae05c52e841d253a85" exitCode=0 Oct 30 04:18:01.980099 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:01.979943 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:01.980099 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:01.979977 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"664a4b10-dd85-4d60-8521-71a51d9d3728","Type":"ContainerDied","Data":"17ad5a952ef9430ffaf7425d998c43a7ca16a812e6ac60ae05c52e841d253a85"} Oct 30 04:18:01.980099 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:01.980031 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"664a4b10-dd85-4d60-8521-71a51d9d3728","Type":"ContainerDied","Data":"a1471d12c141cbb25f2019751f7d964d87e10241278dce44922404bc27240643"} Oct 30 04:18:01.980099 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:01.980053 4925 scope.go:122] "RemoveContainer" containerID="17ad5a952ef9430ffaf7425d998c43a7ca16a812e6ac60ae05c52e841d253a85" Oct 30 04:18:01.988513 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:01.988496 4925 scope.go:122] "RemoveContainer" containerID="17ad5a952ef9430ffaf7425d998c43a7ca16a812e6ac60ae05c52e841d253a85" Oct 30 04:18:01.988791 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:18:01.988771 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ad5a952ef9430ffaf7425d998c43a7ca16a812e6ac60ae05c52e841d253a85\": container with ID starting with 17ad5a952ef9430ffaf7425d998c43a7ca16a812e6ac60ae05c52e841d253a85 not found: ID does not exist" containerID="17ad5a952ef9430ffaf7425d998c43a7ca16a812e6ac60ae05c52e841d253a85" Oct 30 04:18:01.988885 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:01.988801 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ad5a952ef9430ffaf7425d998c43a7ca16a812e6ac60ae05c52e841d253a85"} err="failed to get container status \"17ad5a952ef9430ffaf7425d998c43a7ca16a812e6ac60ae05c52e841d253a85\": rpc error: code = NotFound desc = could not find container \"17ad5a952ef9430ffaf7425d998c43a7ca16a812e6ac60ae05c52e841d253a85\": container with ID starting with 17ad5a952ef9430ffaf7425d998c43a7ca16a812e6ac60ae05c52e841d253a85 not found: ID does not exist" Oct 30 04:18:01.996133 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:01.996108 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-monitoring-tls-certs\") pod \"664a4b10-dd85-4d60-8521-71a51d9d3728\" (UID: \"664a4b10-dd85-4d60-8521-71a51d9d3728\") " Oct 30 04:18:01.996247 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:01.996161 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-tls-certs\" (UniqueName: \"kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-tls-certs\") pod \"664a4b10-dd85-4d60-8521-71a51d9d3728\" (UID: \"664a4b10-dd85-4d60-8521-71a51d9d3728\") " Oct 30 04:18:01.996247 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:01.996233 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-data-science-collector-ta-client-cert\") pod \"664a4b10-dd85-4d60-8521-71a51d9d3728\" (UID: \"664a4b10-dd85-4d60-8521-71a51d9d3728\") " Oct 30 04:18:01.996359 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:01.996271 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/664a4b10-dd85-4d60-8521-71a51d9d3728-otc-internal\" (UniqueName: \"kubernetes.io/configmap/664a4b10-dd85-4d60-8521-71a51d9d3728-otc-internal\") pod \"664a4b10-dd85-4d60-8521-71a51d9d3728\" (UID: \"664a4b10-dd85-4d60-8521-71a51d9d3728\") " Oct 30 04:18:01.996359 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:01.996315 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/664a4b10-dd85-4d60-8521-71a51d9d3728-kube-api-access-nvsvx\" (UniqueName: \"kubernetes.io/projected/664a4b10-dd85-4d60-8521-71a51d9d3728-kube-api-access-nvsvx\") pod \"664a4b10-dd85-4d60-8521-71a51d9d3728\" (UID: \"664a4b10-dd85-4d60-8521-71a51d9d3728\") " Oct 30 04:18:01.996688 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:01.996660 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/664a4b10-dd85-4d60-8521-71a51d9d3728-otc-internal" pod "664a4b10-dd85-4d60-8521-71a51d9d3728" (UID: "664a4b10-dd85-4d60-8521-71a51d9d3728"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:18:01.998261 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:01.998239 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-monitoring-tls-certs" pod "664a4b10-dd85-4d60-8521-71a51d9d3728" (UID: "664a4b10-dd85-4d60-8521-71a51d9d3728"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:01.998638 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:01.998622 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-data-science-collector-ta-client-cert" pod "664a4b10-dd85-4d60-8521-71a51d9d3728" (UID: "664a4b10-dd85-4d60-8521-71a51d9d3728"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:01.998801 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:01.998776 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/664a4b10-dd85-4d60-8521-71a51d9d3728-kube-api-access-nvsvx" pod "664a4b10-dd85-4d60-8521-71a51d9d3728" (UID: "664a4b10-dd85-4d60-8521-71a51d9d3728"). InnerVolumeSpecName "kube-api-access-nvsvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:01.998896 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:01.998833 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-tls-certs" pod "664a4b10-dd85-4d60-8521-71a51d9d3728" (UID: "664a4b10-dd85-4d60-8521-71a51d9d3728"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:02.097686 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.097649 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvsvx\" (UniqueName: \"kubernetes.io/projected/664a4b10-dd85-4d60-8521-71a51d9d3728-kube-api-access-nvsvx\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:02.097686 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.097682 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:02.097686 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.097691 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:02.097686 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.097702 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/664a4b10-dd85-4d60-8521-71a51d9d3728-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:02.098010 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.097711 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/664a4b10-dd85-4d60-8521-71a51d9d3728-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:02.327077 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.327044 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:18:02.331686 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.331656 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:18:02.368527 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.368487 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:18:02.368969 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.368951 4925 cpu_manager.go:414] "containerMap: removing container" podUID="664a4b10-dd85-4d60-8521-71a51d9d3728" containerName="otc-container" Oct 30 04:18:02.369075 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.368971 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="664a4b10-dd85-4d60-8521-71a51d9d3728" containerName="otc-container" Oct 30 04:18:02.369075 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.369068 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="664a4b10-dd85-4d60-8521-71a51d9d3728" containerName="otc-container" Oct 30 04:18:02.372280 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.372257 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:02.388025 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.388000 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:18:02.388025 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.388028 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:18:02.388025 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.388041 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-m9xwj\"" Oct 30 04:18:02.388025 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.388032 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:18:02.388244 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.388088 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:18:02.394395 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.394372 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:18:02.501847 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.501811 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:02.502026 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.501862 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/19c5b22f-f46d-424e-9a12-9293f0a3e74d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:02.502026 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.501977 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dflpj\" (UniqueName: \"kubernetes.io/projected/19c5b22f-f46d-424e-9a12-9293f0a3e74d-kube-api-access-dflpj\") pod \"data-science-collector-collector-1\" (UID: \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:02.502102 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.502044 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:02.502102 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.502070 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:02.603275 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.603236 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/19c5b22f-f46d-424e-9a12-9293f0a3e74d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:02.603767 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.603282 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dflpj\" (UniqueName: \"kubernetes.io/projected/19c5b22f-f46d-424e-9a12-9293f0a3e74d-kube-api-access-dflpj\") pod \"data-science-collector-collector-1\" (UID: \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:02.603767 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.603320 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:02.603767 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.603448 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:02.603767 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.603544 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:02.604093 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.604067 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/19c5b22f-f46d-424e-9a12-9293f0a3e74d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:02.605914 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.605892 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:02.606146 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.606123 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:02.606218 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.606149 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:02.627468 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.627439 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dflpj\" (UniqueName: \"kubernetes.io/projected/19c5b22f-f46d-424e-9a12-9293f0a3e74d-kube-api-access-dflpj\") pod \"data-science-collector-collector-1\" (UID: \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:02.685196 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.685160 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:02.992207 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.992173 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:18:02.994260 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:18:02.994235 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c5b22f_f46d_424e_9a12_9293f0a3e74d.slice/crio-4d6ce06f36b2f49d8d5a3ce9316573fcf414156477173da0d8f958e5a8b2b367 WatchSource:0}: Error finding container 4d6ce06f36b2f49d8d5a3ce9316573fcf414156477173da0d8f958e5a8b2b367: Status 404 returned error can't find the container with id 4d6ce06f36b2f49d8d5a3ce9316573fcf414156477173da0d8f958e5a8b2b367 Oct 30 04:18:02.996687 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.996662 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:18:02.996765 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:02.996712 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:18:03.121170 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:03.121132 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="664a4b10-dd85-4d60-8521-71a51d9d3728" path="/var/lib/kubelet/pods/664a4b10-dd85-4d60-8521-71a51d9d3728/volumes" Oct 30 04:18:03.999031 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:03.998991 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"19c5b22f-f46d-424e-9a12-9293f0a3e74d","Type":"ContainerStarted","Data":"8c94e4c380a7bb42d84469e0adcae46880abd1bfe1a42a54510097680fbba85e"} Oct 30 04:18:03.999527 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:03.999051 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"19c5b22f-f46d-424e-9a12-9293f0a3e74d","Type":"ContainerStarted","Data":"4d6ce06f36b2f49d8d5a3ce9316573fcf414156477173da0d8f958e5a8b2b367"} Oct 30 04:18:04.042701 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:04.042642 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.042624997 podStartE2EDuration="2.042624997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:04.042182633 +0000 UTC m=+5845.055460738" watchObservedRunningTime="2026-10-30 04:18:04.042624997 +0000 UTC m=+5845.055903102" Oct 30 04:18:06.184963 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.184923 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk"] Oct 30 04:18:06.185408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.185137 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" podUID="34e141d9-381b-43ed-8a09-869b3dc16e0a" containerName="kube-rbac-proxy" containerID="cri-o://a070a7c2b6e0b55034cc0e20ce4ceb43b7ede343bb51cbc90c0e59ab380c7399" gracePeriod=30 Oct 30 04:18:06.197579 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.197546 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9"] Oct 30 04:18:06.197789 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.197739 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" podUID="1762351e-ba22-436d-aed8-7fe7a6e5cc4a" containerName="kube-rbac-proxy" containerID="cri-o://635c0539792283f91524def775fceecd2aa47a77e68af45d3b39c88073fefc26" gracePeriod=30 Oct 30 04:18:06.197849 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.197774 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" podUID="1762351e-ba22-436d-aed8-7fe7a6e5cc4a" containerName="prom-label-proxy" containerID="cri-o://ea15d210e8822d275465d38867d7f86c518e56b94aeff10f0764bebf0c1c6c65" gracePeriod=30 Oct 30 04:18:06.216706 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.216674 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv"] Oct 30 04:18:06.217045 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.216994 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" podUID="f46907db-6f62-4d68-835b-3bf31401eb66" containerName="kube-rbac-proxy" containerID="cri-o://f57c9a3da461cec50631b1ea47fc952d94b4a1a4b7a479f8a4e34f1c7dd17aab" gracePeriod=30 Oct 30 04:18:06.217211 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.217192 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" podUID="f46907db-6f62-4d68-835b-3bf31401eb66" containerName="prom-label-proxy" containerID="cri-o://a9c36577dadf33db32c5c29e6a9753a081c474048b29c93ec56c74b69f04d993" gracePeriod=30 Oct 30 04:18:06.238745 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.238715 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p"] Oct 30 04:18:06.238934 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.238913 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" podUID="504573d1-3937-4775-a6d5-24320706eb55" containerName="korrel8r" containerID="cri-o://f3109690ebd68068ff5617203f9fb1d4d919e094d459278534f55f1cd7ee2cee" gracePeriod=30 Oct 30 04:18:06.521369 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.521341 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:18:06.553933 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.553902 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" Oct 30 04:18:06.561797 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.561739 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" Oct 30 04:18:06.572510 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.572482 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" Oct 30 04:18:06.637024 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.636906 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-tls\" (UniqueName: \"kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-tls\") pod \"34e141d9-381b-43ed-8a09-869b3dc16e0a\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " Oct 30 04:18:06.637024 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.636964 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-kube-rbac-proxy-config\") pod \"34e141d9-381b-43ed-8a09-869b3dc16e0a\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " Oct 30 04:18:06.637024 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.636999 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34e141d9-381b-43ed-8a09-869b3dc16e0a-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/34e141d9-381b-43ed-8a09-869b3dc16e0a-prometheus-ca\") pod \"34e141d9-381b-43ed-8a09-869b3dc16e0a\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " Oct 30 04:18:06.637363 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.637042 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34e141d9-381b-43ed-8a09-869b3dc16e0a-kube-api-access-cltwb\" (UniqueName: \"kubernetes.io/projected/34e141d9-381b-43ed-8a09-869b3dc16e0a-kube-api-access-cltwb\") pod \"34e141d9-381b-43ed-8a09-869b3dc16e0a\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " Oct 30 04:18:06.637363 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.637152 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-prometheus-client-cert\") pod \"34e141d9-381b-43ed-8a09-869b3dc16e0a\" (UID: \"34e141d9-381b-43ed-8a09-869b3dc16e0a\") " Oct 30 04:18:06.637860 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.637822 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34e141d9-381b-43ed-8a09-869b3dc16e0a-prometheus-ca" pod "34e141d9-381b-43ed-8a09-869b3dc16e0a" (UID: "34e141d9-381b-43ed-8a09-869b3dc16e0a"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:18:06.639746 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.639715 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e141d9-381b-43ed-8a09-869b3dc16e0a-kube-api-access-cltwb" pod "34e141d9-381b-43ed-8a09-869b3dc16e0a" (UID: "34e141d9-381b-43ed-8a09-869b3dc16e0a"). InnerVolumeSpecName "kube-api-access-cltwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:06.639945 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.639862 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-prometheus-client-cert" pod "34e141d9-381b-43ed-8a09-869b3dc16e0a" (UID: "34e141d9-381b-43ed-8a09-869b3dc16e0a"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:06.640246 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.640036 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-tls" pod "34e141d9-381b-43ed-8a09-869b3dc16e0a" (UID: "34e141d9-381b-43ed-8a09-869b3dc16e0a"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:06.640338 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.640275 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-kube-rbac-proxy-config" pod "34e141d9-381b-43ed-8a09-869b3dc16e0a" (UID: "34e141d9-381b-43ed-8a09-869b3dc16e0a"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:06.714981 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.714895 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:18:06.715395 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.715169 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="12802aaa-666e-41ee-9f1c-25700d52148c" containerName="alertmanager" containerID="cri-o://22313c2ac466d574abfb1af42c01cdc3297eb67095ac581062bb58cc6f12a6d3" gracePeriod=120 Oct 30 04:18:06.715504 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.715392 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="12802aaa-666e-41ee-9f1c-25700d52148c" containerName="config-reloader" containerID="cri-o://72930da9ed4e0eefb874b4db3a6555333a94f021e10114909cd172cb83bedbdf" gracePeriod=120 Oct 30 04:18:06.737987 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.737948 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46907db-6f62-4d68-835b-3bf31401eb66-tls\" (UniqueName: \"kubernetes.io/secret/f46907db-6f62-4d68-835b-3bf31401eb66-tls\") pod \"f46907db-6f62-4d68-835b-3bf31401eb66\" (UID: \"f46907db-6f62-4d68-835b-3bf31401eb66\") " Oct 30 04:18:06.738308 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.738280 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-prometheus-tls-ca\") pod \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\" (UID: \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\") " Oct 30 04:18:06.738415 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.738324 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-config\" (UniqueName: \"kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-config\") pod \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\" (UID: \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\") " Oct 30 04:18:06.738415 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.738344 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-kube-api-access-vmqnb\" (UniqueName: \"kubernetes.io/projected/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-kube-api-access-vmqnb\") pod \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\" (UID: \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\") " Oct 30 04:18:06.738415 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.738375 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/504573d1-3937-4775-a6d5-24320706eb55-config\" (UniqueName: \"kubernetes.io/configmap/504573d1-3937-4775-a6d5-24320706eb55-config\") pod \"504573d1-3937-4775-a6d5-24320706eb55\" (UID: \"504573d1-3937-4775-a6d5-24320706eb55\") " Oct 30 04:18:06.738603 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.738424 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46907db-6f62-4d68-835b-3bf31401eb66-config\" (UniqueName: \"kubernetes.io/configmap/f46907db-6f62-4d68-835b-3bf31401eb66-config\") pod \"f46907db-6f62-4d68-835b-3bf31401eb66\" (UID: \"f46907db-6f62-4d68-835b-3bf31401eb66\") " Oct 30 04:18:06.739121 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.738781 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-config" pod "1762351e-ba22-436d-aed8-7fe7a6e5cc4a" (UID: "1762351e-ba22-436d-aed8-7fe7a6e5cc4a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:18:06.739121 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.738872 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-prometheus-tls-ca" pod "1762351e-ba22-436d-aed8-7fe7a6e5cc4a" (UID: "1762351e-ba22-436d-aed8-7fe7a6e5cc4a"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:18:06.739121 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.739096 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46907db-6f62-4d68-835b-3bf31401eb66-config" pod "f46907db-6f62-4d68-835b-3bf31401eb66" (UID: "f46907db-6f62-4d68-835b-3bf31401eb66"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:18:06.739269 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.739201 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/504573d1-3937-4775-a6d5-24320706eb55-config" pod "504573d1-3937-4775-a6d5-24320706eb55" (UID: "504573d1-3937-4775-a6d5-24320706eb55"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:18:06.739319 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.739264 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-tls\" (UniqueName: \"kubernetes.io/secret/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-tls\") pod \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\" (UID: \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\") " Oct 30 04:18:06.739319 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.739312 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46907db-6f62-4d68-835b-3bf31401eb66-kube-api-access-9xkp4\" (UniqueName: \"kubernetes.io/projected/f46907db-6f62-4d68-835b-3bf31401eb66-kube-api-access-9xkp4\") pod \"f46907db-6f62-4d68-835b-3bf31401eb66\" (UID: \"f46907db-6f62-4d68-835b-3bf31401eb66\") " Oct 30 04:18:06.739401 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.739340 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-client-ca\" (UniqueName: \"kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-client-ca\") pod \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\" (UID: \"1762351e-ba22-436d-aed8-7fe7a6e5cc4a\") " Oct 30 04:18:06.739401 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.739380 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/504573d1-3937-4775-a6d5-24320706eb55-tls\" (UniqueName: \"kubernetes.io/secret/504573d1-3937-4775-a6d5-24320706eb55-tls\") pod \"504573d1-3937-4775-a6d5-24320706eb55\" (UID: \"504573d1-3937-4775-a6d5-24320706eb55\") " Oct 30 04:18:06.739517 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.739422 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/504573d1-3937-4775-a6d5-24320706eb55-kube-api-access-sblcm\" (UniqueName: \"kubernetes.io/projected/504573d1-3937-4775-a6d5-24320706eb55-kube-api-access-sblcm\") pod \"504573d1-3937-4775-a6d5-24320706eb55\" (UID: \"504573d1-3937-4775-a6d5-24320706eb55\") " Oct 30 04:18:06.739678 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.739660 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cltwb\" (UniqueName: \"kubernetes.io/projected/34e141d9-381b-43ed-8a09-869b3dc16e0a-kube-api-access-cltwb\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:06.739865 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.739684 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-prometheus-tls-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:06.739865 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.739698 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:06.739865 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.739711 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/504573d1-3937-4775-a6d5-24320706eb55-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:06.739865 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.739722 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f46907db-6f62-4d68-835b-3bf31401eb66-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:06.739865 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.739735 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-prometheus-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:06.739865 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.739774 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:06.739865 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.739788 4925 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34e141d9-381b-43ed-8a09-869b3dc16e0a-kube-rbac-proxy-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:06.739865 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.739801 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/34e141d9-381b-43ed-8a09-869b3dc16e0a-prometheus-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:06.740347 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.739997 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-client-ca" pod "1762351e-ba22-436d-aed8-7fe7a6e5cc4a" (UID: "1762351e-ba22-436d-aed8-7fe7a6e5cc4a"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:18:06.741969 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.741934 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46907db-6f62-4d68-835b-3bf31401eb66-kube-api-access-9xkp4" pod "f46907db-6f62-4d68-835b-3bf31401eb66" (UID: "f46907db-6f62-4d68-835b-3bf31401eb66"). InnerVolumeSpecName "kube-api-access-9xkp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:06.742289 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.742266 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46907db-6f62-4d68-835b-3bf31401eb66-tls" pod "f46907db-6f62-4d68-835b-3bf31401eb66" (UID: "f46907db-6f62-4d68-835b-3bf31401eb66"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:06.742372 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.742306 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-tls" pod "1762351e-ba22-436d-aed8-7fe7a6e5cc4a" (UID: "1762351e-ba22-436d-aed8-7fe7a6e5cc4a"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:06.742372 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.742311 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504573d1-3937-4775-a6d5-24320706eb55-kube-api-access-sblcm" pod "504573d1-3937-4775-a6d5-24320706eb55" (UID: "504573d1-3937-4775-a6d5-24320706eb55"). InnerVolumeSpecName "kube-api-access-sblcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:06.742460 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.742433 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-kube-api-access-vmqnb" pod "1762351e-ba22-436d-aed8-7fe7a6e5cc4a" (UID: "1762351e-ba22-436d-aed8-7fe7a6e5cc4a"). InnerVolumeSpecName "kube-api-access-vmqnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:06.743585 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.743565 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/504573d1-3937-4775-a6d5-24320706eb55-tls" pod "504573d1-3937-4775-a6d5-24320706eb55" (UID: "504573d1-3937-4775-a6d5-24320706eb55"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:06.840581 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.840549 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f46907db-6f62-4d68-835b-3bf31401eb66-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:06.840581 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.840580 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmqnb\" (UniqueName: \"kubernetes.io/projected/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-kube-api-access-vmqnb\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:06.840784 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.840594 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:06.840784 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.840604 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xkp4\" (UniqueName: \"kubernetes.io/projected/f46907db-6f62-4d68-835b-3bf31401eb66-kube-api-access-9xkp4\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:06.840784 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.840612 4925 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/1762351e-ba22-436d-aed8-7fe7a6e5cc4a-client-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:06.840784 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.840621 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/504573d1-3937-4775-a6d5-24320706eb55-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:06.840784 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:06.840628 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sblcm\" (UniqueName: \"kubernetes.io/projected/504573d1-3937-4775-a6d5-24320706eb55-kube-api-access-sblcm\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.014953 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.014865 4925 generic.go:356] "Generic (PLEG): container finished" podID="f46907db-6f62-4d68-835b-3bf31401eb66" containerID="a9c36577dadf33db32c5c29e6a9753a081c474048b29c93ec56c74b69f04d993" exitCode=0 Oct 30 04:18:07.014953 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.014893 4925 generic.go:356] "Generic (PLEG): container finished" podID="f46907db-6f62-4d68-835b-3bf31401eb66" containerID="f57c9a3da461cec50631b1ea47fc952d94b4a1a4b7a479f8a4e34f1c7dd17aab" exitCode=0 Oct 30 04:18:07.014953 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.014924 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" Oct 30 04:18:07.014953 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.014941 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" event={"ID":"f46907db-6f62-4d68-835b-3bf31401eb66","Type":"ContainerDied","Data":"a9c36577dadf33db32c5c29e6a9753a081c474048b29c93ec56c74b69f04d993"} Oct 30 04:18:07.015271 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.015013 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" event={"ID":"f46907db-6f62-4d68-835b-3bf31401eb66","Type":"ContainerDied","Data":"f57c9a3da461cec50631b1ea47fc952d94b4a1a4b7a479f8a4e34f1c7dd17aab"} Oct 30 04:18:07.015271 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.015049 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv" event={"ID":"f46907db-6f62-4d68-835b-3bf31401eb66","Type":"ContainerDied","Data":"d6bae1bef6d0ce0095a1cba6c732dcae39f16b376acc0e7de63196379bb5a03a"} Oct 30 04:18:07.015271 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.015087 4925 scope.go:122] "RemoveContainer" containerID="a9c36577dadf33db32c5c29e6a9753a081c474048b29c93ec56c74b69f04d993" Oct 30 04:18:07.016684 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.016664 4925 generic.go:356] "Generic (PLEG): container finished" podID="1762351e-ba22-436d-aed8-7fe7a6e5cc4a" containerID="ea15d210e8822d275465d38867d7f86c518e56b94aeff10f0764bebf0c1c6c65" exitCode=0 Oct 30 04:18:07.016684 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.016681 4925 generic.go:356] "Generic (PLEG): container finished" podID="1762351e-ba22-436d-aed8-7fe7a6e5cc4a" containerID="635c0539792283f91524def775fceecd2aa47a77e68af45d3b39c88073fefc26" exitCode=0 Oct 30 04:18:07.016883 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.016736 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" event={"ID":"1762351e-ba22-436d-aed8-7fe7a6e5cc4a","Type":"ContainerDied","Data":"ea15d210e8822d275465d38867d7f86c518e56b94aeff10f0764bebf0c1c6c65"} Oct 30 04:18:07.016883 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.016808 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" event={"ID":"1762351e-ba22-436d-aed8-7fe7a6e5cc4a","Type":"ContainerDied","Data":"635c0539792283f91524def775fceecd2aa47a77e68af45d3b39c88073fefc26"} Oct 30 04:18:07.016883 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.016745 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" Oct 30 04:18:07.017023 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.016839 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9" event={"ID":"1762351e-ba22-436d-aed8-7fe7a6e5cc4a","Type":"ContainerDied","Data":"15a6735c08fb497d59340ce04cb7de7a7289cc10f3a23be908b241b76bd89ff4"} Oct 30 04:18:07.019307 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.019287 4925 generic.go:356] "Generic (PLEG): container finished" podID="12802aaa-666e-41ee-9f1c-25700d52148c" containerID="72930da9ed4e0eefb874b4db3a6555333a94f021e10114909cd172cb83bedbdf" exitCode=0 Oct 30 04:18:07.019413 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.019359 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"12802aaa-666e-41ee-9f1c-25700d52148c","Type":"ContainerDied","Data":"72930da9ed4e0eefb874b4db3a6555333a94f021e10114909cd172cb83bedbdf"} Oct 30 04:18:07.020561 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.020540 4925 generic.go:356] "Generic (PLEG): container finished" podID="34e141d9-381b-43ed-8a09-869b3dc16e0a" containerID="a070a7c2b6e0b55034cc0e20ce4ceb43b7ede343bb51cbc90c0e59ab380c7399" exitCode=0 Oct 30 04:18:07.020715 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.020667 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" event={"ID":"34e141d9-381b-43ed-8a09-869b3dc16e0a","Type":"ContainerDied","Data":"a070a7c2b6e0b55034cc0e20ce4ceb43b7ede343bb51cbc90c0e59ab380c7399"} Oct 30 04:18:07.020715 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.020704 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" event={"ID":"34e141d9-381b-43ed-8a09-869b3dc16e0a","Type":"ContainerDied","Data":"58bce57734608b99182810b6310807cb198c9312576a132829157f579c2eda35"} Oct 30 04:18:07.020869 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.020784 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk" Oct 30 04:18:07.024110 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.024055 4925 generic.go:356] "Generic (PLEG): container finished" podID="504573d1-3937-4775-a6d5-24320706eb55" containerID="f3109690ebd68068ff5617203f9fb1d4d919e094d459278534f55f1cd7ee2cee" exitCode=0 Oct 30 04:18:07.024110 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.024099 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" Oct 30 04:18:07.024266 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.024114 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" event={"ID":"504573d1-3937-4775-a6d5-24320706eb55","Type":"ContainerDied","Data":"f3109690ebd68068ff5617203f9fb1d4d919e094d459278534f55f1cd7ee2cee"} Oct 30 04:18:07.024266 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.024145 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p" event={"ID":"504573d1-3937-4775-a6d5-24320706eb55","Type":"ContainerDied","Data":"4e72461d43701a4fc3d204354bbd7c37a1bffea6abbe3dda2be9e29033653eec"} Oct 30 04:18:07.025883 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.025866 4925 scope.go:122] "RemoveContainer" containerID="f57c9a3da461cec50631b1ea47fc952d94b4a1a4b7a479f8a4e34f1c7dd17aab" Oct 30 04:18:07.037310 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.037286 4925 scope.go:122] "RemoveContainer" containerID="a9c36577dadf33db32c5c29e6a9753a081c474048b29c93ec56c74b69f04d993" Oct 30 04:18:07.037583 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:18:07.037563 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c36577dadf33db32c5c29e6a9753a081c474048b29c93ec56c74b69f04d993\": container with ID starting with a9c36577dadf33db32c5c29e6a9753a081c474048b29c93ec56c74b69f04d993 not found: ID does not exist" containerID="a9c36577dadf33db32c5c29e6a9753a081c474048b29c93ec56c74b69f04d993" Oct 30 04:18:07.037658 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.037595 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c36577dadf33db32c5c29e6a9753a081c474048b29c93ec56c74b69f04d993"} err="failed to get container status \"a9c36577dadf33db32c5c29e6a9753a081c474048b29c93ec56c74b69f04d993\": rpc error: code = NotFound desc = could not find container \"a9c36577dadf33db32c5c29e6a9753a081c474048b29c93ec56c74b69f04d993\": container with ID starting with a9c36577dadf33db32c5c29e6a9753a081c474048b29c93ec56c74b69f04d993 not found: ID does not exist" Oct 30 04:18:07.037658 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.037618 4925 scope.go:122] "RemoveContainer" containerID="f57c9a3da461cec50631b1ea47fc952d94b4a1a4b7a479f8a4e34f1c7dd17aab" Oct 30 04:18:07.037884 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:18:07.037866 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57c9a3da461cec50631b1ea47fc952d94b4a1a4b7a479f8a4e34f1c7dd17aab\": container with ID starting with f57c9a3da461cec50631b1ea47fc952d94b4a1a4b7a479f8a4e34f1c7dd17aab not found: ID does not exist" containerID="f57c9a3da461cec50631b1ea47fc952d94b4a1a4b7a479f8a4e34f1c7dd17aab" Oct 30 04:18:07.037932 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.037890 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57c9a3da461cec50631b1ea47fc952d94b4a1a4b7a479f8a4e34f1c7dd17aab"} err="failed to get container status \"f57c9a3da461cec50631b1ea47fc952d94b4a1a4b7a479f8a4e34f1c7dd17aab\": rpc error: code = NotFound desc = could not find container \"f57c9a3da461cec50631b1ea47fc952d94b4a1a4b7a479f8a4e34f1c7dd17aab\": container with ID starting with f57c9a3da461cec50631b1ea47fc952d94b4a1a4b7a479f8a4e34f1c7dd17aab not found: ID does not exist" Oct 30 04:18:07.037932 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.037904 4925 scope.go:122] "RemoveContainer" containerID="a9c36577dadf33db32c5c29e6a9753a081c474048b29c93ec56c74b69f04d993" Oct 30 04:18:07.038122 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.038103 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c36577dadf33db32c5c29e6a9753a081c474048b29c93ec56c74b69f04d993"} err="failed to get container status \"a9c36577dadf33db32c5c29e6a9753a081c474048b29c93ec56c74b69f04d993\": rpc error: code = NotFound desc = could not find container \"a9c36577dadf33db32c5c29e6a9753a081c474048b29c93ec56c74b69f04d993\": container with ID starting with a9c36577dadf33db32c5c29e6a9753a081c474048b29c93ec56c74b69f04d993 not found: ID does not exist" Oct 30 04:18:07.038173 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.038123 4925 scope.go:122] "RemoveContainer" containerID="f57c9a3da461cec50631b1ea47fc952d94b4a1a4b7a479f8a4e34f1c7dd17aab" Oct 30 04:18:07.038320 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.038305 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57c9a3da461cec50631b1ea47fc952d94b4a1a4b7a479f8a4e34f1c7dd17aab"} err="failed to get container status \"f57c9a3da461cec50631b1ea47fc952d94b4a1a4b7a479f8a4e34f1c7dd17aab\": rpc error: code = NotFound desc = could not find container \"f57c9a3da461cec50631b1ea47fc952d94b4a1a4b7a479f8a4e34f1c7dd17aab\": container with ID starting with f57c9a3da461cec50631b1ea47fc952d94b4a1a4b7a479f8a4e34f1c7dd17aab not found: ID does not exist" Oct 30 04:18:07.038363 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.038320 4925 scope.go:122] "RemoveContainer" containerID="ea15d210e8822d275465d38867d7f86c518e56b94aeff10f0764bebf0c1c6c65" Oct 30 04:18:07.046467 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.046451 4925 scope.go:122] "RemoveContainer" containerID="635c0539792283f91524def775fceecd2aa47a77e68af45d3b39c88073fefc26" Oct 30 04:18:07.054359 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.054340 4925 scope.go:122] "RemoveContainer" containerID="ea15d210e8822d275465d38867d7f86c518e56b94aeff10f0764bebf0c1c6c65" Oct 30 04:18:07.054616 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:18:07.054598 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea15d210e8822d275465d38867d7f86c518e56b94aeff10f0764bebf0c1c6c65\": container with ID starting with ea15d210e8822d275465d38867d7f86c518e56b94aeff10f0764bebf0c1c6c65 not found: ID does not exist" containerID="ea15d210e8822d275465d38867d7f86c518e56b94aeff10f0764bebf0c1c6c65" Oct 30 04:18:07.054676 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.054621 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea15d210e8822d275465d38867d7f86c518e56b94aeff10f0764bebf0c1c6c65"} err="failed to get container status \"ea15d210e8822d275465d38867d7f86c518e56b94aeff10f0764bebf0c1c6c65\": rpc error: code = NotFound desc = could not find container \"ea15d210e8822d275465d38867d7f86c518e56b94aeff10f0764bebf0c1c6c65\": container with ID starting with ea15d210e8822d275465d38867d7f86c518e56b94aeff10f0764bebf0c1c6c65 not found: ID does not exist" Oct 30 04:18:07.054676 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.054636 4925 scope.go:122] "RemoveContainer" containerID="635c0539792283f91524def775fceecd2aa47a77e68af45d3b39c88073fefc26" Oct 30 04:18:07.054903 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:18:07.054885 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"635c0539792283f91524def775fceecd2aa47a77e68af45d3b39c88073fefc26\": container with ID starting with 635c0539792283f91524def775fceecd2aa47a77e68af45d3b39c88073fefc26 not found: ID does not exist" containerID="635c0539792283f91524def775fceecd2aa47a77e68af45d3b39c88073fefc26" Oct 30 04:18:07.054949 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.054907 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635c0539792283f91524def775fceecd2aa47a77e68af45d3b39c88073fefc26"} err="failed to get container status \"635c0539792283f91524def775fceecd2aa47a77e68af45d3b39c88073fefc26\": rpc error: code = NotFound desc = could not find container \"635c0539792283f91524def775fceecd2aa47a77e68af45d3b39c88073fefc26\": container with ID starting with 635c0539792283f91524def775fceecd2aa47a77e68af45d3b39c88073fefc26 not found: ID does not exist" Oct 30 04:18:07.054949 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.054923 4925 scope.go:122] "RemoveContainer" containerID="ea15d210e8822d275465d38867d7f86c518e56b94aeff10f0764bebf0c1c6c65" Oct 30 04:18:07.055166 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.055146 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea15d210e8822d275465d38867d7f86c518e56b94aeff10f0764bebf0c1c6c65"} err="failed to get container status \"ea15d210e8822d275465d38867d7f86c518e56b94aeff10f0764bebf0c1c6c65\": rpc error: code = NotFound desc = could not find container \"ea15d210e8822d275465d38867d7f86c518e56b94aeff10f0764bebf0c1c6c65\": container with ID starting with ea15d210e8822d275465d38867d7f86c518e56b94aeff10f0764bebf0c1c6c65 not found: ID does not exist" Oct 30 04:18:07.055211 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.055166 4925 scope.go:122] "RemoveContainer" containerID="635c0539792283f91524def775fceecd2aa47a77e68af45d3b39c88073fefc26" Oct 30 04:18:07.055399 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.055380 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635c0539792283f91524def775fceecd2aa47a77e68af45d3b39c88073fefc26"} err="failed to get container status \"635c0539792283f91524def775fceecd2aa47a77e68af45d3b39c88073fefc26\": rpc error: code = NotFound desc = could not find container \"635c0539792283f91524def775fceecd2aa47a77e68af45d3b39c88073fefc26\": container with ID starting with 635c0539792283f91524def775fceecd2aa47a77e68af45d3b39c88073fefc26 not found: ID does not exist" Oct 30 04:18:07.055464 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.055401 4925 scope.go:122] "RemoveContainer" containerID="a070a7c2b6e0b55034cc0e20ce4ceb43b7ede343bb51cbc90c0e59ab380c7399" Oct 30 04:18:07.062899 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.062883 4925 scope.go:122] "RemoveContainer" containerID="a070a7c2b6e0b55034cc0e20ce4ceb43b7ede343bb51cbc90c0e59ab380c7399" Oct 30 04:18:07.063160 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:18:07.063142 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a070a7c2b6e0b55034cc0e20ce4ceb43b7ede343bb51cbc90c0e59ab380c7399\": container with ID starting with a070a7c2b6e0b55034cc0e20ce4ceb43b7ede343bb51cbc90c0e59ab380c7399 not found: ID does not exist" containerID="a070a7c2b6e0b55034cc0e20ce4ceb43b7ede343bb51cbc90c0e59ab380c7399" Oct 30 04:18:07.063240 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.063167 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a070a7c2b6e0b55034cc0e20ce4ceb43b7ede343bb51cbc90c0e59ab380c7399"} err="failed to get container status \"a070a7c2b6e0b55034cc0e20ce4ceb43b7ede343bb51cbc90c0e59ab380c7399\": rpc error: code = NotFound desc = could not find container \"a070a7c2b6e0b55034cc0e20ce4ceb43b7ede343bb51cbc90c0e59ab380c7399\": container with ID starting with a070a7c2b6e0b55034cc0e20ce4ceb43b7ede343bb51cbc90c0e59ab380c7399 not found: ID does not exist" Oct 30 04:18:07.063240 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.063190 4925 scope.go:122] "RemoveContainer" containerID="f3109690ebd68068ff5617203f9fb1d4d919e094d459278534f55f1cd7ee2cee" Oct 30 04:18:07.070738 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.070719 4925 scope.go:122] "RemoveContainer" containerID="f3109690ebd68068ff5617203f9fb1d4d919e094d459278534f55f1cd7ee2cee" Oct 30 04:18:07.071028 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:18:07.071009 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3109690ebd68068ff5617203f9fb1d4d919e094d459278534f55f1cd7ee2cee\": container with ID starting with f3109690ebd68068ff5617203f9fb1d4d919e094d459278534f55f1cd7ee2cee not found: ID does not exist" containerID="f3109690ebd68068ff5617203f9fb1d4d919e094d459278534f55f1cd7ee2cee" Oct 30 04:18:07.071071 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.071037 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3109690ebd68068ff5617203f9fb1d4d919e094d459278534f55f1cd7ee2cee"} err="failed to get container status \"f3109690ebd68068ff5617203f9fb1d4d919e094d459278534f55f1cd7ee2cee\": rpc error: code = NotFound desc = could not find container \"f3109690ebd68068ff5617203f9fb1d4d919e094d459278534f55f1cd7ee2cee\": container with ID starting with f3109690ebd68068ff5617203f9fb1d4d919e094d459278534f55f1cd7ee2cee not found: ID does not exist" Oct 30 04:18:07.098067 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.098033 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv"] Oct 30 04:18:07.099510 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.099486 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-65lfv"] Oct 30 04:18:07.124203 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.124163 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f46907db-6f62-4d68-835b-3bf31401eb66" path="/var/lib/kubelet/pods/f46907db-6f62-4d68-835b-3bf31401eb66/volumes" Oct 30 04:18:07.150601 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.150572 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk"] Oct 30 04:18:07.156579 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.156556 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-ks8pk"] Oct 30 04:18:07.191048 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.191016 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p"] Oct 30 04:18:07.197289 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.197263 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-c7h2p"] Oct 30 04:18:07.232298 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.232269 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9"] Oct 30 04:18:07.236437 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.236414 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-f8hb9"] Oct 30 04:18:07.353957 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.353922 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:18:07.354190 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.354160 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="19c5b22f-f46d-424e-9a12-9293f0a3e74d" containerName="otc-container" containerID="cri-o://8c94e4c380a7bb42d84469e0adcae46880abd1bfe1a42a54510097680fbba85e" gracePeriod=30 Oct 30 04:18:07.631453 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.631428 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:07.640320 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.640296 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:07.747856 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.747811 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-monitoring-tls-certs\") pod \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\" (UID: \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\") " Oct 30 04:18:07.748043 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.747876 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-web-config\" (UniqueName: \"kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-web-config\") pod \"12802aaa-666e-41ee-9f1c-25700d52148c\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " Oct 30 04:18:07.748043 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.747909 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-tls-certs\" (UniqueName: \"kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-tls-certs\") pod \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\" (UID: \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\") " Oct 30 04:18:07.748043 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.747937 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c5b22f-f46d-424e-9a12-9293f0a3e74d-kube-api-access-dflpj\" (UniqueName: \"kubernetes.io/projected/19c5b22f-f46d-424e-9a12-9293f0a3e74d-kube-api-access-dflpj\") pod \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\" (UID: \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\") " Oct 30 04:18:07.748043 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.747972 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12802aaa-666e-41ee-9f1c-25700d52148c-tls-assets\" (UniqueName: \"kubernetes.io/projected/12802aaa-666e-41ee-9f1c-25700d52148c-tls-assets\") pod \"12802aaa-666e-41ee-9f1c-25700d52148c\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " Oct 30 04:18:07.748271 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.748165 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-config-volume\" (UniqueName: \"kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-config-volume\") pod \"12802aaa-666e-41ee-9f1c-25700d52148c\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " Oct 30 04:18:07.748271 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.748209 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12802aaa-666e-41ee-9f1c-25700d52148c-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/12802aaa-666e-41ee-9f1c-25700d52148c-alertmanager-data-science-monitoringstack-db\") pod \"12802aaa-666e-41ee-9f1c-25700d52148c\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " Oct 30 04:18:07.748271 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.748247 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c5b22f-f46d-424e-9a12-9293f0a3e74d-otc-internal\" (UniqueName: \"kubernetes.io/configmap/19c5b22f-f46d-424e-9a12-9293f0a3e74d-otc-internal\") pod \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\" (UID: \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\") " Oct 30 04:18:07.748433 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.748335 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12802aaa-666e-41ee-9f1c-25700d52148c-kube-api-access-c8zgf\" (UniqueName: \"kubernetes.io/projected/12802aaa-666e-41ee-9f1c-25700d52148c-kube-api-access-c8zgf\") pod \"12802aaa-666e-41ee-9f1c-25700d52148c\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " Oct 30 04:18:07.748433 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.748389 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12802aaa-666e-41ee-9f1c-25700d52148c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/12802aaa-666e-41ee-9f1c-25700d52148c-config-out\") pod \"12802aaa-666e-41ee-9f1c-25700d52148c\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " Oct 30 04:18:07.748433 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.748414 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-cluster-tls-config\") pod \"12802aaa-666e-41ee-9f1c-25700d52148c\" (UID: \"12802aaa-666e-41ee-9f1c-25700d52148c\") " Oct 30 04:18:07.748594 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.748502 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-data-science-collector-ta-client-cert\") pod \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\" (UID: \"19c5b22f-f46d-424e-9a12-9293f0a3e74d\") " Oct 30 04:18:07.749278 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.748866 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c5b22f-f46d-424e-9a12-9293f0a3e74d-otc-internal" pod "19c5b22f-f46d-424e-9a12-9293f0a3e74d" (UID: "19c5b22f-f46d-424e-9a12-9293f0a3e74d"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:18:07.749278 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.748987 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12802aaa-666e-41ee-9f1c-25700d52148c-alertmanager-data-science-monitoringstack-db" pod "12802aaa-666e-41ee-9f1c-25700d52148c" (UID: "12802aaa-666e-41ee-9f1c-25700d52148c"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:18:07.750741 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.750716 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12802aaa-666e-41ee-9f1c-25700d52148c-tls-assets" pod "12802aaa-666e-41ee-9f1c-25700d52148c" (UID: "12802aaa-666e-41ee-9f1c-25700d52148c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:07.753898 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.753871 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12802aaa-666e-41ee-9f1c-25700d52148c-kube-api-access-c8zgf" pod "12802aaa-666e-41ee-9f1c-25700d52148c" (UID: "12802aaa-666e-41ee-9f1c-25700d52148c"). InnerVolumeSpecName "kube-api-access-c8zgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:07.754077 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.754059 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-config-volume" pod "12802aaa-666e-41ee-9f1c-25700d52148c" (UID: "12802aaa-666e-41ee-9f1c-25700d52148c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:07.754154 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.754075 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-monitoring-tls-certs" pod "19c5b22f-f46d-424e-9a12-9293f0a3e74d" (UID: "19c5b22f-f46d-424e-9a12-9293f0a3e74d"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:07.754154 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.754093 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-tls-certs" pod "19c5b22f-f46d-424e-9a12-9293f0a3e74d" (UID: "19c5b22f-f46d-424e-9a12-9293f0a3e74d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:07.754154 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.754116 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-data-science-collector-ta-client-cert" pod "19c5b22f-f46d-424e-9a12-9293f0a3e74d" (UID: "19c5b22f-f46d-424e-9a12-9293f0a3e74d"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:07.756118 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.756080 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c5b22f-f46d-424e-9a12-9293f0a3e74d-kube-api-access-dflpj" pod "19c5b22f-f46d-424e-9a12-9293f0a3e74d" (UID: "19c5b22f-f46d-424e-9a12-9293f0a3e74d"). InnerVolumeSpecName "kube-api-access-dflpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:07.756880 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.756851 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12802aaa-666e-41ee-9f1c-25700d52148c-config-out" pod "12802aaa-666e-41ee-9f1c-25700d52148c" (UID: "12802aaa-666e-41ee-9f1c-25700d52148c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:18:07.759802 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.759777 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-cluster-tls-config" pod "12802aaa-666e-41ee-9f1c-25700d52148c" (UID: "12802aaa-666e-41ee-9f1c-25700d52148c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:07.765053 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.765030 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-web-config" pod "12802aaa-666e-41ee-9f1c-25700d52148c" (UID: "12802aaa-666e-41ee-9f1c-25700d52148c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:07.849995 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.849962 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8zgf\" (UniqueName: \"kubernetes.io/projected/12802aaa-666e-41ee-9f1c-25700d52148c-kube-api-access-c8zgf\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.849995 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.849992 4925 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/12802aaa-666e-41ee-9f1c-25700d52148c-config-out\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.849995 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.850003 4925 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-cluster-tls-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.850223 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.850013 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.850223 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.850023 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.850223 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.850032 4925 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-web-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.850223 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.850040 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c5b22f-f46d-424e-9a12-9293f0a3e74d-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.850223 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.850048 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dflpj\" (UniqueName: \"kubernetes.io/projected/19c5b22f-f46d-424e-9a12-9293f0a3e74d-kube-api-access-dflpj\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.850223 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.850056 4925 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/12802aaa-666e-41ee-9f1c-25700d52148c-tls-assets\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.850223 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.850064 4925 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/12802aaa-666e-41ee-9f1c-25700d52148c-config-volume\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.850223 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.850071 4925 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/12802aaa-666e-41ee-9f1c-25700d52148c-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:07.850223 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:07.850080 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/19c5b22f-f46d-424e-9a12-9293f0a3e74d-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:18:08.031354 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.031267 4925 generic.go:356] "Generic (PLEG): container finished" podID="19c5b22f-f46d-424e-9a12-9293f0a3e74d" containerID="8c94e4c380a7bb42d84469e0adcae46880abd1bfe1a42a54510097680fbba85e" exitCode=0 Oct 30 04:18:08.031354 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.031305 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"19c5b22f-f46d-424e-9a12-9293f0a3e74d","Type":"ContainerDied","Data":"8c94e4c380a7bb42d84469e0adcae46880abd1bfe1a42a54510097680fbba85e"} Oct 30 04:18:08.031354 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.031327 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:08.031642 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.031369 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"19c5b22f-f46d-424e-9a12-9293f0a3e74d","Type":"ContainerDied","Data":"4d6ce06f36b2f49d8d5a3ce9316573fcf414156477173da0d8f958e5a8b2b367"} Oct 30 04:18:08.031642 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.031401 4925 scope.go:122] "RemoveContainer" containerID="8c94e4c380a7bb42d84469e0adcae46880abd1bfe1a42a54510097680fbba85e" Oct 30 04:18:08.035200 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.035177 4925 generic.go:356] "Generic (PLEG): container finished" podID="12802aaa-666e-41ee-9f1c-25700d52148c" containerID="22313c2ac466d574abfb1af42c01cdc3297eb67095ac581062bb58cc6f12a6d3" exitCode=0 Oct 30 04:18:08.035332 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.035238 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:08.035332 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.035257 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"12802aaa-666e-41ee-9f1c-25700d52148c","Type":"ContainerDied","Data":"22313c2ac466d574abfb1af42c01cdc3297eb67095ac581062bb58cc6f12a6d3"} Oct 30 04:18:08.035332 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.035316 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"12802aaa-666e-41ee-9f1c-25700d52148c","Type":"ContainerDied","Data":"9c52d3768d22812e3cc6c1fe09737055c6769a6aeda0f87777f2a2aeb262a1c9"} Oct 30 04:18:08.041314 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.041295 4925 scope.go:122] "RemoveContainer" containerID="8c94e4c380a7bb42d84469e0adcae46880abd1bfe1a42a54510097680fbba85e" Oct 30 04:18:08.041590 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:18:08.041567 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c94e4c380a7bb42d84469e0adcae46880abd1bfe1a42a54510097680fbba85e\": container with ID starting with 8c94e4c380a7bb42d84469e0adcae46880abd1bfe1a42a54510097680fbba85e not found: ID does not exist" containerID="8c94e4c380a7bb42d84469e0adcae46880abd1bfe1a42a54510097680fbba85e" Oct 30 04:18:08.041659 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.041601 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c94e4c380a7bb42d84469e0adcae46880abd1bfe1a42a54510097680fbba85e"} err="failed to get container status \"8c94e4c380a7bb42d84469e0adcae46880abd1bfe1a42a54510097680fbba85e\": rpc error: code = NotFound desc = could not find container \"8c94e4c380a7bb42d84469e0adcae46880abd1bfe1a42a54510097680fbba85e\": container with ID starting with 8c94e4c380a7bb42d84469e0adcae46880abd1bfe1a42a54510097680fbba85e not found: ID does not exist" Oct 30 04:18:08.041659 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.041623 4925 scope.go:122] "RemoveContainer" containerID="72930da9ed4e0eefb874b4db3a6555333a94f021e10114909cd172cb83bedbdf" Oct 30 04:18:08.049698 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.049679 4925 scope.go:122] "RemoveContainer" containerID="22313c2ac466d574abfb1af42c01cdc3297eb67095ac581062bb58cc6f12a6d3" Oct 30 04:18:08.057671 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.057647 4925 scope.go:122] "RemoveContainer" containerID="4325235cb55a43cc9c9bf88578aa1b582601c21d756ab36e197cdb564197ae8d" Oct 30 04:18:08.065647 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.065629 4925 scope.go:122] "RemoveContainer" containerID="72930da9ed4e0eefb874b4db3a6555333a94f021e10114909cd172cb83bedbdf" Oct 30 04:18:08.065943 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:18:08.065918 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72930da9ed4e0eefb874b4db3a6555333a94f021e10114909cd172cb83bedbdf\": container with ID starting with 72930da9ed4e0eefb874b4db3a6555333a94f021e10114909cd172cb83bedbdf not found: ID does not exist" containerID="72930da9ed4e0eefb874b4db3a6555333a94f021e10114909cd172cb83bedbdf" Oct 30 04:18:08.066001 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.065947 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72930da9ed4e0eefb874b4db3a6555333a94f021e10114909cd172cb83bedbdf"} err="failed to get container status \"72930da9ed4e0eefb874b4db3a6555333a94f021e10114909cd172cb83bedbdf\": rpc error: code = NotFound desc = could not find container \"72930da9ed4e0eefb874b4db3a6555333a94f021e10114909cd172cb83bedbdf\": container with ID starting with 72930da9ed4e0eefb874b4db3a6555333a94f021e10114909cd172cb83bedbdf not found: ID does not exist" Oct 30 04:18:08.066001 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.065962 4925 scope.go:122] "RemoveContainer" containerID="22313c2ac466d574abfb1af42c01cdc3297eb67095ac581062bb58cc6f12a6d3" Oct 30 04:18:08.066236 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:18:08.066217 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22313c2ac466d574abfb1af42c01cdc3297eb67095ac581062bb58cc6f12a6d3\": container with ID starting with 22313c2ac466d574abfb1af42c01cdc3297eb67095ac581062bb58cc6f12a6d3 not found: ID does not exist" containerID="22313c2ac466d574abfb1af42c01cdc3297eb67095ac581062bb58cc6f12a6d3" Oct 30 04:18:08.066290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.066243 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22313c2ac466d574abfb1af42c01cdc3297eb67095ac581062bb58cc6f12a6d3"} err="failed to get container status \"22313c2ac466d574abfb1af42c01cdc3297eb67095ac581062bb58cc6f12a6d3\": rpc error: code = NotFound desc = could not find container \"22313c2ac466d574abfb1af42c01cdc3297eb67095ac581062bb58cc6f12a6d3\": container with ID starting with 22313c2ac466d574abfb1af42c01cdc3297eb67095ac581062bb58cc6f12a6d3 not found: ID does not exist" Oct 30 04:18:08.066290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.066259 4925 scope.go:122] "RemoveContainer" containerID="4325235cb55a43cc9c9bf88578aa1b582601c21d756ab36e197cdb564197ae8d" Oct 30 04:18:08.066490 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:18:08.066474 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4325235cb55a43cc9c9bf88578aa1b582601c21d756ab36e197cdb564197ae8d\": container with ID starting with 4325235cb55a43cc9c9bf88578aa1b582601c21d756ab36e197cdb564197ae8d not found: ID does not exist" containerID="4325235cb55a43cc9c9bf88578aa1b582601c21d756ab36e197cdb564197ae8d" Oct 30 04:18:08.066530 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.066493 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4325235cb55a43cc9c9bf88578aa1b582601c21d756ab36e197cdb564197ae8d"} err="failed to get container status \"4325235cb55a43cc9c9bf88578aa1b582601c21d756ab36e197cdb564197ae8d\": rpc error: code = NotFound desc = could not find container \"4325235cb55a43cc9c9bf88578aa1b582601c21d756ab36e197cdb564197ae8d\": container with ID starting with 4325235cb55a43cc9c9bf88578aa1b582601c21d756ab36e197cdb564197ae8d not found: ID does not exist" Oct 30 04:18:08.084375 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.084347 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:18:08.097024 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.097000 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:18:08.122868 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.122840 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:18:08.126756 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:08.126725 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:18:09.120174 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:09.120142 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12802aaa-666e-41ee-9f1c-25700d52148c" path="/var/lib/kubelet/pods/12802aaa-666e-41ee-9f1c-25700d52148c/volumes" Oct 30 04:18:09.120620 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:09.120606 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1762351e-ba22-436d-aed8-7fe7a6e5cc4a" path="/var/lib/kubelet/pods/1762351e-ba22-436d-aed8-7fe7a6e5cc4a/volumes" Oct 30 04:18:09.121013 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:09.121001 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c5b22f-f46d-424e-9a12-9293f0a3e74d" path="/var/lib/kubelet/pods/19c5b22f-f46d-424e-9a12-9293f0a3e74d/volumes" Oct 30 04:18:09.121646 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:09.121632 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34e141d9-381b-43ed-8a09-869b3dc16e0a" path="/var/lib/kubelet/pods/34e141d9-381b-43ed-8a09-869b3dc16e0a/volumes" Oct 30 04:18:09.122005 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:09.121993 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="504573d1-3937-4775-a6d5-24320706eb55" path="/var/lib/kubelet/pods/504573d1-3937-4775-a6d5-24320706eb55/volumes" Oct 30 04:18:18.161642 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:18:18.161604 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:18:27.768268 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768235 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2"] Oct 30 04:18:27.768655 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768566 4925 cpu_manager.go:414] "containerMap: removing container" podUID="12802aaa-666e-41ee-9f1c-25700d52148c" containerName="init-config-reloader" Oct 30 04:18:27.768655 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768577 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12802aaa-666e-41ee-9f1c-25700d52148c" containerName="init-config-reloader" Oct 30 04:18:27.768655 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768590 4925 cpu_manager.go:414] "containerMap: removing container" podUID="34e141d9-381b-43ed-8a09-869b3dc16e0a" containerName="kube-rbac-proxy" Oct 30 04:18:27.768655 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768596 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34e141d9-381b-43ed-8a09-869b3dc16e0a" containerName="kube-rbac-proxy" Oct 30 04:18:27.768655 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768605 4925 cpu_manager.go:414] "containerMap: removing container" podUID="12802aaa-666e-41ee-9f1c-25700d52148c" containerName="config-reloader" Oct 30 04:18:27.768655 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768610 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12802aaa-666e-41ee-9f1c-25700d52148c" containerName="config-reloader" Oct 30 04:18:27.768655 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768618 4925 cpu_manager.go:414] "containerMap: removing container" podUID="504573d1-3937-4775-a6d5-24320706eb55" containerName="korrel8r" Oct 30 04:18:27.768655 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768623 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="504573d1-3937-4775-a6d5-24320706eb55" containerName="korrel8r" Oct 30 04:18:27.768655 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768638 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f46907db-6f62-4d68-835b-3bf31401eb66" containerName="prom-label-proxy" Oct 30 04:18:27.768655 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768643 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46907db-6f62-4d68-835b-3bf31401eb66" containerName="prom-label-proxy" Oct 30 04:18:27.768655 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768652 4925 cpu_manager.go:414] "containerMap: removing container" podUID="f46907db-6f62-4d68-835b-3bf31401eb66" containerName="kube-rbac-proxy" Oct 30 04:18:27.768655 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768657 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46907db-6f62-4d68-835b-3bf31401eb66" containerName="kube-rbac-proxy" Oct 30 04:18:27.769111 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768664 4925 cpu_manager.go:414] "containerMap: removing container" podUID="19c5b22f-f46d-424e-9a12-9293f0a3e74d" containerName="otc-container" Oct 30 04:18:27.769111 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768668 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c5b22f-f46d-424e-9a12-9293f0a3e74d" containerName="otc-container" Oct 30 04:18:27.769111 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768676 4925 cpu_manager.go:414] "containerMap: removing container" podUID="1762351e-ba22-436d-aed8-7fe7a6e5cc4a" containerName="kube-rbac-proxy" Oct 30 04:18:27.769111 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768680 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1762351e-ba22-436d-aed8-7fe7a6e5cc4a" containerName="kube-rbac-proxy" Oct 30 04:18:27.769111 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768687 4925 cpu_manager.go:414] "containerMap: removing container" podUID="12802aaa-666e-41ee-9f1c-25700d52148c" containerName="alertmanager" Oct 30 04:18:27.769111 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768691 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12802aaa-666e-41ee-9f1c-25700d52148c" containerName="alertmanager" Oct 30 04:18:27.769111 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768699 4925 cpu_manager.go:414] "containerMap: removing container" podUID="1762351e-ba22-436d-aed8-7fe7a6e5cc4a" containerName="prom-label-proxy" Oct 30 04:18:27.769111 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.768703 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1762351e-ba22-436d-aed8-7fe7a6e5cc4a" containerName="prom-label-proxy" Oct 30 04:18:27.769468 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.769171 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46907db-6f62-4d68-835b-3bf31401eb66" containerName="kube-rbac-proxy" Oct 30 04:18:27.769468 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.769264 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c5b22f-f46d-424e-9a12-9293f0a3e74d" containerName="otc-container" Oct 30 04:18:27.769468 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.769278 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="1762351e-ba22-436d-aed8-7fe7a6e5cc4a" containerName="kube-rbac-proxy" Oct 30 04:18:27.769468 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.769299 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="12802aaa-666e-41ee-9f1c-25700d52148c" containerName="alertmanager" Oct 30 04:18:27.769468 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.769308 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="1762351e-ba22-436d-aed8-7fe7a6e5cc4a" containerName="prom-label-proxy" Oct 30 04:18:27.769468 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.769332 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="34e141d9-381b-43ed-8a09-869b3dc16e0a" containerName="kube-rbac-proxy" Oct 30 04:18:27.769468 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.769341 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="12802aaa-666e-41ee-9f1c-25700d52148c" containerName="config-reloader" Oct 30 04:18:27.769468 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.769351 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="504573d1-3937-4775-a6d5-24320706eb55" containerName="korrel8r" Oct 30 04:18:27.769468 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.769369 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46907db-6f62-4d68-835b-3bf31401eb66" containerName="prom-label-proxy" Oct 30 04:18:27.775436 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.775403 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:18:27.791635 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.791613 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 04:18:27.792022 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.792001 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2"] Oct 30 04:18:27.802784 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.802742 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 04:18:27.804355 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.804337 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 04:18:27.804433 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.804358 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-fbn79\"" Oct 30 04:18:27.806710 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.806688 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7nxf2\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:18:27.806815 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.806737 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7nxf2\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:18:27.806815 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.806785 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7nxf2\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:18:27.806906 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.806852 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/1f0abd3a-9d98-4070-9826-c4121f039945-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7nxf2\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:18:27.806906 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.806877 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmkhd\" (UniqueName: \"kubernetes.io/projected/1f0abd3a-9d98-4070-9826-c4121f039945-kube-api-access-zmkhd\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7nxf2\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:18:27.807105 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.807084 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 04:18:27.807171 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.807093 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 04:18:27.817046 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.817024 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w"] Oct 30 04:18:27.820528 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.820514 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" Oct 30 04:18:27.835803 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.835776 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w"] Oct 30 04:18:27.858567 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.858533 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6"] Oct 30 04:18:27.862573 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.862553 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" Oct 30 04:18:27.882003 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.881978 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6"] Oct 30 04:18:27.891869 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.891845 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 04:18:27.892016 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.891991 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 04:18:27.892016 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.892003 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-gn92p\"" Oct 30 04:18:27.892148 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.892019 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-wss88\"" Oct 30 04:18:27.892148 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.892058 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 04:18:27.894945 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.894927 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 04:18:27.895044 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.894953 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 04:18:27.895232 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.895214 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 04:18:27.895778 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.895762 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd"] Oct 30 04:18:27.898941 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.898924 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" Oct 30 04:18:27.907271 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.907248 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5c876df5-5e2a-4362-8cae-c36d2c7d4637-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-kwp9w\" (UID: \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" Oct 30 04:18:27.907368 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.907293 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-kwp9w\" (UID: \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" Oct 30 04:18:27.907368 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.907337 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svvkq\" (UniqueName: \"kubernetes.io/projected/5c876df5-5e2a-4362-8cae-c36d2c7d4637-kube-api-access-svvkq\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-kwp9w\" (UID: \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" Oct 30 04:18:27.907509 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.907418 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx8lt\" (UniqueName: \"kubernetes.io/projected/84622b49-99e7-41d2-ad5d-ca8687d3e883-kube-api-access-jx8lt\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-x47m6\" (UID: \"84622b49-99e7-41d2-ad5d-ca8687d3e883\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" Oct 30 04:18:27.907509 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.907448 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7nxf2\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:18:27.907509 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.907484 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/84622b49-99e7-41d2-ad5d-ca8687d3e883-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-x47m6\" (UID: \"84622b49-99e7-41d2-ad5d-ca8687d3e883\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" Oct 30 04:18:27.907674 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:18:27.907574 4925 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 04:18:27.907674 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:18:27.907628 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-prometheus-client-cert podName:1f0abd3a-9d98-4070-9826-c4121f039945 nodeName:}" failed. No retries permitted until 2026-10-30 04:18:28.407614333 +0000 UTC m=+5869.420892439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" (UID: "1f0abd3a-9d98-4070-9826-c4121f039945") : secret "prometheus-operated-tls" not found Oct 30 04:18:27.907674 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.907663 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7nxf2\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:18:27.907869 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.907697 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-kwp9w\" (UID: \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" Oct 30 04:18:27.907869 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.907721 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7nxf2\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:18:27.907869 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.907828 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/1f0abd3a-9d98-4070-9826-c4121f039945-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7nxf2\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:18:27.907869 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.907860 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/84622b49-99e7-41d2-ad5d-ca8687d3e883-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-x47m6\" (UID: \"84622b49-99e7-41d2-ad5d-ca8687d3e883\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" Oct 30 04:18:27.908042 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.907882 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmkhd\" (UniqueName: \"kubernetes.io/projected/1f0abd3a-9d98-4070-9826-c4121f039945-kube-api-access-zmkhd\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7nxf2\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:18:27.908042 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.907898 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-kwp9w\" (UID: \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" Oct 30 04:18:27.908601 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.908581 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/1f0abd3a-9d98-4070-9826-c4121f039945-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7nxf2\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:18:27.910091 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.910068 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7nxf2\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:18:27.910215 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.910194 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7nxf2\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:18:27.920296 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.920274 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd"] Oct 30 04:18:27.933514 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.933485 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmkhd\" (UniqueName: \"kubernetes.io/projected/1f0abd3a-9d98-4070-9826-c4121f039945-kube-api-access-zmkhd\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7nxf2\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:18:27.991981 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.991956 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 04:18:27.992098 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.992004 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-8g8qh\"" Oct 30 04:18:27.995608 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:27.995589 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 04:18:28.009118 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.009087 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5c876df5-5e2a-4362-8cae-c36d2c7d4637-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-kwp9w\" (UID: \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" Oct 30 04:18:28.009213 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.009127 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-kwp9w\" (UID: \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" Oct 30 04:18:28.009213 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.009145 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svvkq\" (UniqueName: \"kubernetes.io/projected/5c876df5-5e2a-4362-8cae-c36d2c7d4637-kube-api-access-svvkq\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-kwp9w\" (UID: \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" Oct 30 04:18:28.009213 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.009170 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx8lt\" (UniqueName: \"kubernetes.io/projected/84622b49-99e7-41d2-ad5d-ca8687d3e883-kube-api-access-jx8lt\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-x47m6\" (UID: \"84622b49-99e7-41d2-ad5d-ca8687d3e883\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" Oct 30 04:18:28.009372 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.009234 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/84622b49-99e7-41d2-ad5d-ca8687d3e883-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-x47m6\" (UID: \"84622b49-99e7-41d2-ad5d-ca8687d3e883\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" Oct 30 04:18:28.009372 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.009281 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-kwp9w\" (UID: \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" Oct 30 04:18:28.009372 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.009309 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/34e697cb-4277-469b-8c69-5f1a97708d25-tls\") pod \"korrel8r-5cbcc89b8f-nndsd\" (UID: \"34e697cb-4277-469b-8c69-5f1a97708d25\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" Oct 30 04:18:28.009372 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.009334 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znxwd\" (UniqueName: \"kubernetes.io/projected/34e697cb-4277-469b-8c69-5f1a97708d25-kube-api-access-znxwd\") pod \"korrel8r-5cbcc89b8f-nndsd\" (UID: \"34e697cb-4277-469b-8c69-5f1a97708d25\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" Oct 30 04:18:28.009589 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.009375 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/84622b49-99e7-41d2-ad5d-ca8687d3e883-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-x47m6\" (UID: \"84622b49-99e7-41d2-ad5d-ca8687d3e883\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" Oct 30 04:18:28.009589 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.009484 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34e697cb-4277-469b-8c69-5f1a97708d25-config\") pod \"korrel8r-5cbcc89b8f-nndsd\" (UID: \"34e697cb-4277-469b-8c69-5f1a97708d25\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" Oct 30 04:18:28.009589 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.009527 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-kwp9w\" (UID: \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" Oct 30 04:18:28.010011 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.009989 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-kwp9w\" (UID: \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" Oct 30 04:18:28.010011 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.010005 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/84622b49-99e7-41d2-ad5d-ca8687d3e883-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-x47m6\" (UID: \"84622b49-99e7-41d2-ad5d-ca8687d3e883\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" Oct 30 04:18:28.010199 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.010092 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-kwp9w\" (UID: \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" Oct 30 04:18:28.010199 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.010103 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-kwp9w\" (UID: \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" Oct 30 04:18:28.011597 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.011580 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5c876df5-5e2a-4362-8cae-c36d2c7d4637-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-kwp9w\" (UID: \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" Oct 30 04:18:28.011825 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.011809 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/84622b49-99e7-41d2-ad5d-ca8687d3e883-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-x47m6\" (UID: \"84622b49-99e7-41d2-ad5d-ca8687d3e883\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" Oct 30 04:18:28.032605 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.032556 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svvkq\" (UniqueName: \"kubernetes.io/projected/5c876df5-5e2a-4362-8cae-c36d2c7d4637-kube-api-access-svvkq\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-kwp9w\" (UID: \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" Oct 30 04:18:28.032605 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.032558 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx8lt\" (UniqueName: \"kubernetes.io/projected/84622b49-99e7-41d2-ad5d-ca8687d3e883-kube-api-access-jx8lt\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-x47m6\" (UID: \"84622b49-99e7-41d2-ad5d-ca8687d3e883\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" Oct 30 04:18:28.110954 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.110923 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34e697cb-4277-469b-8c69-5f1a97708d25-config\") pod \"korrel8r-5cbcc89b8f-nndsd\" (UID: \"34e697cb-4277-469b-8c69-5f1a97708d25\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" Oct 30 04:18:28.111077 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.111020 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/34e697cb-4277-469b-8c69-5f1a97708d25-tls\") pod \"korrel8r-5cbcc89b8f-nndsd\" (UID: \"34e697cb-4277-469b-8c69-5f1a97708d25\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" Oct 30 04:18:28.111077 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.111038 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znxwd\" (UniqueName: \"kubernetes.io/projected/34e697cb-4277-469b-8c69-5f1a97708d25-kube-api-access-znxwd\") pod \"korrel8r-5cbcc89b8f-nndsd\" (UID: \"34e697cb-4277-469b-8c69-5f1a97708d25\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" Oct 30 04:18:28.111690 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.111660 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34e697cb-4277-469b-8c69-5f1a97708d25-config\") pod \"korrel8r-5cbcc89b8f-nndsd\" (UID: \"34e697cb-4277-469b-8c69-5f1a97708d25\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" Oct 30 04:18:28.113276 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.113252 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/34e697cb-4277-469b-8c69-5f1a97708d25-tls\") pod \"korrel8r-5cbcc89b8f-nndsd\" (UID: \"34e697cb-4277-469b-8c69-5f1a97708d25\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" Oct 30 04:18:28.135032 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.135013 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" Oct 30 04:18:28.135328 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.135306 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znxwd\" (UniqueName: \"kubernetes.io/projected/34e697cb-4277-469b-8c69-5f1a97708d25-kube-api-access-znxwd\") pod \"korrel8r-5cbcc89b8f-nndsd\" (UID: \"34e697cb-4277-469b-8c69-5f1a97708d25\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" Oct 30 04:18:28.176796 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.176765 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" Oct 30 04:18:28.213894 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.213859 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" Oct 30 04:18:28.415395 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.415360 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7nxf2\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:18:28.417844 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.417824 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-7nxf2\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:18:28.433027 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.432981 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w"] Oct 30 04:18:28.435392 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:18:28.435364 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c876df5_5e2a_4362_8cae_c36d2c7d4637.slice/crio-ab71eb6e1cb6b20b6311d7c502649664ec746cc04921e2214b0ff87e65bfbdb6 WatchSource:0}: Error finding container ab71eb6e1cb6b20b6311d7c502649664ec746cc04921e2214b0ff87e65bfbdb6: Status 404 returned error can't find the container with id ab71eb6e1cb6b20b6311d7c502649664ec746cc04921e2214b0ff87e65bfbdb6 Oct 30 04:18:28.485508 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.485481 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6"] Oct 30 04:18:28.487670 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:18:28.487646 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84622b49_99e7_41d2_ad5d_ca8687d3e883.slice/crio-7771a458db7caedecd57ea97126dfff7e223fa059bda6303fb5094e84fdc0cb8 WatchSource:0}: Error finding container 7771a458db7caedecd57ea97126dfff7e223fa059bda6303fb5094e84fdc0cb8: Status 404 returned error can't find the container with id 7771a458db7caedecd57ea97126dfff7e223fa059bda6303fb5094e84fdc0cb8 Oct 30 04:18:28.523967 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.523943 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd"] Oct 30 04:18:28.527424 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:18:28.527396 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e697cb_4277_469b_8c69_5f1a97708d25.slice/crio-4ddf109ed9581008bf6809dd8fca27aa7c54e90e566a88ff5b88e6f09dedfe93 WatchSource:0}: Error finding container 4ddf109ed9581008bf6809dd8fca27aa7c54e90e566a88ff5b88e6f09dedfe93: Status 404 returned error can't find the container with id 4ddf109ed9581008bf6809dd8fca27aa7c54e90e566a88ff5b88e6f09dedfe93 Oct 30 04:18:28.689729 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.689650 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:18:28.992054 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:28.991898 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2"] Oct 30 04:18:28.994589 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:18:28.994563 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0abd3a_9d98_4070_9826_c4121f039945.slice/crio-9aacdb034448b89e4ce0ce2a7e337963f58672e4a91de4fe9ae6bf063ba0dde6 WatchSource:0}: Error finding container 9aacdb034448b89e4ce0ce2a7e337963f58672e4a91de4fe9ae6bf063ba0dde6: Status 404 returned error can't find the container with id 9aacdb034448b89e4ce0ce2a7e337963f58672e4a91de4fe9ae6bf063ba0dde6 Oct 30 04:18:29.145000 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:29.144964 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" event={"ID":"1f0abd3a-9d98-4070-9826-c4121f039945","Type":"ContainerStarted","Data":"9aacdb034448b89e4ce0ce2a7e337963f58672e4a91de4fe9ae6bf063ba0dde6"} Oct 30 04:18:29.146726 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:29.146698 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" event={"ID":"84622b49-99e7-41d2-ad5d-ca8687d3e883","Type":"ContainerStarted","Data":"a880e6babc9f7564107144cd397530216823792ecbf05bce7e8d693e90e81607"} Oct 30 04:18:29.146858 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:29.146772 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" event={"ID":"84622b49-99e7-41d2-ad5d-ca8687d3e883","Type":"ContainerStarted","Data":"f55cedcda59dcccf8d3e12ea9991510eb7058af2f0268b6477e441901fd51832"} Oct 30 04:18:29.146858 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:29.146803 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" event={"ID":"84622b49-99e7-41d2-ad5d-ca8687d3e883","Type":"ContainerStarted","Data":"7771a458db7caedecd57ea97126dfff7e223fa059bda6303fb5094e84fdc0cb8"} Oct 30 04:18:29.148540 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:29.148516 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" event={"ID":"5c876df5-5e2a-4362-8cae-c36d2c7d4637","Type":"ContainerStarted","Data":"5e08478e128ccadd2dab8358fc6f82f9ac3e6642cbb29e5ac762bf6bdca879d7"} Oct 30 04:18:29.148674 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:29.148559 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" event={"ID":"5c876df5-5e2a-4362-8cae-c36d2c7d4637","Type":"ContainerStarted","Data":"4573c643efeb8f5b258664f07574416e3f0fd4cea963715f5d579428071e3f43"} Oct 30 04:18:29.148674 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:29.148579 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" event={"ID":"5c876df5-5e2a-4362-8cae-c36d2c7d4637","Type":"ContainerStarted","Data":"ab71eb6e1cb6b20b6311d7c502649664ec746cc04921e2214b0ff87e65bfbdb6"} Oct 30 04:18:29.149978 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:29.149958 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" event={"ID":"34e697cb-4277-469b-8c69-5f1a97708d25","Type":"ContainerStarted","Data":"3bb23270d3c60bf6e8267789c841c26bf74b1381133c27b0d67b1deaaf18aec4"} Oct 30 04:18:29.150069 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:29.149991 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" event={"ID":"34e697cb-4277-469b-8c69-5f1a97708d25","Type":"ContainerStarted","Data":"4ddf109ed9581008bf6809dd8fca27aa7c54e90e566a88ff5b88e6f09dedfe93"} Oct 30 04:18:29.150069 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:29.150015 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" Oct 30 04:18:29.189464 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:29.189420 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" podStartSLOduration=2.189407701 podStartE2EDuration="2.189407701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:29.187706528 +0000 UTC m=+5870.200984633" watchObservedRunningTime="2026-10-30 04:18:29.189407701 +0000 UTC m=+5870.202686048" Oct 30 04:18:29.236128 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:29.236079 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" podStartSLOduration=2.236065802 podStartE2EDuration="2.236065802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:29.234558918 +0000 UTC m=+5870.247837023" watchObservedRunningTime="2026-10-30 04:18:29.236065802 +0000 UTC m=+5870.249343904" Oct 30 04:18:29.279167 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:29.279073 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" podStartSLOduration=2.2790589 podStartE2EDuration="2.2790589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:29.278195326 +0000 UTC m=+5870.291473430" watchObservedRunningTime="2026-10-30 04:18:29.2790589 +0000 UTC m=+5870.292337056" Oct 30 04:18:30.157036 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:30.156980 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" event={"ID":"1f0abd3a-9d98-4070-9826-c4121f039945","Type":"ContainerStarted","Data":"4399af80091ec075a36d9d0aca89882037631bcd9832126bcd906fbc14ec8d52"} Oct 30 04:18:30.157653 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:30.157364 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:18:30.198970 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:30.198925 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" podStartSLOduration=3.198912107 podStartE2EDuration="3.198912107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:30.198499044 +0000 UTC m=+5871.211777147" watchObservedRunningTime="2026-10-30 04:18:30.198912107 +0000 UTC m=+5871.212190211" Oct 30 04:18:33.111293 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.111259 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:18:33.115114 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.115087 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.145293 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.145267 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:18:33.157073 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.157047 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.157209 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.157092 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg8ps\" (UniqueName: \"kubernetes.io/projected/64a87316-cfca-4902-b446-225b525fc669-kube-api-access-dg8ps\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.157209 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.157125 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.157288 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.157231 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/64a87316-cfca-4902-b446-225b525fc669-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.157288 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.157271 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.157373 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.157297 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64a87316-cfca-4902-b446-225b525fc669-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.157373 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.157350 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64a87316-cfca-4902-b446-225b525fc669-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.208669 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.208639 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 04:18:33.208854 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.208777 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 04:18:33.208923 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.208855 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 04:18:33.208923 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.208895 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-47r6j\"" Oct 30 04:18:33.208991 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.208943 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:18:33.257901 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.257875 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg8ps\" (UniqueName: \"kubernetes.io/projected/64a87316-cfca-4902-b446-225b525fc669-kube-api-access-dg8ps\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.258054 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.257905 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.258054 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.257975 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/64a87316-cfca-4902-b446-225b525fc669-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.258054 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.258001 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.258229 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.258062 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64a87316-cfca-4902-b446-225b525fc669-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.258229 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.258112 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64a87316-cfca-4902-b446-225b525fc669-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.258229 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.258153 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.258426 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.258404 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/64a87316-cfca-4902-b446-225b525fc669-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.260887 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.260859 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.261062 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.261039 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64a87316-cfca-4902-b446-225b525fc669-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.261147 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.261090 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64a87316-cfca-4902-b446-225b525fc669-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.261147 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.261111 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.261244 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.261228 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.283294 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.283268 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg8ps\" (UniqueName: \"kubernetes.io/projected/64a87316-cfca-4902-b446-225b525fc669-kube-api-access-dg8ps\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.435455 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.435379 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:33.766403 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:33.766208 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:18:33.769183 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:18:33.769139 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a87316_cfca_4902_b446_225b525fc669.slice/crio-97fa820383513be7fefdf99a46188b2ffe8acb01e804f20b7194a36e99030246 WatchSource:0}: Error finding container 97fa820383513be7fefdf99a46188b2ffe8acb01e804f20b7194a36e99030246: Status 404 returned error can't find the container with id 97fa820383513be7fefdf99a46188b2ffe8acb01e804f20b7194a36e99030246 Oct 30 04:18:34.177919 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:34.177870 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"64a87316-cfca-4902-b446-225b525fc669","Type":"ContainerStarted","Data":"41d9d7b3b5a8774583048e4f5682c240ba9f92f42c029d6ad70a887834b6a953"} Oct 30 04:18:34.178366 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:34.177957 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"64a87316-cfca-4902-b446-225b525fc669","Type":"ContainerStarted","Data":"97fa820383513be7fefdf99a46188b2ffe8acb01e804f20b7194a36e99030246"} Oct 30 04:18:37.168614 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:37.168579 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:18:38.201013 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:38.200980 4925 generic.go:356] "Generic (PLEG): container finished" podID="64a87316-cfca-4902-b446-225b525fc669" containerID="41d9d7b3b5a8774583048e4f5682c240ba9f92f42c029d6ad70a887834b6a953" exitCode=0 Oct 30 04:18:38.201533 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:38.201052 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"64a87316-cfca-4902-b446-225b525fc669","Type":"ContainerDied","Data":"41d9d7b3b5a8774583048e4f5682c240ba9f92f42c029d6ad70a887834b6a953"} Oct 30 04:18:39.208985 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:39.208948 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"64a87316-cfca-4902-b446-225b525fc669","Type":"ContainerStarted","Data":"9dfab8fd4fe7449ef6bd29501834f492385fb8b231cfe38a760916c13d2605fe"} Oct 30 04:18:39.209478 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:39.209016 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"64a87316-cfca-4902-b446-225b525fc669","Type":"ContainerStarted","Data":"a5ce3f95a756013281b742137987e6b84cf5d0e13e0ddee140340e18fbe70592"} Oct 30 04:18:39.209478 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:39.209209 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:39.281663 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:39.281608 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.895932781 podStartE2EDuration="6.281593944s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.385661163s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:39.267906096 +0000 UTC m=+5880.281184202" watchObservedRunningTime="2026-10-30 04:18:39.281593944 +0000 UTC m=+5880.294872047" Oct 30 04:18:40.159960 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:40.159929 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" Oct 30 04:18:45.216411 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:45.216378 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:18:48.161647 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:18:48.161604 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:18:51.785104 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:51.785012 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:18:51.790086 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:51.790064 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:51.818057 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:51.818025 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:18:51.818215 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:51.818084 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:18:51.818215 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:51.818084 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:18:51.818215 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:51.818091 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:18:51.818215 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:51.818124 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8q6hm\"" Oct 30 04:18:51.821274 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:51.821251 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:18:51.924888 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:51.924848 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:51.925053 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:51.924901 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:51.925053 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:51.924931 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:51.925053 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:51.924985 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:51.925204 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:51.925053 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xcbl\" (UniqueName: \"kubernetes.io/projected/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-kube-api-access-5xcbl\") pod \"data-science-collector-collector-1\" (UID: \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:52.025537 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:52.025495 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:52.025735 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:52.025553 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xcbl\" (UniqueName: \"kubernetes.io/projected/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-kube-api-access-5xcbl\") pod \"data-science-collector-collector-1\" (UID: \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:52.025735 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:52.025602 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:52.025735 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:52.025632 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:52.025735 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:52.025652 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:52.026389 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:52.026366 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:52.028196 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:52.028169 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:52.028504 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:52.028488 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:52.028583 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:52.028567 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:52.049654 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:52.049589 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xcbl\" (UniqueName: \"kubernetes.io/projected/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-kube-api-access-5xcbl\") pod \"data-science-collector-collector-1\" (UID: \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:52.102627 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:52.102589 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:18:52.404930 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:52.404897 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:18:52.408142 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:18:52.408113 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab1aec3_d420_4112_a3f8_50e91b66bdb5.slice/crio-4123c25c1cefa0d4da02b5dcd6d46778acdc7c230ac0d8306136d32848a6a345 WatchSource:0}: Error finding container 4123c25c1cefa0d4da02b5dcd6d46778acdc7c230ac0d8306136d32848a6a345: Status 404 returned error can't find the container with id 4123c25c1cefa0d4da02b5dcd6d46778acdc7c230ac0d8306136d32848a6a345 Oct 30 04:18:52.410348 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:52.410321 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:18:52.410423 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:52.410374 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:18:53.278314 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:53.278283 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5ab1aec3-d420-4112-a3f8-50e91b66bdb5","Type":"ContainerStarted","Data":"c137ccec5f01cb91bc462c4870884f2b52bd7aa9064a724b208eeab1ecb8b06d"} Oct 30 04:18:53.278777 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:53.278334 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5ab1aec3-d420-4112-a3f8-50e91b66bdb5","Type":"ContainerStarted","Data":"4123c25c1cefa0d4da02b5dcd6d46778acdc7c230ac0d8306136d32848a6a345"} Oct 30 04:18:53.321955 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:18:53.321902 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.32188912 podStartE2EDuration="2.32188912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:53.320823025 +0000 UTC m=+5894.334101128" watchObservedRunningTime="2026-10-30 04:18:53.32188912 +0000 UTC m=+5894.335167224" Oct 30 04:19:06.043654 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:06.043616 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:19:06.044360 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:06.043834 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="5ab1aec3-d420-4112-a3f8-50e91b66bdb5" containerName="otc-container" containerID="cri-o://c137ccec5f01cb91bc462c4870884f2b52bd7aa9064a724b208eeab1ecb8b06d" gracePeriod=30 Oct 30 04:19:07.290498 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.290463 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:07.347412 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.347378 4925 generic.go:356] "Generic (PLEG): container finished" podID="5ab1aec3-d420-4112-a3f8-50e91b66bdb5" containerID="c137ccec5f01cb91bc462c4870884f2b52bd7aa9064a724b208eeab1ecb8b06d" exitCode=0 Oct 30 04:19:07.347586 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.347429 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:07.347586 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.347463 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5ab1aec3-d420-4112-a3f8-50e91b66bdb5","Type":"ContainerDied","Data":"c137ccec5f01cb91bc462c4870884f2b52bd7aa9064a724b208eeab1ecb8b06d"} Oct 30 04:19:07.347586 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.347529 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5ab1aec3-d420-4112-a3f8-50e91b66bdb5","Type":"ContainerDied","Data":"4123c25c1cefa0d4da02b5dcd6d46778acdc7c230ac0d8306136d32848a6a345"} Oct 30 04:19:07.347586 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.347563 4925 scope.go:122] "RemoveContainer" containerID="c137ccec5f01cb91bc462c4870884f2b52bd7aa9064a724b208eeab1ecb8b06d" Oct 30 04:19:07.357097 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.357064 4925 scope.go:122] "RemoveContainer" containerID="c137ccec5f01cb91bc462c4870884f2b52bd7aa9064a724b208eeab1ecb8b06d" Oct 30 04:19:07.357403 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:19:07.357374 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c137ccec5f01cb91bc462c4870884f2b52bd7aa9064a724b208eeab1ecb8b06d\": container with ID starting with c137ccec5f01cb91bc462c4870884f2b52bd7aa9064a724b208eeab1ecb8b06d not found: ID does not exist" containerID="c137ccec5f01cb91bc462c4870884f2b52bd7aa9064a724b208eeab1ecb8b06d" Oct 30 04:19:07.357498 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.357413 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c137ccec5f01cb91bc462c4870884f2b52bd7aa9064a724b208eeab1ecb8b06d"} err="failed to get container status \"c137ccec5f01cb91bc462c4870884f2b52bd7aa9064a724b208eeab1ecb8b06d\": rpc error: code = NotFound desc = could not find container \"c137ccec5f01cb91bc462c4870884f2b52bd7aa9064a724b208eeab1ecb8b06d\": container with ID starting with c137ccec5f01cb91bc462c4870884f2b52bd7aa9064a724b208eeab1ecb8b06d not found: ID does not exist" Oct 30 04:19:07.458290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.458212 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-data-science-collector-ta-client-cert\") pod \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\" (UID: \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\") " Oct 30 04:19:07.458290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.458263 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-monitoring-tls-certs\") pod \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\" (UID: \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\") " Oct 30 04:19:07.458497 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.458367 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-kube-api-access-5xcbl\" (UniqueName: \"kubernetes.io/projected/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-kube-api-access-5xcbl\") pod \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\" (UID: \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\") " Oct 30 04:19:07.458497 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.458409 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-tls-certs\") pod \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\" (UID: \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\") " Oct 30 04:19:07.458497 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.458449 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-otc-internal\") pod \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\" (UID: \"5ab1aec3-d420-4112-a3f8-50e91b66bdb5\") " Oct 30 04:19:07.458899 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.458866 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-otc-internal" pod "5ab1aec3-d420-4112-a3f8-50e91b66bdb5" (UID: "5ab1aec3-d420-4112-a3f8-50e91b66bdb5"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:07.460466 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.460442 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-kube-api-access-5xcbl" pod "5ab1aec3-d420-4112-a3f8-50e91b66bdb5" (UID: "5ab1aec3-d420-4112-a3f8-50e91b66bdb5"). InnerVolumeSpecName "kube-api-access-5xcbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:07.460574 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.460524 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-monitoring-tls-certs" pod "5ab1aec3-d420-4112-a3f8-50e91b66bdb5" (UID: "5ab1aec3-d420-4112-a3f8-50e91b66bdb5"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:07.460877 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.460861 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-tls-certs" pod "5ab1aec3-d420-4112-a3f8-50e91b66bdb5" (UID: "5ab1aec3-d420-4112-a3f8-50e91b66bdb5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:07.460940 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.460883 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-data-science-collector-ta-client-cert" pod "5ab1aec3-d420-4112-a3f8-50e91b66bdb5" (UID: "5ab1aec3-d420-4112-a3f8-50e91b66bdb5"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:07.559141 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.559110 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:07.559141 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.559138 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:07.559141 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.559149 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:07.559380 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.559157 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:07.559380 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.559166 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xcbl\" (UniqueName: \"kubernetes.io/projected/5ab1aec3-d420-4112-a3f8-50e91b66bdb5-kube-api-access-5xcbl\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:07.692985 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.692945 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:19:07.695914 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.695882 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:19:07.733668 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.733594 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:19:07.733998 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.733985 4925 cpu_manager.go:414] "containerMap: removing container" podUID="5ab1aec3-d420-4112-a3f8-50e91b66bdb5" containerName="otc-container" Oct 30 04:19:07.734042 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.733999 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab1aec3-d420-4112-a3f8-50e91b66bdb5" containerName="otc-container" Oct 30 04:19:07.734077 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.734067 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab1aec3-d420-4112-a3f8-50e91b66bdb5" containerName="otc-container" Oct 30 04:19:07.738549 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.738526 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:07.756433 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.756248 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:19:07.756592 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.756517 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:19:07.756592 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.756578 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:19:07.756786 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.756626 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:19:07.756786 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.756762 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8q6hm\"" Oct 30 04:19:07.759357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.759334 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:19:07.861499 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.861465 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:07.861668 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.861512 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:07.861668 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.861540 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:07.861764 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.861691 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:07.861852 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.861825 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7f6q\" (UniqueName: \"kubernetes.io/projected/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-kube-api-access-n7f6q\") pod \"data-science-collector-collector-1\" (UID: \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:07.963306 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.963263 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:07.963306 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.963306 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:07.963567 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.963344 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:07.963567 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.963390 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7f6q\" (UniqueName: \"kubernetes.io/projected/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-kube-api-access-n7f6q\") pod \"data-science-collector-collector-1\" (UID: \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:07.963567 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.963419 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:07.964013 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.963985 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:07.965860 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.965840 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:07.965961 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.965897 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:07.966005 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.965972 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:07.986831 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:07.986769 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7f6q\" (UniqueName: \"kubernetes.io/projected/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-kube-api-access-n7f6q\") pod \"data-science-collector-collector-1\" (UID: \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:08.051138 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:08.051110 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:08.350935 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:08.350723 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:19:08.353520 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:19:08.353491 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7a2e4b_34b7_4abb_a186_0f8573153dc1.slice/crio-5972d809dbb45a1536d0209695d89f267f26de8aa7fade150940f837d23ab18c WatchSource:0}: Error finding container 5972d809dbb45a1536d0209695d89f267f26de8aa7fade150940f837d23ab18c: Status 404 returned error can't find the container with id 5972d809dbb45a1536d0209695d89f267f26de8aa7fade150940f837d23ab18c Oct 30 04:19:08.355742 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:08.355707 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:19:08.355845 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:08.355813 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:19:09.119791 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:09.119736 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ab1aec3-d420-4112-a3f8-50e91b66bdb5" path="/var/lib/kubelet/pods/5ab1aec3-d420-4112-a3f8-50e91b66bdb5/volumes" Oct 30 04:19:09.361053 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:09.361021 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0b7a2e4b-34b7-4abb-a186-0f8573153dc1","Type":"ContainerStarted","Data":"009ff41c8ed17076457a9f27ff8a291424cb10042b866bed122f3e7eb2909964"} Oct 30 04:19:09.361458 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:09.361069 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0b7a2e4b-34b7-4abb-a186-0f8573153dc1","Type":"ContainerStarted","Data":"5972d809dbb45a1536d0209695d89f267f26de8aa7fade150940f837d23ab18c"} Oct 30 04:19:09.403909 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:09.403812 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.403797118 podStartE2EDuration="2.403797118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:09.403138122 +0000 UTC m=+5910.416416241" watchObservedRunningTime="2026-10-30 04:19:09.403797118 +0000 UTC m=+5910.417075221" Oct 30 04:19:10.267966 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.267930 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2"] Oct 30 04:19:10.268135 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.268115 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" podUID="1f0abd3a-9d98-4070-9826-c4121f039945" containerName="kube-rbac-proxy" containerID="cri-o://4399af80091ec075a36d9d0aca89882037631bcd9832126bcd906fbc14ec8d52" gracePeriod=30 Oct 30 04:19:10.282146 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.282118 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w"] Oct 30 04:19:10.282350 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.282328 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" podUID="5c876df5-5e2a-4362-8cae-c36d2c7d4637" containerName="kube-rbac-proxy" containerID="cri-o://4573c643efeb8f5b258664f07574416e3f0fd4cea963715f5d579428071e3f43" gracePeriod=30 Oct 30 04:19:10.282426 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.282334 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" podUID="5c876df5-5e2a-4362-8cae-c36d2c7d4637" containerName="prom-label-proxy" containerID="cri-o://5e08478e128ccadd2dab8358fc6f82f9ac3e6642cbb29e5ac762bf6bdca879d7" gracePeriod=30 Oct 30 04:19:10.301580 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.301542 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6"] Oct 30 04:19:10.301798 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.301772 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" podUID="84622b49-99e7-41d2-ad5d-ca8687d3e883" containerName="prom-label-proxy" containerID="cri-o://a880e6babc9f7564107144cd397530216823792ecbf05bce7e8d693e90e81607" gracePeriod=30 Oct 30 04:19:10.301914 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.301772 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" podUID="84622b49-99e7-41d2-ad5d-ca8687d3e883" containerName="kube-rbac-proxy" containerID="cri-o://f55cedcda59dcccf8d3e12ea9991510eb7058af2f0268b6477e441901fd51832" gracePeriod=30 Oct 30 04:19:10.325104 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.325033 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd"] Oct 30 04:19:10.325320 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.325270 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" podUID="34e697cb-4277-469b-8c69-5f1a97708d25" containerName="korrel8r" containerID="cri-o://3bb23270d3c60bf6e8267789c841c26bf74b1381133c27b0d67b1deaaf18aec4" gracePeriod=30 Oct 30 04:19:10.562854 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.562828 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" Oct 30 04:19:10.629228 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.629195 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" Oct 30 04:19:10.681194 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.681160 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" Oct 30 04:19:10.684588 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.684551 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:19:10.688462 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.688435 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84622b49-99e7-41d2-ad5d-ca8687d3e883-kube-api-access-jx8lt\" (UniqueName: \"kubernetes.io/projected/84622b49-99e7-41d2-ad5d-ca8687d3e883-kube-api-access-jx8lt\") pod \"84622b49-99e7-41d2-ad5d-ca8687d3e883\" (UID: \"84622b49-99e7-41d2-ad5d-ca8687d3e883\") " Oct 30 04:19:10.688624 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.688488 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84622b49-99e7-41d2-ad5d-ca8687d3e883-tls\" (UniqueName: \"kubernetes.io/secret/84622b49-99e7-41d2-ad5d-ca8687d3e883-tls\") pod \"84622b49-99e7-41d2-ad5d-ca8687d3e883\" (UID: \"84622b49-99e7-41d2-ad5d-ca8687d3e883\") " Oct 30 04:19:10.688624 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.688528 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84622b49-99e7-41d2-ad5d-ca8687d3e883-config\" (UniqueName: \"kubernetes.io/configmap/84622b49-99e7-41d2-ad5d-ca8687d3e883-config\") pod \"84622b49-99e7-41d2-ad5d-ca8687d3e883\" (UID: \"84622b49-99e7-41d2-ad5d-ca8687d3e883\") " Oct 30 04:19:10.689003 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.688976 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84622b49-99e7-41d2-ad5d-ca8687d3e883-config" pod "84622b49-99e7-41d2-ad5d-ca8687d3e883" (UID: "84622b49-99e7-41d2-ad5d-ca8687d3e883"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:10.691664 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.691641 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84622b49-99e7-41d2-ad5d-ca8687d3e883-tls" pod "84622b49-99e7-41d2-ad5d-ca8687d3e883" (UID: "84622b49-99e7-41d2-ad5d-ca8687d3e883"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:10.691855 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.691677 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84622b49-99e7-41d2-ad5d-ca8687d3e883-kube-api-access-jx8lt" pod "84622b49-99e7-41d2-ad5d-ca8687d3e883" (UID: "84622b49-99e7-41d2-ad5d-ca8687d3e883"). InnerVolumeSpecName "kube-api-access-jx8lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:10.789516 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.789435 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34e697cb-4277-469b-8c69-5f1a97708d25-kube-api-access-znxwd\" (UniqueName: \"kubernetes.io/projected/34e697cb-4277-469b-8c69-5f1a97708d25-kube-api-access-znxwd\") pod \"34e697cb-4277-469b-8c69-5f1a97708d25\" (UID: \"34e697cb-4277-469b-8c69-5f1a97708d25\") " Oct 30 04:19:10.789516 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.789477 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f0abd3a-9d98-4070-9826-c4121f039945-kube-api-access-zmkhd\" (UniqueName: \"kubernetes.io/projected/1f0abd3a-9d98-4070-9826-c4121f039945-kube-api-access-zmkhd\") pod \"1f0abd3a-9d98-4070-9826-c4121f039945\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " Oct 30 04:19:10.789762 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.789533 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-prometheus-tls-ca\") pod \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\" (UID: \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\") " Oct 30 04:19:10.789762 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.789572 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34e697cb-4277-469b-8c69-5f1a97708d25-tls\" (UniqueName: \"kubernetes.io/secret/34e697cb-4277-469b-8c69-5f1a97708d25-tls\") pod \"34e697cb-4277-469b-8c69-5f1a97708d25\" (UID: \"34e697cb-4277-469b-8c69-5f1a97708d25\") " Oct 30 04:19:10.789762 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.789598 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c876df5-5e2a-4362-8cae-c36d2c7d4637-tls\" (UniqueName: \"kubernetes.io/secret/5c876df5-5e2a-4362-8cae-c36d2c7d4637-tls\") pod \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\" (UID: \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\") " Oct 30 04:19:10.789762 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.789626 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f0abd3a-9d98-4070-9826-c4121f039945-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/1f0abd3a-9d98-4070-9826-c4121f039945-prometheus-ca\") pod \"1f0abd3a-9d98-4070-9826-c4121f039945\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " Oct 30 04:19:10.789762 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.789654 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-tls\" (UniqueName: \"kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-tls\") pod \"1f0abd3a-9d98-4070-9826-c4121f039945\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " Oct 30 04:19:10.789762 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.789685 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-client-ca\") pod \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\" (UID: \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\") " Oct 30 04:19:10.789762 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.789717 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34e697cb-4277-469b-8c69-5f1a97708d25-config\" (UniqueName: \"kubernetes.io/configmap/34e697cb-4277-469b-8c69-5f1a97708d25-config\") pod \"34e697cb-4277-469b-8c69-5f1a97708d25\" (UID: \"34e697cb-4277-469b-8c69-5f1a97708d25\") " Oct 30 04:19:10.790122 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.789815 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-prometheus-client-cert\") pod \"1f0abd3a-9d98-4070-9826-c4121f039945\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " Oct 30 04:19:10.790122 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.789863 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-kube-rbac-proxy-config\") pod \"1f0abd3a-9d98-4070-9826-c4121f039945\" (UID: \"1f0abd3a-9d98-4070-9826-c4121f039945\") " Oct 30 04:19:10.790122 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.790114 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c876df5-5e2a-4362-8cae-c36d2c7d4637-kube-api-access-svvkq\" (UniqueName: \"kubernetes.io/projected/5c876df5-5e2a-4362-8cae-c36d2c7d4637-kube-api-access-svvkq\") pod \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\" (UID: \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\") " Oct 30 04:19:10.790274 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.790153 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-config\" (UniqueName: \"kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-config\") pod \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\" (UID: \"5c876df5-5e2a-4362-8cae-c36d2c7d4637\") " Oct 30 04:19:10.790604 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.790576 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-prometheus-tls-ca" pod "5c876df5-5e2a-4362-8cae-c36d2c7d4637" (UID: "5c876df5-5e2a-4362-8cae-c36d2c7d4637"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:10.790796 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.790776 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx8lt\" (UniqueName: \"kubernetes.io/projected/84622b49-99e7-41d2-ad5d-ca8687d3e883-kube-api-access-jx8lt\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:10.790877 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.790802 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-prometheus-tls-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:10.790877 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.790819 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/84622b49-99e7-41d2-ad5d-ca8687d3e883-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:10.790877 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.790834 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/84622b49-99e7-41d2-ad5d-ca8687d3e883-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:10.791170 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.791141 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f0abd3a-9d98-4070-9826-c4121f039945-prometheus-ca" pod "1f0abd3a-9d98-4070-9826-c4121f039945" (UID: "1f0abd3a-9d98-4070-9826-c4121f039945"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:10.791331 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.791167 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-config" pod "5c876df5-5e2a-4362-8cae-c36d2c7d4637" (UID: "5c876df5-5e2a-4362-8cae-c36d2c7d4637"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:10.791331 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.791299 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-client-ca" pod "5c876df5-5e2a-4362-8cae-c36d2c7d4637" (UID: "5c876df5-5e2a-4362-8cae-c36d2c7d4637"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:10.791635 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.791595 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34e697cb-4277-469b-8c69-5f1a97708d25-config" pod "34e697cb-4277-469b-8c69-5f1a97708d25" (UID: "34e697cb-4277-469b-8c69-5f1a97708d25"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:10.794278 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.794239 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34e697cb-4277-469b-8c69-5f1a97708d25-tls" pod "34e697cb-4277-469b-8c69-5f1a97708d25" (UID: "34e697cb-4277-469b-8c69-5f1a97708d25"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:10.794377 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.794352 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0abd3a-9d98-4070-9826-c4121f039945-kube-api-access-zmkhd" pod "1f0abd3a-9d98-4070-9826-c4121f039945" (UID: "1f0abd3a-9d98-4070-9826-c4121f039945"). InnerVolumeSpecName "kube-api-access-zmkhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:10.794826 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.794484 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-tls" pod "1f0abd3a-9d98-4070-9826-c4121f039945" (UID: "1f0abd3a-9d98-4070-9826-c4121f039945"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:10.794826 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.794593 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c876df5-5e2a-4362-8cae-c36d2c7d4637-kube-api-access-svvkq" pod "5c876df5-5e2a-4362-8cae-c36d2c7d4637" (UID: "5c876df5-5e2a-4362-8cae-c36d2c7d4637"). InnerVolumeSpecName "kube-api-access-svvkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:10.795270 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.795233 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-prometheus-client-cert" pod "1f0abd3a-9d98-4070-9826-c4121f039945" (UID: "1f0abd3a-9d98-4070-9826-c4121f039945"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:10.795354 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.795274 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e697cb-4277-469b-8c69-5f1a97708d25-kube-api-access-znxwd" pod "34e697cb-4277-469b-8c69-5f1a97708d25" (UID: "34e697cb-4277-469b-8c69-5f1a97708d25"). InnerVolumeSpecName "kube-api-access-znxwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:10.796014 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.795992 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-kube-rbac-proxy-config" pod "1f0abd3a-9d98-4070-9826-c4121f039945" (UID: "1f0abd3a-9d98-4070-9826-c4121f039945"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:10.796232 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.796205 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c876df5-5e2a-4362-8cae-c36d2c7d4637-tls" pod "5c876df5-5e2a-4362-8cae-c36d2c7d4637" (UID: "5c876df5-5e2a-4362-8cae-c36d2c7d4637"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:10.820014 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.819979 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:19:10.820295 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.820254 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="64a87316-cfca-4902-b446-225b525fc669" containerName="alertmanager" containerID="cri-o://a5ce3f95a756013281b742137987e6b84cf5d0e13e0ddee140340e18fbe70592" gracePeriod=120 Oct 30 04:19:10.820407 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.820285 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="64a87316-cfca-4902-b446-225b525fc669" containerName="config-reloader" containerID="cri-o://9dfab8fd4fe7449ef6bd29501834f492385fb8b231cfe38a760916c13d2605fe" gracePeriod=120 Oct 30 04:19:10.891268 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.891230 4925 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-kube-rbac-proxy-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:10.891268 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.891261 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svvkq\" (UniqueName: \"kubernetes.io/projected/5c876df5-5e2a-4362-8cae-c36d2c7d4637-kube-api-access-svvkq\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:10.891268 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.891270 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:10.891572 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.891279 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znxwd\" (UniqueName: \"kubernetes.io/projected/34e697cb-4277-469b-8c69-5f1a97708d25-kube-api-access-znxwd\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:10.891572 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.891288 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmkhd\" (UniqueName: \"kubernetes.io/projected/1f0abd3a-9d98-4070-9826-c4121f039945-kube-api-access-zmkhd\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:10.891572 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.891298 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/34e697cb-4277-469b-8c69-5f1a97708d25-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:10.891572 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.891308 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5c876df5-5e2a-4362-8cae-c36d2c7d4637-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:10.891572 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.891316 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/1f0abd3a-9d98-4070-9826-c4121f039945-prometheus-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:10.891572 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.891323 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:10.891572 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.891330 4925 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/5c876df5-5e2a-4362-8cae-c36d2c7d4637-client-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:10.891572 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.891337 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34e697cb-4277-469b-8c69-5f1a97708d25-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:10.891572 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:10.891344 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/1f0abd3a-9d98-4070-9826-c4121f039945-prometheus-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:11.372679 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.372640 4925 generic.go:356] "Generic (PLEG): container finished" podID="1f0abd3a-9d98-4070-9826-c4121f039945" containerID="4399af80091ec075a36d9d0aca89882037631bcd9832126bcd906fbc14ec8d52" exitCode=0 Oct 30 04:19:11.372908 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.372713 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" Oct 30 04:19:11.372908 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.372735 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" event={"ID":"1f0abd3a-9d98-4070-9826-c4121f039945","Type":"ContainerDied","Data":"4399af80091ec075a36d9d0aca89882037631bcd9832126bcd906fbc14ec8d52"} Oct 30 04:19:11.372908 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.372827 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2" event={"ID":"1f0abd3a-9d98-4070-9826-c4121f039945","Type":"ContainerDied","Data":"9aacdb034448b89e4ce0ce2a7e337963f58672e4a91de4fe9ae6bf063ba0dde6"} Oct 30 04:19:11.372908 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.372860 4925 scope.go:122] "RemoveContainer" containerID="4399af80091ec075a36d9d0aca89882037631bcd9832126bcd906fbc14ec8d52" Oct 30 04:19:11.377646 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.374824 4925 generic.go:356] "Generic (PLEG): container finished" podID="84622b49-99e7-41d2-ad5d-ca8687d3e883" containerID="a880e6babc9f7564107144cd397530216823792ecbf05bce7e8d693e90e81607" exitCode=0 Oct 30 04:19:11.377646 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.374845 4925 generic.go:356] "Generic (PLEG): container finished" podID="84622b49-99e7-41d2-ad5d-ca8687d3e883" containerID="f55cedcda59dcccf8d3e12ea9991510eb7058af2f0268b6477e441901fd51832" exitCode=0 Oct 30 04:19:11.377646 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.374916 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" event={"ID":"84622b49-99e7-41d2-ad5d-ca8687d3e883","Type":"ContainerDied","Data":"a880e6babc9f7564107144cd397530216823792ecbf05bce7e8d693e90e81607"} Oct 30 04:19:11.377646 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.374969 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" event={"ID":"84622b49-99e7-41d2-ad5d-ca8687d3e883","Type":"ContainerDied","Data":"f55cedcda59dcccf8d3e12ea9991510eb7058af2f0268b6477e441901fd51832"} Oct 30 04:19:11.377646 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.375002 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" event={"ID":"84622b49-99e7-41d2-ad5d-ca8687d3e883","Type":"ContainerDied","Data":"7771a458db7caedecd57ea97126dfff7e223fa059bda6303fb5094e84fdc0cb8"} Oct 30 04:19:11.377646 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.375104 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6" Oct 30 04:19:11.377955 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.377648 4925 generic.go:356] "Generic (PLEG): container finished" podID="5c876df5-5e2a-4362-8cae-c36d2c7d4637" containerID="5e08478e128ccadd2dab8358fc6f82f9ac3e6642cbb29e5ac762bf6bdca879d7" exitCode=0 Oct 30 04:19:11.377955 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.377664 4925 generic.go:356] "Generic (PLEG): container finished" podID="5c876df5-5e2a-4362-8cae-c36d2c7d4637" containerID="4573c643efeb8f5b258664f07574416e3f0fd4cea963715f5d579428071e3f43" exitCode=0 Oct 30 04:19:11.377955 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.377676 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" event={"ID":"5c876df5-5e2a-4362-8cae-c36d2c7d4637","Type":"ContainerDied","Data":"5e08478e128ccadd2dab8358fc6f82f9ac3e6642cbb29e5ac762bf6bdca879d7"} Oct 30 04:19:11.377955 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.377724 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" Oct 30 04:19:11.377955 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.377736 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" event={"ID":"5c876df5-5e2a-4362-8cae-c36d2c7d4637","Type":"ContainerDied","Data":"4573c643efeb8f5b258664f07574416e3f0fd4cea963715f5d579428071e3f43"} Oct 30 04:19:11.377955 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.377789 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w" event={"ID":"5c876df5-5e2a-4362-8cae-c36d2c7d4637","Type":"ContainerDied","Data":"ab71eb6e1cb6b20b6311d7c502649664ec746cc04921e2214b0ff87e65bfbdb6"} Oct 30 04:19:11.379866 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.379840 4925 generic.go:356] "Generic (PLEG): container finished" podID="34e697cb-4277-469b-8c69-5f1a97708d25" containerID="3bb23270d3c60bf6e8267789c841c26bf74b1381133c27b0d67b1deaaf18aec4" exitCode=0 Oct 30 04:19:11.380008 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.379915 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" Oct 30 04:19:11.380008 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.379977 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" event={"ID":"34e697cb-4277-469b-8c69-5f1a97708d25","Type":"ContainerDied","Data":"3bb23270d3c60bf6e8267789c841c26bf74b1381133c27b0d67b1deaaf18aec4"} Oct 30 04:19:11.380125 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.380018 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd" event={"ID":"34e697cb-4277-469b-8c69-5f1a97708d25","Type":"ContainerDied","Data":"4ddf109ed9581008bf6809dd8fca27aa7c54e90e566a88ff5b88e6f09dedfe93"} Oct 30 04:19:11.382792 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.382775 4925 generic.go:356] "Generic (PLEG): container finished" podID="64a87316-cfca-4902-b446-225b525fc669" containerID="9dfab8fd4fe7449ef6bd29501834f492385fb8b231cfe38a760916c13d2605fe" exitCode=0 Oct 30 04:19:11.382883 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.382829 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"64a87316-cfca-4902-b446-225b525fc669","Type":"ContainerDied","Data":"9dfab8fd4fe7449ef6bd29501834f492385fb8b231cfe38a760916c13d2605fe"} Oct 30 04:19:11.388069 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.388057 4925 scope.go:122] "RemoveContainer" containerID="4399af80091ec075a36d9d0aca89882037631bcd9832126bcd906fbc14ec8d52" Oct 30 04:19:11.388348 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:19:11.388327 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4399af80091ec075a36d9d0aca89882037631bcd9832126bcd906fbc14ec8d52\": container with ID starting with 4399af80091ec075a36d9d0aca89882037631bcd9832126bcd906fbc14ec8d52 not found: ID does not exist" containerID="4399af80091ec075a36d9d0aca89882037631bcd9832126bcd906fbc14ec8d52" Oct 30 04:19:11.388401 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.388356 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4399af80091ec075a36d9d0aca89882037631bcd9832126bcd906fbc14ec8d52"} err="failed to get container status \"4399af80091ec075a36d9d0aca89882037631bcd9832126bcd906fbc14ec8d52\": rpc error: code = NotFound desc = could not find container \"4399af80091ec075a36d9d0aca89882037631bcd9832126bcd906fbc14ec8d52\": container with ID starting with 4399af80091ec075a36d9d0aca89882037631bcd9832126bcd906fbc14ec8d52 not found: ID does not exist" Oct 30 04:19:11.388401 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.388373 4925 scope.go:122] "RemoveContainer" containerID="a880e6babc9f7564107144cd397530216823792ecbf05bce7e8d693e90e81607" Oct 30 04:19:11.396072 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.396055 4925 scope.go:122] "RemoveContainer" containerID="f55cedcda59dcccf8d3e12ea9991510eb7058af2f0268b6477e441901fd51832" Oct 30 04:19:11.403658 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.403643 4925 scope.go:122] "RemoveContainer" containerID="a880e6babc9f7564107144cd397530216823792ecbf05bce7e8d693e90e81607" Oct 30 04:19:11.403950 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:19:11.403926 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a880e6babc9f7564107144cd397530216823792ecbf05bce7e8d693e90e81607\": container with ID starting with a880e6babc9f7564107144cd397530216823792ecbf05bce7e8d693e90e81607 not found: ID does not exist" containerID="a880e6babc9f7564107144cd397530216823792ecbf05bce7e8d693e90e81607" Oct 30 04:19:11.404031 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.403956 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a880e6babc9f7564107144cd397530216823792ecbf05bce7e8d693e90e81607"} err="failed to get container status \"a880e6babc9f7564107144cd397530216823792ecbf05bce7e8d693e90e81607\": rpc error: code = NotFound desc = could not find container \"a880e6babc9f7564107144cd397530216823792ecbf05bce7e8d693e90e81607\": container with ID starting with a880e6babc9f7564107144cd397530216823792ecbf05bce7e8d693e90e81607 not found: ID does not exist" Oct 30 04:19:11.404031 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.403974 4925 scope.go:122] "RemoveContainer" containerID="f55cedcda59dcccf8d3e12ea9991510eb7058af2f0268b6477e441901fd51832" Oct 30 04:19:11.404240 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:19:11.404220 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f55cedcda59dcccf8d3e12ea9991510eb7058af2f0268b6477e441901fd51832\": container with ID starting with f55cedcda59dcccf8d3e12ea9991510eb7058af2f0268b6477e441901fd51832 not found: ID does not exist" containerID="f55cedcda59dcccf8d3e12ea9991510eb7058af2f0268b6477e441901fd51832" Oct 30 04:19:11.404302 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.404249 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f55cedcda59dcccf8d3e12ea9991510eb7058af2f0268b6477e441901fd51832"} err="failed to get container status \"f55cedcda59dcccf8d3e12ea9991510eb7058af2f0268b6477e441901fd51832\": rpc error: code = NotFound desc = could not find container \"f55cedcda59dcccf8d3e12ea9991510eb7058af2f0268b6477e441901fd51832\": container with ID starting with f55cedcda59dcccf8d3e12ea9991510eb7058af2f0268b6477e441901fd51832 not found: ID does not exist" Oct 30 04:19:11.404302 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.404270 4925 scope.go:122] "RemoveContainer" containerID="a880e6babc9f7564107144cd397530216823792ecbf05bce7e8d693e90e81607" Oct 30 04:19:11.404497 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.404477 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a880e6babc9f7564107144cd397530216823792ecbf05bce7e8d693e90e81607"} err="failed to get container status \"a880e6babc9f7564107144cd397530216823792ecbf05bce7e8d693e90e81607\": rpc error: code = NotFound desc = could not find container \"a880e6babc9f7564107144cd397530216823792ecbf05bce7e8d693e90e81607\": container with ID starting with a880e6babc9f7564107144cd397530216823792ecbf05bce7e8d693e90e81607 not found: ID does not exist" Oct 30 04:19:11.404542 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.404498 4925 scope.go:122] "RemoveContainer" containerID="f55cedcda59dcccf8d3e12ea9991510eb7058af2f0268b6477e441901fd51832" Oct 30 04:19:11.404692 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.404673 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f55cedcda59dcccf8d3e12ea9991510eb7058af2f0268b6477e441901fd51832"} err="failed to get container status \"f55cedcda59dcccf8d3e12ea9991510eb7058af2f0268b6477e441901fd51832\": rpc error: code = NotFound desc = could not find container \"f55cedcda59dcccf8d3e12ea9991510eb7058af2f0268b6477e441901fd51832\": container with ID starting with f55cedcda59dcccf8d3e12ea9991510eb7058af2f0268b6477e441901fd51832 not found: ID does not exist" Oct 30 04:19:11.404790 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.404693 4925 scope.go:122] "RemoveContainer" containerID="5e08478e128ccadd2dab8358fc6f82f9ac3e6642cbb29e5ac762bf6bdca879d7" Oct 30 04:19:11.412412 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.412393 4925 scope.go:122] "RemoveContainer" containerID="4573c643efeb8f5b258664f07574416e3f0fd4cea963715f5d579428071e3f43" Oct 30 04:19:11.419806 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.419787 4925 scope.go:122] "RemoveContainer" containerID="5e08478e128ccadd2dab8358fc6f82f9ac3e6642cbb29e5ac762bf6bdca879d7" Oct 30 04:19:11.420073 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:19:11.420053 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e08478e128ccadd2dab8358fc6f82f9ac3e6642cbb29e5ac762bf6bdca879d7\": container with ID starting with 5e08478e128ccadd2dab8358fc6f82f9ac3e6642cbb29e5ac762bf6bdca879d7 not found: ID does not exist" containerID="5e08478e128ccadd2dab8358fc6f82f9ac3e6642cbb29e5ac762bf6bdca879d7" Oct 30 04:19:11.420115 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.420080 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e08478e128ccadd2dab8358fc6f82f9ac3e6642cbb29e5ac762bf6bdca879d7"} err="failed to get container status \"5e08478e128ccadd2dab8358fc6f82f9ac3e6642cbb29e5ac762bf6bdca879d7\": rpc error: code = NotFound desc = could not find container \"5e08478e128ccadd2dab8358fc6f82f9ac3e6642cbb29e5ac762bf6bdca879d7\": container with ID starting with 5e08478e128ccadd2dab8358fc6f82f9ac3e6642cbb29e5ac762bf6bdca879d7 not found: ID does not exist" Oct 30 04:19:11.420115 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.420097 4925 scope.go:122] "RemoveContainer" containerID="4573c643efeb8f5b258664f07574416e3f0fd4cea963715f5d579428071e3f43" Oct 30 04:19:11.420343 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:19:11.420324 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4573c643efeb8f5b258664f07574416e3f0fd4cea963715f5d579428071e3f43\": container with ID starting with 4573c643efeb8f5b258664f07574416e3f0fd4cea963715f5d579428071e3f43 not found: ID does not exist" containerID="4573c643efeb8f5b258664f07574416e3f0fd4cea963715f5d579428071e3f43" Oct 30 04:19:11.420383 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.420348 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4573c643efeb8f5b258664f07574416e3f0fd4cea963715f5d579428071e3f43"} err="failed to get container status \"4573c643efeb8f5b258664f07574416e3f0fd4cea963715f5d579428071e3f43\": rpc error: code = NotFound desc = could not find container \"4573c643efeb8f5b258664f07574416e3f0fd4cea963715f5d579428071e3f43\": container with ID starting with 4573c643efeb8f5b258664f07574416e3f0fd4cea963715f5d579428071e3f43 not found: ID does not exist" Oct 30 04:19:11.420383 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.420362 4925 scope.go:122] "RemoveContainer" containerID="5e08478e128ccadd2dab8358fc6f82f9ac3e6642cbb29e5ac762bf6bdca879d7" Oct 30 04:19:11.420577 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.420559 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e08478e128ccadd2dab8358fc6f82f9ac3e6642cbb29e5ac762bf6bdca879d7"} err="failed to get container status \"5e08478e128ccadd2dab8358fc6f82f9ac3e6642cbb29e5ac762bf6bdca879d7\": rpc error: code = NotFound desc = could not find container \"5e08478e128ccadd2dab8358fc6f82f9ac3e6642cbb29e5ac762bf6bdca879d7\": container with ID starting with 5e08478e128ccadd2dab8358fc6f82f9ac3e6642cbb29e5ac762bf6bdca879d7 not found: ID does not exist" Oct 30 04:19:11.420621 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.420578 4925 scope.go:122] "RemoveContainer" containerID="4573c643efeb8f5b258664f07574416e3f0fd4cea963715f5d579428071e3f43" Oct 30 04:19:11.420842 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.420821 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4573c643efeb8f5b258664f07574416e3f0fd4cea963715f5d579428071e3f43"} err="failed to get container status \"4573c643efeb8f5b258664f07574416e3f0fd4cea963715f5d579428071e3f43\": rpc error: code = NotFound desc = could not find container \"4573c643efeb8f5b258664f07574416e3f0fd4cea963715f5d579428071e3f43\": container with ID starting with 4573c643efeb8f5b258664f07574416e3f0fd4cea963715f5d579428071e3f43 not found: ID does not exist" Oct 30 04:19:11.420842 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.420842 4925 scope.go:122] "RemoveContainer" containerID="3bb23270d3c60bf6e8267789c841c26bf74b1381133c27b0d67b1deaaf18aec4" Oct 30 04:19:11.424344 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.424322 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd"] Oct 30 04:19:11.429740 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.429711 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-nndsd"] Oct 30 04:19:11.429831 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.429814 4925 scope.go:122] "RemoveContainer" containerID="3bb23270d3c60bf6e8267789c841c26bf74b1381133c27b0d67b1deaaf18aec4" Oct 30 04:19:11.430083 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:19:11.430066 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb23270d3c60bf6e8267789c841c26bf74b1381133c27b0d67b1deaaf18aec4\": container with ID starting with 3bb23270d3c60bf6e8267789c841c26bf74b1381133c27b0d67b1deaaf18aec4 not found: ID does not exist" containerID="3bb23270d3c60bf6e8267789c841c26bf74b1381133c27b0d67b1deaaf18aec4" Oct 30 04:19:11.430139 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.430087 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb23270d3c60bf6e8267789c841c26bf74b1381133c27b0d67b1deaaf18aec4"} err="failed to get container status \"3bb23270d3c60bf6e8267789c841c26bf74b1381133c27b0d67b1deaaf18aec4\": rpc error: code = NotFound desc = could not find container \"3bb23270d3c60bf6e8267789c841c26bf74b1381133c27b0d67b1deaaf18aec4\": container with ID starting with 3bb23270d3c60bf6e8267789c841c26bf74b1381133c27b0d67b1deaaf18aec4 not found: ID does not exist" Oct 30 04:19:11.462858 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.462829 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6"] Oct 30 04:19:11.466900 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.466880 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-x47m6"] Oct 30 04:19:11.506076 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.506044 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w"] Oct 30 04:19:11.510497 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.510474 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-kwp9w"] Oct 30 04:19:11.546190 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.546159 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2"] Oct 30 04:19:11.551035 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.551010 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-7nxf2"] Oct 30 04:19:11.907784 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.907733 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:19:11.908157 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:11.907928 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="0b7a2e4b-34b7-4abb-a186-0f8573153dc1" containerName="otc-container" containerID="cri-o://009ff41c8ed17076457a9f27ff8a291424cb10042b866bed122f3e7eb2909964" gracePeriod=30 Oct 30 04:19:12.177563 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.177539 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:12.304369 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.304337 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-tls-certs\") pod \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\" (UID: \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\") " Oct 30 04:19:12.304556 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.304397 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-otc-internal\" (UniqueName: \"kubernetes.io/configmap/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-otc-internal\") pod \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\" (UID: \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\") " Oct 30 04:19:12.304556 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.304504 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-kube-api-access-n7f6q\" (UniqueName: \"kubernetes.io/projected/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-kube-api-access-n7f6q\") pod \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\" (UID: \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\") " Oct 30 04:19:12.304556 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.304540 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-monitoring-tls-certs\") pod \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\" (UID: \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\") " Oct 30 04:19:12.304718 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.304568 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-data-science-collector-ta-client-cert\") pod \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\" (UID: \"0b7a2e4b-34b7-4abb-a186-0f8573153dc1\") " Oct 30 04:19:12.304785 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.304727 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-otc-internal" pod "0b7a2e4b-34b7-4abb-a186-0f8573153dc1" (UID: "0b7a2e4b-34b7-4abb-a186-0f8573153dc1"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:12.305154 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.304910 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:12.306616 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.306590 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-monitoring-tls-certs" pod "0b7a2e4b-34b7-4abb-a186-0f8573153dc1" (UID: "0b7a2e4b-34b7-4abb-a186-0f8573153dc1"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:12.306739 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.306713 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-data-science-collector-ta-client-cert" pod "0b7a2e4b-34b7-4abb-a186-0f8573153dc1" (UID: "0b7a2e4b-34b7-4abb-a186-0f8573153dc1"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:12.307027 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.307007 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-kube-api-access-n7f6q" pod "0b7a2e4b-34b7-4abb-a186-0f8573153dc1" (UID: "0b7a2e4b-34b7-4abb-a186-0f8573153dc1"). InnerVolumeSpecName "kube-api-access-n7f6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:12.307130 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.307053 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-tls-certs" pod "0b7a2e4b-34b7-4abb-a186-0f8573153dc1" (UID: "0b7a2e4b-34b7-4abb-a186-0f8573153dc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:12.391315 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.391280 4925 generic.go:356] "Generic (PLEG): container finished" podID="0b7a2e4b-34b7-4abb-a186-0f8573153dc1" containerID="009ff41c8ed17076457a9f27ff8a291424cb10042b866bed122f3e7eb2909964" exitCode=0 Oct 30 04:19:12.391467 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.391338 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:12.391467 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.391362 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0b7a2e4b-34b7-4abb-a186-0f8573153dc1","Type":"ContainerDied","Data":"009ff41c8ed17076457a9f27ff8a291424cb10042b866bed122f3e7eb2909964"} Oct 30 04:19:12.391467 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.391416 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0b7a2e4b-34b7-4abb-a186-0f8573153dc1","Type":"ContainerDied","Data":"5972d809dbb45a1536d0209695d89f267f26de8aa7fade150940f837d23ab18c"} Oct 30 04:19:12.391467 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.391444 4925 scope.go:122] "RemoveContainer" containerID="009ff41c8ed17076457a9f27ff8a291424cb10042b866bed122f3e7eb2909964" Oct 30 04:19:12.400915 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.400893 4925 scope.go:122] "RemoveContainer" containerID="009ff41c8ed17076457a9f27ff8a291424cb10042b866bed122f3e7eb2909964" Oct 30 04:19:12.401199 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:19:12.401178 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"009ff41c8ed17076457a9f27ff8a291424cb10042b866bed122f3e7eb2909964\": container with ID starting with 009ff41c8ed17076457a9f27ff8a291424cb10042b866bed122f3e7eb2909964 not found: ID does not exist" containerID="009ff41c8ed17076457a9f27ff8a291424cb10042b866bed122f3e7eb2909964" Oct 30 04:19:12.401264 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.401209 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"009ff41c8ed17076457a9f27ff8a291424cb10042b866bed122f3e7eb2909964"} err="failed to get container status \"009ff41c8ed17076457a9f27ff8a291424cb10042b866bed122f3e7eb2909964\": rpc error: code = NotFound desc = could not find container \"009ff41c8ed17076457a9f27ff8a291424cb10042b866bed122f3e7eb2909964\": container with ID starting with 009ff41c8ed17076457a9f27ff8a291424cb10042b866bed122f3e7eb2909964 not found: ID does not exist" Oct 30 04:19:12.405578 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.405554 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:12.405578 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.405578 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7f6q\" (UniqueName: \"kubernetes.io/projected/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-kube-api-access-n7f6q\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:12.405719 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.405588 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:12.405719 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.405597 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0b7a2e4b-34b7-4abb-a186-0f8573153dc1-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:12.445701 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.445628 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:19:12.455382 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:12.455356 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:19:13.119135 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:13.119102 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b7a2e4b-34b7-4abb-a186-0f8573153dc1" path="/var/lib/kubelet/pods/0b7a2e4b-34b7-4abb-a186-0f8573153dc1/volumes" Oct 30 04:19:13.119503 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:13.119473 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f0abd3a-9d98-4070-9826-c4121f039945" path="/var/lib/kubelet/pods/1f0abd3a-9d98-4070-9826-c4121f039945/volumes" Oct 30 04:19:13.119853 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:13.119839 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34e697cb-4277-469b-8c69-5f1a97708d25" path="/var/lib/kubelet/pods/34e697cb-4277-469b-8c69-5f1a97708d25/volumes" Oct 30 04:19:13.120486 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:13.120472 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c876df5-5e2a-4362-8cae-c36d2c7d4637" path="/var/lib/kubelet/pods/5c876df5-5e2a-4362-8cae-c36d2c7d4637/volumes" Oct 30 04:19:13.120913 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:13.120900 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84622b49-99e7-41d2-ad5d-ca8687d3e883" path="/var/lib/kubelet/pods/84622b49-99e7-41d2-ad5d-ca8687d3e883/volumes" Oct 30 04:19:15.215280 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:15.215240 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="64a87316-cfca-4902-b446-225b525fc669" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.146:9093/-/ready\": dial tcp 10.132.0.146:9093: connect: connection refused" Oct 30 04:19:18.162044 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:19:18.162008 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:19:20.215347 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:20.215305 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="64a87316-cfca-4902-b446-225b525fc669" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.146:9093/-/ready\": dial tcp 10.132.0.146:9093: connect: connection refused" Oct 30 04:19:21.065765 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.065727 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:21.073685 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.073665 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-web-config\" (UniqueName: \"kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-web-config\") pod \"64a87316-cfca-4902-b446-225b525fc669\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " Oct 30 04:19:21.073762 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.073701 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a87316-cfca-4902-b446-225b525fc669-kube-api-access-dg8ps\" (UniqueName: \"kubernetes.io/projected/64a87316-cfca-4902-b446-225b525fc669-kube-api-access-dg8ps\") pod \"64a87316-cfca-4902-b446-225b525fc669\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " Oct 30 04:19:21.073840 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.073773 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a87316-cfca-4902-b446-225b525fc669-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/64a87316-cfca-4902-b446-225b525fc669-alertmanager-data-science-monitoringstack-db\") pod \"64a87316-cfca-4902-b446-225b525fc669\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " Oct 30 04:19:21.073840 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.073806 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-config-volume\" (UniqueName: \"kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-config-volume\") pod \"64a87316-cfca-4902-b446-225b525fc669\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " Oct 30 04:19:21.073955 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.073852 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-cluster-tls-config\") pod \"64a87316-cfca-4902-b446-225b525fc669\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " Oct 30 04:19:21.073955 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.073884 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a87316-cfca-4902-b446-225b525fc669-config-out\" (UniqueName: \"kubernetes.io/empty-dir/64a87316-cfca-4902-b446-225b525fc669-config-out\") pod \"64a87316-cfca-4902-b446-225b525fc669\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " Oct 30 04:19:21.073955 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.073924 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a87316-cfca-4902-b446-225b525fc669-tls-assets\" (UniqueName: \"kubernetes.io/projected/64a87316-cfca-4902-b446-225b525fc669-tls-assets\") pod \"64a87316-cfca-4902-b446-225b525fc669\" (UID: \"64a87316-cfca-4902-b446-225b525fc669\") " Oct 30 04:19:21.074113 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.074077 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a87316-cfca-4902-b446-225b525fc669-alertmanager-data-science-monitoringstack-db" pod "64a87316-cfca-4902-b446-225b525fc669" (UID: "64a87316-cfca-4902-b446-225b525fc669"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:19:21.074206 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.074185 4925 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/64a87316-cfca-4902-b446-225b525fc669-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:21.075884 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.075859 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-config-volume" pod "64a87316-cfca-4902-b446-225b525fc669" (UID: "64a87316-cfca-4902-b446-225b525fc669"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:21.076238 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.076214 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a87316-cfca-4902-b446-225b525fc669-kube-api-access-dg8ps" pod "64a87316-cfca-4902-b446-225b525fc669" (UID: "64a87316-cfca-4902-b446-225b525fc669"). InnerVolumeSpecName "kube-api-access-dg8ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:21.076395 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.076369 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a87316-cfca-4902-b446-225b525fc669-tls-assets" pod "64a87316-cfca-4902-b446-225b525fc669" (UID: "64a87316-cfca-4902-b446-225b525fc669"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:21.076495 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.076418 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a87316-cfca-4902-b446-225b525fc669-config-out" pod "64a87316-cfca-4902-b446-225b525fc669" (UID: "64a87316-cfca-4902-b446-225b525fc669"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:19:21.080234 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.080176 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-cluster-tls-config" pod "64a87316-cfca-4902-b446-225b525fc669" (UID: "64a87316-cfca-4902-b446-225b525fc669"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:21.087824 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.087797 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-web-config" pod "64a87316-cfca-4902-b446-225b525fc669" (UID: "64a87316-cfca-4902-b446-225b525fc669"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:21.174720 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.174685 4925 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-config-volume\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:21.174720 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.174713 4925 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-cluster-tls-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:21.174720 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.174723 4925 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64a87316-cfca-4902-b446-225b525fc669-config-out\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:21.174960 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.174731 4925 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64a87316-cfca-4902-b446-225b525fc669-tls-assets\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:21.174960 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.174777 4925 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64a87316-cfca-4902-b446-225b525fc669-web-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:21.174960 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.174785 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg8ps\" (UniqueName: \"kubernetes.io/projected/64a87316-cfca-4902-b446-225b525fc669-kube-api-access-dg8ps\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:19:21.437557 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.437475 4925 generic.go:356] "Generic (PLEG): container finished" podID="64a87316-cfca-4902-b446-225b525fc669" containerID="a5ce3f95a756013281b742137987e6b84cf5d0e13e0ddee140340e18fbe70592" exitCode=0 Oct 30 04:19:21.437557 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.437536 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:21.438139 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.437566 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"64a87316-cfca-4902-b446-225b525fc669","Type":"ContainerDied","Data":"a5ce3f95a756013281b742137987e6b84cf5d0e13e0ddee140340e18fbe70592"} Oct 30 04:19:21.438139 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.437637 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"64a87316-cfca-4902-b446-225b525fc669","Type":"ContainerDied","Data":"97fa820383513be7fefdf99a46188b2ffe8acb01e804f20b7194a36e99030246"} Oct 30 04:19:21.438139 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.437680 4925 scope.go:122] "RemoveContainer" containerID="9dfab8fd4fe7449ef6bd29501834f492385fb8b231cfe38a760916c13d2605fe" Oct 30 04:19:21.446079 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.446058 4925 scope.go:122] "RemoveContainer" containerID="a5ce3f95a756013281b742137987e6b84cf5d0e13e0ddee140340e18fbe70592" Oct 30 04:19:21.453947 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.453928 4925 scope.go:122] "RemoveContainer" containerID="41d9d7b3b5a8774583048e4f5682c240ba9f92f42c029d6ad70a887834b6a953" Oct 30 04:19:21.461484 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.461464 4925 scope.go:122] "RemoveContainer" containerID="9dfab8fd4fe7449ef6bd29501834f492385fb8b231cfe38a760916c13d2605fe" Oct 30 04:19:21.461769 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:19:21.461732 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dfab8fd4fe7449ef6bd29501834f492385fb8b231cfe38a760916c13d2605fe\": container with ID starting with 9dfab8fd4fe7449ef6bd29501834f492385fb8b231cfe38a760916c13d2605fe not found: ID does not exist" containerID="9dfab8fd4fe7449ef6bd29501834f492385fb8b231cfe38a760916c13d2605fe" Oct 30 04:19:21.461822 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.461771 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dfab8fd4fe7449ef6bd29501834f492385fb8b231cfe38a760916c13d2605fe"} err="failed to get container status \"9dfab8fd4fe7449ef6bd29501834f492385fb8b231cfe38a760916c13d2605fe\": rpc error: code = NotFound desc = could not find container \"9dfab8fd4fe7449ef6bd29501834f492385fb8b231cfe38a760916c13d2605fe\": container with ID starting with 9dfab8fd4fe7449ef6bd29501834f492385fb8b231cfe38a760916c13d2605fe not found: ID does not exist" Oct 30 04:19:21.461822 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.461788 4925 scope.go:122] "RemoveContainer" containerID="a5ce3f95a756013281b742137987e6b84cf5d0e13e0ddee140340e18fbe70592" Oct 30 04:19:21.462047 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:19:21.462027 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5ce3f95a756013281b742137987e6b84cf5d0e13e0ddee140340e18fbe70592\": container with ID starting with a5ce3f95a756013281b742137987e6b84cf5d0e13e0ddee140340e18fbe70592 not found: ID does not exist" containerID="a5ce3f95a756013281b742137987e6b84cf5d0e13e0ddee140340e18fbe70592" Oct 30 04:19:21.462097 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.462053 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5ce3f95a756013281b742137987e6b84cf5d0e13e0ddee140340e18fbe70592"} err="failed to get container status \"a5ce3f95a756013281b742137987e6b84cf5d0e13e0ddee140340e18fbe70592\": rpc error: code = NotFound desc = could not find container \"a5ce3f95a756013281b742137987e6b84cf5d0e13e0ddee140340e18fbe70592\": container with ID starting with a5ce3f95a756013281b742137987e6b84cf5d0e13e0ddee140340e18fbe70592 not found: ID does not exist" Oct 30 04:19:21.462097 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.462073 4925 scope.go:122] "RemoveContainer" containerID="41d9d7b3b5a8774583048e4f5682c240ba9f92f42c029d6ad70a887834b6a953" Oct 30 04:19:21.462256 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:19:21.462242 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d9d7b3b5a8774583048e4f5682c240ba9f92f42c029d6ad70a887834b6a953\": container with ID starting with 41d9d7b3b5a8774583048e4f5682c240ba9f92f42c029d6ad70a887834b6a953 not found: ID does not exist" containerID="41d9d7b3b5a8774583048e4f5682c240ba9f92f42c029d6ad70a887834b6a953" Oct 30 04:19:21.462293 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.462260 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d9d7b3b5a8774583048e4f5682c240ba9f92f42c029d6ad70a887834b6a953"} err="failed to get container status \"41d9d7b3b5a8774583048e4f5682c240ba9f92f42c029d6ad70a887834b6a953\": rpc error: code = NotFound desc = could not find container \"41d9d7b3b5a8774583048e4f5682c240ba9f92f42c029d6ad70a887834b6a953\": container with ID starting with 41d9d7b3b5a8774583048e4f5682c240ba9f92f42c029d6ad70a887834b6a953 not found: ID does not exist" Oct 30 04:19:21.491923 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.491896 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:19:21.504538 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:21.504514 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:19:23.124304 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:23.124260 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64a87316-cfca-4902-b446-225b525fc669" path="/var/lib/kubelet/pods/64a87316-cfca-4902-b446-225b525fc669/volumes" Oct 30 04:19:27.093688 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.093653 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87"] Oct 30 04:19:27.094091 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094008 4925 cpu_manager.go:414] "containerMap: removing container" podUID="1f0abd3a-9d98-4070-9826-c4121f039945" containerName="kube-rbac-proxy" Oct 30 04:19:27.094091 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094021 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0abd3a-9d98-4070-9826-c4121f039945" containerName="kube-rbac-proxy" Oct 30 04:19:27.094091 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094031 4925 cpu_manager.go:414] "containerMap: removing container" podUID="34e697cb-4277-469b-8c69-5f1a97708d25" containerName="korrel8r" Oct 30 04:19:27.094091 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094036 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34e697cb-4277-469b-8c69-5f1a97708d25" containerName="korrel8r" Oct 30 04:19:27.094091 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094046 4925 cpu_manager.go:414] "containerMap: removing container" podUID="64a87316-cfca-4902-b446-225b525fc669" containerName="config-reloader" Oct 30 04:19:27.094091 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094051 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a87316-cfca-4902-b446-225b525fc669" containerName="config-reloader" Oct 30 04:19:27.094091 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094060 4925 cpu_manager.go:414] "containerMap: removing container" podUID="84622b49-99e7-41d2-ad5d-ca8687d3e883" containerName="kube-rbac-proxy" Oct 30 04:19:27.094091 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094064 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84622b49-99e7-41d2-ad5d-ca8687d3e883" containerName="kube-rbac-proxy" Oct 30 04:19:27.094091 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094072 4925 cpu_manager.go:414] "containerMap: removing container" podUID="64a87316-cfca-4902-b446-225b525fc669" containerName="init-config-reloader" Oct 30 04:19:27.094091 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094076 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a87316-cfca-4902-b446-225b525fc669" containerName="init-config-reloader" Oct 30 04:19:27.094091 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094086 4925 cpu_manager.go:414] "containerMap: removing container" podUID="64a87316-cfca-4902-b446-225b525fc669" containerName="alertmanager" Oct 30 04:19:27.094091 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094090 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a87316-cfca-4902-b446-225b525fc669" containerName="alertmanager" Oct 30 04:19:27.094091 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094098 4925 cpu_manager.go:414] "containerMap: removing container" podUID="5c876df5-5e2a-4362-8cae-c36d2c7d4637" containerName="kube-rbac-proxy" Oct 30 04:19:27.094582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094103 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c876df5-5e2a-4362-8cae-c36d2c7d4637" containerName="kube-rbac-proxy" Oct 30 04:19:27.094582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094111 4925 cpu_manager.go:414] "containerMap: removing container" podUID="84622b49-99e7-41d2-ad5d-ca8687d3e883" containerName="prom-label-proxy" Oct 30 04:19:27.094582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094115 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84622b49-99e7-41d2-ad5d-ca8687d3e883" containerName="prom-label-proxy" Oct 30 04:19:27.094582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094131 4925 cpu_manager.go:414] "containerMap: removing container" podUID="0b7a2e4b-34b7-4abb-a186-0f8573153dc1" containerName="otc-container" Oct 30 04:19:27.094582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094135 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7a2e4b-34b7-4abb-a186-0f8573153dc1" containerName="otc-container" Oct 30 04:19:27.094582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094142 4925 cpu_manager.go:414] "containerMap: removing container" podUID="5c876df5-5e2a-4362-8cae-c36d2c7d4637" containerName="prom-label-proxy" Oct 30 04:19:27.094582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094146 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c876df5-5e2a-4362-8cae-c36d2c7d4637" containerName="prom-label-proxy" Oct 30 04:19:27.094582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094191 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="84622b49-99e7-41d2-ad5d-ca8687d3e883" containerName="kube-rbac-proxy" Oct 30 04:19:27.094582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094200 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a87316-cfca-4902-b446-225b525fc669" containerName="alertmanager" Oct 30 04:19:27.094582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094207 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c876df5-5e2a-4362-8cae-c36d2c7d4637" containerName="kube-rbac-proxy" Oct 30 04:19:27.094582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094213 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="84622b49-99e7-41d2-ad5d-ca8687d3e883" containerName="prom-label-proxy" Oct 30 04:19:27.094582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094220 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7a2e4b-34b7-4abb-a186-0f8573153dc1" containerName="otc-container" Oct 30 04:19:27.094582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094226 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c876df5-5e2a-4362-8cae-c36d2c7d4637" containerName="prom-label-proxy" Oct 30 04:19:27.094582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094232 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0abd3a-9d98-4070-9826-c4121f039945" containerName="kube-rbac-proxy" Oct 30 04:19:27.094582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094237 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="34e697cb-4277-469b-8c69-5f1a97708d25" containerName="korrel8r" Oct 30 04:19:27.094582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.094242 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a87316-cfca-4902-b446-225b525fc669" containerName="config-reloader" Oct 30 04:19:27.099089 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.099069 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:19:27.114430 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.114406 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 04:19:27.123630 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.123576 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/2cf03981-68cb-48af-8f02-edb980939be8-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-wcz87\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:19:27.123630 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.123610 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-wcz87\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:19:27.123823 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.123636 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l26q9\" (UniqueName: \"kubernetes.io/projected/2cf03981-68cb-48af-8f02-edb980939be8-kube-api-access-l26q9\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-wcz87\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:19:27.123877 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.123847 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-wcz87\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:19:27.123945 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.123897 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-wcz87\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:19:27.127241 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.127217 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-jhgfg\"" Oct 30 04:19:27.127241 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.127230 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 04:19:27.127241 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.127220 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 04:19:27.127475 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.127268 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 04:19:27.127475 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.127283 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 04:19:27.127475 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.127295 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 04:19:27.129290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.129268 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87"] Oct 30 04:19:27.135073 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.135052 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7"] Oct 30 04:19:27.138511 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.138495 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" Oct 30 04:19:27.159684 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.159646 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7"] Oct 30 04:19:27.173048 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.173017 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t"] Oct 30 04:19:27.176736 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.176711 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" Oct 30 04:19:27.196524 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.196498 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t"] Oct 30 04:19:27.214715 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.214684 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 04:19:27.214890 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.214805 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-68mjl\"" Oct 30 04:19:27.214951 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.214885 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 04:19:27.214951 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.214936 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-txsmm\"" Oct 30 04:19:27.215457 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.215436 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 04:19:27.215532 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.215490 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 04:19:27.215532 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.215498 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 04:19:27.216627 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.216610 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts"] Oct 30 04:19:27.220101 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.220083 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" Oct 30 04:19:27.225230 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.225211 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/18c4211b-62ec-40ee-9c20-600a3617ef96-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-gnc7t\" (UID: \"18c4211b-62ec-40ee-9c20-600a3617ef96\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" Oct 30 04:19:27.225344 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.225245 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-wcz87\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:19:27.225344 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.225273 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-bj7h7\" (UID: \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" Oct 30 04:19:27.225449 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.225413 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-bj7h7\" (UID: \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" Oct 30 04:19:27.225497 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.225483 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/2cf03981-68cb-48af-8f02-edb980939be8-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-wcz87\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:19:27.225554 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.225505 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-wcz87\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:19:27.225609 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.225528 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgbg8\" (UniqueName: \"kubernetes.io/projected/18c4211b-62ec-40ee-9c20-600a3617ef96-kube-api-access-kgbg8\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-gnc7t\" (UID: \"18c4211b-62ec-40ee-9c20-600a3617ef96\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" Oct 30 04:19:27.225609 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.225584 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l26q9\" (UniqueName: \"kubernetes.io/projected/2cf03981-68cb-48af-8f02-edb980939be8-kube-api-access-l26q9\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-wcz87\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:19:27.225609 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:19:27.225600 4925 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 04:19:27.225802 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:19:27.225684 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-prometheus-client-cert podName:2cf03981-68cb-48af-8f02-edb980939be8 nodeName:}" failed. No retries permitted until 2026-10-30 04:19:27.72566174 +0000 UTC m=+5928.738939837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-wcz87" (UID: "2cf03981-68cb-48af-8f02-edb980939be8") : secret "prometheus-operated-tls" not found Oct 30 04:19:27.226096 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.226068 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-bj7h7\" (UID: \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" Oct 30 04:19:27.226187 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.226127 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/2cf03981-68cb-48af-8f02-edb980939be8-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-wcz87\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:19:27.226187 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.226146 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-bj7h7\" (UID: \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" Oct 30 04:19:27.226324 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.226211 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqrxg\" (UniqueName: \"kubernetes.io/projected/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-kube-api-access-zqrxg\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-bj7h7\" (UID: \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" Oct 30 04:19:27.226324 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.226264 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-wcz87\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:19:27.226324 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.226283 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/18c4211b-62ec-40ee-9c20-600a3617ef96-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-gnc7t\" (UID: \"18c4211b-62ec-40ee-9c20-600a3617ef96\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" Oct 30 04:19:27.228192 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.228167 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-wcz87\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:19:27.228459 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.228442 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-wcz87\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:19:27.241925 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.241895 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts"] Oct 30 04:19:27.250726 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.250697 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l26q9\" (UniqueName: \"kubernetes.io/projected/2cf03981-68cb-48af-8f02-edb980939be8-kube-api-access-l26q9\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-wcz87\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:19:27.314877 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.314845 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 04:19:27.314877 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.314851 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-zff99\"" Oct 30 04:19:27.315648 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.315630 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 04:19:27.327018 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.326994 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbg8\" (UniqueName: \"kubernetes.io/projected/18c4211b-62ec-40ee-9c20-600a3617ef96-kube-api-access-kgbg8\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-gnc7t\" (UID: \"18c4211b-62ec-40ee-9c20-600a3617ef96\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" Oct 30 04:19:27.327113 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.327037 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-bj7h7\" (UID: \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" Oct 30 04:19:27.327113 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.327059 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-bj7h7\" (UID: \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" Oct 30 04:19:27.327113 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.327093 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqrxg\" (UniqueName: \"kubernetes.io/projected/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-kube-api-access-zqrxg\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-bj7h7\" (UID: \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" Oct 30 04:19:27.327302 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.327123 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b270c6f6-8593-4237-8877-b9f2ac197942-config\") pod \"korrel8r-5cbcc89b8f-g56ts\" (UID: \"b270c6f6-8593-4237-8877-b9f2ac197942\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" Oct 30 04:19:27.327302 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.327145 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b270c6f6-8593-4237-8877-b9f2ac197942-tls\") pod \"korrel8r-5cbcc89b8f-g56ts\" (UID: \"b270c6f6-8593-4237-8877-b9f2ac197942\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" Oct 30 04:19:27.327421 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.327332 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/18c4211b-62ec-40ee-9c20-600a3617ef96-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-gnc7t\" (UID: \"18c4211b-62ec-40ee-9c20-600a3617ef96\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" Oct 30 04:19:27.327421 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.327361 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/18c4211b-62ec-40ee-9c20-600a3617ef96-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-gnc7t\" (UID: \"18c4211b-62ec-40ee-9c20-600a3617ef96\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" Oct 30 04:19:27.327421 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.327382 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-bj7h7\" (UID: \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" Oct 30 04:19:27.327421 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.327412 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-bj7h7\" (UID: \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" Oct 30 04:19:27.327625 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.327449 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp897\" (UniqueName: \"kubernetes.io/projected/b270c6f6-8593-4237-8877-b9f2ac197942-kube-api-access-wp897\") pod \"korrel8r-5cbcc89b8f-g56ts\" (UID: \"b270c6f6-8593-4237-8877-b9f2ac197942\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" Oct 30 04:19:27.327798 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.327743 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-bj7h7\" (UID: \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" Oct 30 04:19:27.327922 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.327902 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-bj7h7\" (UID: \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" Oct 30 04:19:27.328159 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.328138 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-bj7h7\" (UID: \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" Oct 30 04:19:27.328207 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.328139 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/18c4211b-62ec-40ee-9c20-600a3617ef96-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-gnc7t\" (UID: \"18c4211b-62ec-40ee-9c20-600a3617ef96\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" Oct 30 04:19:27.330049 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.330025 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-bj7h7\" (UID: \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" Oct 30 04:19:27.330132 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.330025 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/18c4211b-62ec-40ee-9c20-600a3617ef96-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-gnc7t\" (UID: \"18c4211b-62ec-40ee-9c20-600a3617ef96\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" Oct 30 04:19:27.351130 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.351071 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqrxg\" (UniqueName: \"kubernetes.io/projected/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-kube-api-access-zqrxg\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-bj7h7\" (UID: \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" Oct 30 04:19:27.351130 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.351111 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgbg8\" (UniqueName: \"kubernetes.io/projected/18c4211b-62ec-40ee-9c20-600a3617ef96-kube-api-access-kgbg8\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-gnc7t\" (UID: \"18c4211b-62ec-40ee-9c20-600a3617ef96\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" Oct 30 04:19:27.428151 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.428112 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b270c6f6-8593-4237-8877-b9f2ac197942-config\") pod \"korrel8r-5cbcc89b8f-g56ts\" (UID: \"b270c6f6-8593-4237-8877-b9f2ac197942\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" Oct 30 04:19:27.428151 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.428156 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b270c6f6-8593-4237-8877-b9f2ac197942-tls\") pod \"korrel8r-5cbcc89b8f-g56ts\" (UID: \"b270c6f6-8593-4237-8877-b9f2ac197942\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" Oct 30 04:19:27.428417 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.428199 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp897\" (UniqueName: \"kubernetes.io/projected/b270c6f6-8593-4237-8877-b9f2ac197942-kube-api-access-wp897\") pod \"korrel8r-5cbcc89b8f-g56ts\" (UID: \"b270c6f6-8593-4237-8877-b9f2ac197942\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" Oct 30 04:19:27.429052 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.429029 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b270c6f6-8593-4237-8877-b9f2ac197942-config\") pod \"korrel8r-5cbcc89b8f-g56ts\" (UID: \"b270c6f6-8593-4237-8877-b9f2ac197942\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" Oct 30 04:19:27.430660 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.430638 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b270c6f6-8593-4237-8877-b9f2ac197942-tls\") pod \"korrel8r-5cbcc89b8f-g56ts\" (UID: \"b270c6f6-8593-4237-8877-b9f2ac197942\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" Oct 30 04:19:27.454227 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.454199 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" Oct 30 04:19:27.455155 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.455129 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp897\" (UniqueName: \"kubernetes.io/projected/b270c6f6-8593-4237-8877-b9f2ac197942-kube-api-access-wp897\") pod \"korrel8r-5cbcc89b8f-g56ts\" (UID: \"b270c6f6-8593-4237-8877-b9f2ac197942\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" Oct 30 04:19:27.491872 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.491829 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" Oct 30 04:19:27.548978 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.548946 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" Oct 30 04:19:27.730510 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.730470 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-wcz87\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:19:27.732833 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.732814 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-wcz87\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:19:27.759025 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.758628 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7"] Oct 30 04:19:27.761215 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:19:27.761187 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfa3bc8_85a6_4387_98fe_bf3d3fdeb91d.slice/crio-73fff250b2764c708c30313735f62316c816b3c31ad4751db3010422ef006b47 WatchSource:0}: Error finding container 73fff250b2764c708c30313735f62316c816b3c31ad4751db3010422ef006b47: Status 404 returned error can't find the container with id 73fff250b2764c708c30313735f62316c816b3c31ad4751db3010422ef006b47 Oct 30 04:19:27.804963 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.804934 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t"] Oct 30 04:19:27.806805 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:19:27.806776 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c4211b_62ec_40ee_9c20_600a3617ef96.slice/crio-687cf0ddf65c5fc3b5409d1ecde9ff0187b74e8f2de0e46eb9fdc4283d6e718e WatchSource:0}: Error finding container 687cf0ddf65c5fc3b5409d1ecde9ff0187b74e8f2de0e46eb9fdc4283d6e718e: Status 404 returned error can't find the container with id 687cf0ddf65c5fc3b5409d1ecde9ff0187b74e8f2de0e46eb9fdc4283d6e718e Oct 30 04:19:27.854903 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:27.854867 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts"] Oct 30 04:19:27.857667 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:19:27.857630 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb270c6f6_8593_4237_8877_b9f2ac197942.slice/crio-51ecc3d8698fb9d4e4de91cea5be39eafaf220d6470fff7f7ece8b2baf535b64 WatchSource:0}: Error finding container 51ecc3d8698fb9d4e4de91cea5be39eafaf220d6470fff7f7ece8b2baf535b64: Status 404 returned error can't find the container with id 51ecc3d8698fb9d4e4de91cea5be39eafaf220d6470fff7f7ece8b2baf535b64 Oct 30 04:19:28.011879 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:28.011847 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:19:28.311973 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:28.311891 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87"] Oct 30 04:19:28.315339 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:19:28.315303 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf03981_68cb_48af_8f02_edb980939be8.slice/crio-1e8d570e02b59d9bd16d2aa60f988ca76df2447646468ae0142eb78181e6c60d WatchSource:0}: Error finding container 1e8d570e02b59d9bd16d2aa60f988ca76df2447646468ae0142eb78181e6c60d: Status 404 returned error can't find the container with id 1e8d570e02b59d9bd16d2aa60f988ca76df2447646468ae0142eb78181e6c60d Oct 30 04:19:28.473740 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:28.473699 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" event={"ID":"2cf03981-68cb-48af-8f02-edb980939be8","Type":"ContainerStarted","Data":"1e8d570e02b59d9bd16d2aa60f988ca76df2447646468ae0142eb78181e6c60d"} Oct 30 04:19:28.475463 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:28.475438 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" event={"ID":"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d","Type":"ContainerStarted","Data":"7270a63289e020b0f3e32c7a9cc183c4715f05cfda5a78b29e40a8d1b59f5ebc"} Oct 30 04:19:28.475614 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:28.475492 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" event={"ID":"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d","Type":"ContainerStarted","Data":"5711fbc6867bc94782d06ec391ec52d22a6acf306dd44f35f35a535ebdbd3bd8"} Oct 30 04:19:28.475614 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:28.475515 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" event={"ID":"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d","Type":"ContainerStarted","Data":"73fff250b2764c708c30313735f62316c816b3c31ad4751db3010422ef006b47"} Oct 30 04:19:28.477150 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:28.477123 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" event={"ID":"18c4211b-62ec-40ee-9c20-600a3617ef96","Type":"ContainerStarted","Data":"fbc41e6c25fcddb29c31a6d085bcd9c92b44a1be7ef36188c0abecafdbd3d0cb"} Oct 30 04:19:28.477263 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:28.477176 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" event={"ID":"18c4211b-62ec-40ee-9c20-600a3617ef96","Type":"ContainerStarted","Data":"7d06c5963ab4664a7bb642e3a7801be08a5ad758acfd77c892ffb7a92bed7923"} Oct 30 04:19:28.477263 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:28.477206 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" event={"ID":"18c4211b-62ec-40ee-9c20-600a3617ef96","Type":"ContainerStarted","Data":"687cf0ddf65c5fc3b5409d1ecde9ff0187b74e8f2de0e46eb9fdc4283d6e718e"} Oct 30 04:19:28.478612 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:28.478593 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" event={"ID":"b270c6f6-8593-4237-8877-b9f2ac197942","Type":"ContainerStarted","Data":"20adf80be713a5a5b4b6d313773de09a3108574148e9c5b368e204bf1564565a"} Oct 30 04:19:28.478717 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:28.478632 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" event={"ID":"b270c6f6-8593-4237-8877-b9f2ac197942","Type":"ContainerStarted","Data":"51ecc3d8698fb9d4e4de91cea5be39eafaf220d6470fff7f7ece8b2baf535b64"} Oct 30 04:19:28.478822 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:28.478808 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" Oct 30 04:19:28.518330 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:28.518278 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" podStartSLOduration=1.5182654580000001 podStartE2EDuration="1.518265458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:28.516838103 +0000 UTC m=+5929.530116208" watchObservedRunningTime="2026-10-30 04:19:28.518265458 +0000 UTC m=+5929.531543560" Oct 30 04:19:28.560178 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:28.560121 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" podStartSLOduration=1.5601078259999999 podStartE2EDuration="1.560107826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:28.559001469 +0000 UTC m=+5929.572279572" watchObservedRunningTime="2026-10-30 04:19:28.560107826 +0000 UTC m=+5929.573385923" Oct 30 04:19:28.601789 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:28.601723 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" podStartSLOduration=1.6017113520000001 podStartE2EDuration="1.601711352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:28.600637991 +0000 UTC m=+5929.613916087" watchObservedRunningTime="2026-10-30 04:19:28.601711352 +0000 UTC m=+5929.614989455" Oct 30 04:19:29.484926 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:29.484882 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" event={"ID":"2cf03981-68cb-48af-8f02-edb980939be8","Type":"ContainerStarted","Data":"9f01a5dc208c9becb7055638b3ba6d1b4fccd7f63bf7af625494a12073bca437"} Oct 30 04:19:29.485590 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:29.485569 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:19:29.531097 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:29.531038 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" podStartSLOduration=2.5310236919999998 podStartE2EDuration="2.531023692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:29.529388194 +0000 UTC m=+5930.542666330" watchObservedRunningTime="2026-10-30 04:19:29.531023692 +0000 UTC m=+5930.544301827" Oct 30 04:19:32.376009 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.375966 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:19:32.380385 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.380353 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.414023 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.413985 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:19:32.470287 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.470252 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:19:32.470449 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.470433 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-5km5j\"" Oct 30 04:19:32.470501 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.470478 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 04:19:32.470501 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.470488 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 04:19:32.470593 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.470501 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 04:19:32.477995 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.477969 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06218436-4431-4f18-a5c6-9871ac950890-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.478098 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.478031 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.478098 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.478074 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2p4c\" (UniqueName: \"kubernetes.io/projected/06218436-4431-4f18-a5c6-9871ac950890-kube-api-access-x2p4c\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.478228 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.478161 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.478288 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.478272 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.478350 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.478310 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/06218436-4431-4f18-a5c6-9871ac950890-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.478404 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.478375 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06218436-4431-4f18-a5c6-9871ac950890-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.579116 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.579081 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06218436-4431-4f18-a5c6-9871ac950890-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.579116 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.579116 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06218436-4431-4f18-a5c6-9871ac950890-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.579439 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.579136 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.579439 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.579169 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2p4c\" (UniqueName: \"kubernetes.io/projected/06218436-4431-4f18-a5c6-9871ac950890-kube-api-access-x2p4c\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.579439 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.579189 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.579439 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.579245 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.579439 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.579269 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/06218436-4431-4f18-a5c6-9871ac950890-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.579672 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.579638 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/06218436-4431-4f18-a5c6-9871ac950890-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.581813 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.581784 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.581974 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.581950 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.582040 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.582022 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.582082 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.582064 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06218436-4431-4f18-a5c6-9871ac950890-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.582121 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.582091 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06218436-4431-4f18-a5c6-9871ac950890-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.603197 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.603168 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2p4c\" (UniqueName: \"kubernetes.io/projected/06218436-4431-4f18-a5c6-9871ac950890-kube-api-access-x2p4c\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:32.702536 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:32.702439 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:33.034212 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:33.034050 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:19:33.036869 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:19:33.036842 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06218436_4431_4f18_a5c6_9871ac950890.slice/crio-43f48a37017384bd49e4b23b831c085acebc95af2e8a327db39a86889326b826 WatchSource:0}: Error finding container 43f48a37017384bd49e4b23b831c085acebc95af2e8a327db39a86889326b826: Status 404 returned error can't find the container with id 43f48a37017384bd49e4b23b831c085acebc95af2e8a327db39a86889326b826 Oct 30 04:19:33.506828 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:33.506785 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"06218436-4431-4f18-a5c6-9871ac950890","Type":"ContainerStarted","Data":"3ddcaec80adcdbe1ffdb1d6d9da3e14468b8e74651d6a917308ba7558b5bd5b8"} Oct 30 04:19:33.507256 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:33.506860 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"06218436-4431-4f18-a5c6-9871ac950890","Type":"ContainerStarted","Data":"43f48a37017384bd49e4b23b831c085acebc95af2e8a327db39a86889326b826"} Oct 30 04:19:36.497803 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:36.497768 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:19:37.528226 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:37.528139 4925 generic.go:356] "Generic (PLEG): container finished" podID="06218436-4431-4f18-a5c6-9871ac950890" containerID="3ddcaec80adcdbe1ffdb1d6d9da3e14468b8e74651d6a917308ba7558b5bd5b8" exitCode=0 Oct 30 04:19:37.528226 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:37.528198 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"06218436-4431-4f18-a5c6-9871ac950890","Type":"ContainerDied","Data":"3ddcaec80adcdbe1ffdb1d6d9da3e14468b8e74651d6a917308ba7558b5bd5b8"} Oct 30 04:19:38.535116 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:38.535083 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"06218436-4431-4f18-a5c6-9871ac950890","Type":"ContainerStarted","Data":"e673a3dc702de87a8ed417eb51443807d2b3f0f72e6bf902679b0cdcc2124e64"} Oct 30 04:19:38.535497 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:38.535145 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"06218436-4431-4f18-a5c6-9871ac950890","Type":"ContainerStarted","Data":"9c0fc4a7f73ede2bf840ee5d37114159824ebe0e282ea18adefc6997d8792cee"} Oct 30 04:19:38.535497 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:38.535316 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:38.613163 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:38.613104 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.828380905 podStartE2EDuration="6.613090592s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.784709687s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:38.599608807 +0000 UTC m=+5939.612886910" watchObservedRunningTime="2026-10-30 04:19:38.613090592 +0000 UTC m=+5939.626368692" Oct 30 04:19:39.487339 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:39.487301 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" Oct 30 04:19:44.542599 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:44.542564 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:19:48.162104 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:19:48.162065 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:19:52.114897 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.114856 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:19:52.120114 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.120091 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:52.162148 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.162103 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:19:52.162386 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.162360 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:19:52.162468 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.162390 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-wvqg5\"" Oct 30 04:19:52.162468 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.162406 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:19:52.162468 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.162418 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:19:52.164374 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.164351 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:19:52.260224 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.260190 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbl26\" (UniqueName: \"kubernetes.io/projected/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-kube-api-access-dbl26\") pod \"data-science-collector-collector-1\" (UID: \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:52.260398 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.260232 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:52.260398 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.260352 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:52.260532 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.260405 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:52.260532 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.260505 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:52.361825 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.361793 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:52.362030 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.361839 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:52.362030 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.361943 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbl26\" (UniqueName: \"kubernetes.io/projected/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-kube-api-access-dbl26\") pod \"data-science-collector-collector-1\" (UID: \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:52.362030 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.361970 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:52.362205 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.362028 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:52.362808 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.362778 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:52.364657 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.364623 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:52.364657 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.364646 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:52.364873 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.364665 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:52.385567 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.385496 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbl26\" (UniqueName: \"kubernetes.io/projected/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-kube-api-access-dbl26\") pod \"data-science-collector-collector-1\" (UID: \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:52.434454 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.434421 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:19:52.733536 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.733508 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:19:52.736177 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:19:52.736154 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0f865f_e5dc_4ca8_840f_8056af28b8a6.slice/crio-95529dbd9e592bd31a373b6821501ff1e88b39f2bd051ece26dc7a5879217253 WatchSource:0}: Error finding container 95529dbd9e592bd31a373b6821501ff1e88b39f2bd051ece26dc7a5879217253: Status 404 returned error can't find the container with id 95529dbd9e592bd31a373b6821501ff1e88b39f2bd051ece26dc7a5879217253 Oct 30 04:19:52.738239 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.738207 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:19:52.738329 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:52.738265 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:19:53.609081 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:53.609037 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0d0f865f-e5dc-4ca8-840f-8056af28b8a6","Type":"ContainerStarted","Data":"9f60b06ac390ad836ec5f6ee55a34bd8dfecb753eee06dd6f0ce998cb5ac5e7e"} Oct 30 04:19:53.609475 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:53.609106 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0d0f865f-e5dc-4ca8-840f-8056af28b8a6","Type":"ContainerStarted","Data":"95529dbd9e592bd31a373b6821501ff1e88b39f2bd051ece26dc7a5879217253"} Oct 30 04:19:53.653029 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:19:53.652970 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.652955358 podStartE2EDuration="1.652955358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:53.650961017 +0000 UTC m=+5954.664239120" watchObservedRunningTime="2026-10-30 04:19:53.652955358 +0000 UTC m=+5954.666233463" Oct 30 04:20:05.446593 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:05.446556 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:20:05.447086 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:05.446737 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="0d0f865f-e5dc-4ca8-840f-8056af28b8a6" containerName="otc-container" containerID="cri-o://9f60b06ac390ad836ec5f6ee55a34bd8dfecb753eee06dd6f0ce998cb5ac5e7e" gracePeriod=30 Oct 30 04:20:06.683029 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:06.683000 4925 generic.go:356] "Generic (PLEG): container finished" podID="0d0f865f-e5dc-4ca8-840f-8056af28b8a6" containerID="9f60b06ac390ad836ec5f6ee55a34bd8dfecb753eee06dd6f0ce998cb5ac5e7e" exitCode=0 Oct 30 04:20:06.683339 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:06.683046 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0d0f865f-e5dc-4ca8-840f-8056af28b8a6","Type":"ContainerDied","Data":"9f60b06ac390ad836ec5f6ee55a34bd8dfecb753eee06dd6f0ce998cb5ac5e7e"} Oct 30 04:20:06.683339 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:06.683114 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0d0f865f-e5dc-4ca8-840f-8056af28b8a6","Type":"ContainerDied","Data":"95529dbd9e592bd31a373b6821501ff1e88b39f2bd051ece26dc7a5879217253"} Oct 30 04:20:06.683339 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:06.683132 4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95529dbd9e592bd31a373b6821501ff1e88b39f2bd051ece26dc7a5879217253" Oct 30 04:20:06.694470 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:06.694447 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:06.791875 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:06.791769 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-tls-certs\") pod \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\" (UID: \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\") " Oct 30 04:20:06.791875 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:06.791817 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-data-science-collector-ta-client-cert\") pod \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\" (UID: \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\") " Oct 30 04:20:06.792119 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:06.791881 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-monitoring-tls-certs\") pod \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\" (UID: \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\") " Oct 30 04:20:06.792119 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:06.791908 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-otc-internal\" (UniqueName: \"kubernetes.io/configmap/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-otc-internal\") pod \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\" (UID: \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\") " Oct 30 04:20:06.792119 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:06.791929 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-kube-api-access-dbl26\" (UniqueName: \"kubernetes.io/projected/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-kube-api-access-dbl26\") pod \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\" (UID: \"0d0f865f-e5dc-4ca8-840f-8056af28b8a6\") " Oct 30 04:20:06.792366 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:06.792319 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-otc-internal" pod "0d0f865f-e5dc-4ca8-840f-8056af28b8a6" (UID: "0d0f865f-e5dc-4ca8-840f-8056af28b8a6"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:06.794355 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:06.794327 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-kube-api-access-dbl26" pod "0d0f865f-e5dc-4ca8-840f-8056af28b8a6" (UID: "0d0f865f-e5dc-4ca8-840f-8056af28b8a6"). InnerVolumeSpecName "kube-api-access-dbl26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:06.794506 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:06.794481 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-monitoring-tls-certs" pod "0d0f865f-e5dc-4ca8-840f-8056af28b8a6" (UID: "0d0f865f-e5dc-4ca8-840f-8056af28b8a6"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:06.794582 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:06.794495 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-tls-certs" pod "0d0f865f-e5dc-4ca8-840f-8056af28b8a6" (UID: "0d0f865f-e5dc-4ca8-840f-8056af28b8a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:06.794648 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:06.794607 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-data-science-collector-ta-client-cert" pod "0d0f865f-e5dc-4ca8-840f-8056af28b8a6" (UID: "0d0f865f-e5dc-4ca8-840f-8056af28b8a6"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:06.893574 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:06.893532 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:06.893574 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:06.893566 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:06.893574 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:06.893576 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:06.893850 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:06.893586 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:06.893850 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:06.893595 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbl26\" (UniqueName: \"kubernetes.io/projected/0d0f865f-e5dc-4ca8-840f-8056af28b8a6-kube-api-access-dbl26\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:07.688055 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:07.688022 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:07.739680 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:07.739647 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:20:07.743944 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:07.743917 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:20:07.781958 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:07.781923 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:20:07.782297 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:07.782285 4925 cpu_manager.go:414] "containerMap: removing container" podUID="0d0f865f-e5dc-4ca8-840f-8056af28b8a6" containerName="otc-container" Oct 30 04:20:07.782336 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:07.782298 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d0f865f-e5dc-4ca8-840f-8056af28b8a6" containerName="otc-container" Oct 30 04:20:07.782386 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:07.782376 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d0f865f-e5dc-4ca8-840f-8056af28b8a6" containerName="otc-container" Oct 30 04:20:07.785763 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:07.785726 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:07.801606 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:07.801579 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-wvqg5\"" Oct 30 04:20:07.801606 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:07.801579 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:20:07.801778 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:07.801621 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:20:07.801778 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:07.801613 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:20:07.801778 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:07.801644 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:20:07.807286 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:07.807259 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:20:07.902322 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:07.902283 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d50428f2-cc0e-49d8-a542-31715459a591\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:07.902502 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:07.902337 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d50428f2-cc0e-49d8-a542-31715459a591\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:07.902502 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:07.902414 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktd5g\" (UniqueName: \"kubernetes.io/projected/d50428f2-cc0e-49d8-a542-31715459a591-kube-api-access-ktd5g\") pod \"data-science-collector-collector-1\" (UID: \"d50428f2-cc0e-49d8-a542-31715459a591\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:07.902502 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:07.902449 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d50428f2-cc0e-49d8-a542-31715459a591-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d50428f2-cc0e-49d8-a542-31715459a591\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:07.902502 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:07.902467 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d50428f2-cc0e-49d8-a542-31715459a591\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:08.003814 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:08.003710 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d50428f2-cc0e-49d8-a542-31715459a591\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:08.003814 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:08.003786 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d50428f2-cc0e-49d8-a542-31715459a591\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:08.003814 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:08.003809 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktd5g\" (UniqueName: \"kubernetes.io/projected/d50428f2-cc0e-49d8-a542-31715459a591-kube-api-access-ktd5g\") pod \"data-science-collector-collector-1\" (UID: \"d50428f2-cc0e-49d8-a542-31715459a591\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:08.004028 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:08.003827 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d50428f2-cc0e-49d8-a542-31715459a591-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d50428f2-cc0e-49d8-a542-31715459a591\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:08.004028 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:08.003842 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d50428f2-cc0e-49d8-a542-31715459a591\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:08.004598 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:08.004572 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d50428f2-cc0e-49d8-a542-31715459a591-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d50428f2-cc0e-49d8-a542-31715459a591\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:08.006327 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:08.006284 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d50428f2-cc0e-49d8-a542-31715459a591\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:08.006506 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:08.006481 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d50428f2-cc0e-49d8-a542-31715459a591\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:08.006577 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:08.006523 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d50428f2-cc0e-49d8-a542-31715459a591\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:08.035632 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:08.035603 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktd5g\" (UniqueName: \"kubernetes.io/projected/d50428f2-cc0e-49d8-a542-31715459a591-kube-api-access-ktd5g\") pod \"data-science-collector-collector-1\" (UID: \"d50428f2-cc0e-49d8-a542-31715459a591\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:08.099054 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:08.099021 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:08.403331 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:08.403302 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:20:08.405003 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:20:08.404971 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50428f2_cc0e_49d8_a542_31715459a591.slice/crio-e1c1e312dc3e75a272cc303cac924bcc80605319c3218c13426657e10544e1be WatchSource:0}: Error finding container e1c1e312dc3e75a272cc303cac924bcc80605319c3218c13426657e10544e1be: Status 404 returned error can't find the container with id e1c1e312dc3e75a272cc303cac924bcc80605319c3218c13426657e10544e1be Oct 30 04:20:08.407060 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:08.407029 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:20:08.407150 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:08.407087 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:20:08.695361 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:08.695268 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d50428f2-cc0e-49d8-a542-31715459a591","Type":"ContainerStarted","Data":"ebad9d1ee78244ed7ef623c17e6190eea43dbe35ad85a3c1031a8c03c6b05653"} Oct 30 04:20:08.695361 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:08.695328 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d50428f2-cc0e-49d8-a542-31715459a591","Type":"ContainerStarted","Data":"e1c1e312dc3e75a272cc303cac924bcc80605319c3218c13426657e10544e1be"} Oct 30 04:20:08.739588 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:08.739541 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.739527679 podStartE2EDuration="1.739527679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:08.738304299 +0000 UTC m=+5969.751582400" watchObservedRunningTime="2026-10-30 04:20:08.739527679 +0000 UTC m=+5969.752805781" Oct 30 04:20:09.119373 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:09.119334 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d0f865f-e5dc-4ca8-840f-8056af28b8a6" path="/var/lib/kubelet/pods/0d0f865f-e5dc-4ca8-840f-8056af28b8a6/volumes" Oct 30 04:20:09.973296 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:09.973260 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87"] Oct 30 04:20:09.973734 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:09.973467 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" podUID="2cf03981-68cb-48af-8f02-edb980939be8" containerName="kube-rbac-proxy" containerID="cri-o://9f01a5dc208c9becb7055638b3ba6d1b4fccd7f63bf7af625494a12073bca437" gracePeriod=30 Oct 30 04:20:09.989025 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:09.989000 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7"] Oct 30 04:20:09.989225 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:09.989205 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" podUID="7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d" containerName="kube-rbac-proxy" containerID="cri-o://5711fbc6867bc94782d06ec391ec52d22a6acf306dd44f35f35a535ebdbd3bd8" gracePeriod=30 Oct 30 04:20:09.989225 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:09.989212 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" podUID="7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d" containerName="prom-label-proxy" containerID="cri-o://7270a63289e020b0f3e32c7a9cc183c4715f05cfda5a78b29e40a8d1b59f5ebc" gracePeriod=30 Oct 30 04:20:10.003238 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.003207 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t"] Oct 30 04:20:10.003495 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.003470 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" podUID="18c4211b-62ec-40ee-9c20-600a3617ef96" containerName="kube-rbac-proxy" containerID="cri-o://7d06c5963ab4664a7bb642e3a7801be08a5ad758acfd77c892ffb7a92bed7923" gracePeriod=30 Oct 30 04:20:10.003599 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.003477 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" podUID="18c4211b-62ec-40ee-9c20-600a3617ef96" containerName="prom-label-proxy" containerID="cri-o://fbc41e6c25fcddb29c31a6d085bcd9c92b44a1be7ef36188c0abecafdbd3d0cb" gracePeriod=30 Oct 30 04:20:10.025236 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.024479 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts"] Oct 30 04:20:10.025236 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.024809 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" podUID="b270c6f6-8593-4237-8877-b9f2ac197942" containerName="korrel8r" containerID="cri-o://20adf80be713a5a5b4b6d313773de09a3108574148e9c5b368e204bf1564565a" gracePeriod=30 Oct 30 04:20:10.293718 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.293686 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" Oct 30 04:20:10.332695 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.322463 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18c4211b-62ec-40ee-9c20-600a3617ef96-kube-api-access-kgbg8\" (UniqueName: \"kubernetes.io/projected/18c4211b-62ec-40ee-9c20-600a3617ef96-kube-api-access-kgbg8\") pod \"18c4211b-62ec-40ee-9c20-600a3617ef96\" (UID: \"18c4211b-62ec-40ee-9c20-600a3617ef96\") " Oct 30 04:20:10.332695 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.322587 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18c4211b-62ec-40ee-9c20-600a3617ef96-config\" (UniqueName: \"kubernetes.io/configmap/18c4211b-62ec-40ee-9c20-600a3617ef96-config\") pod \"18c4211b-62ec-40ee-9c20-600a3617ef96\" (UID: \"18c4211b-62ec-40ee-9c20-600a3617ef96\") " Oct 30 04:20:10.332695 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.322630 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18c4211b-62ec-40ee-9c20-600a3617ef96-tls\" (UniqueName: \"kubernetes.io/secret/18c4211b-62ec-40ee-9c20-600a3617ef96-tls\") pod \"18c4211b-62ec-40ee-9c20-600a3617ef96\" (UID: \"18c4211b-62ec-40ee-9c20-600a3617ef96\") " Oct 30 04:20:10.332695 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.324020 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c4211b-62ec-40ee-9c20-600a3617ef96-config" pod "18c4211b-62ec-40ee-9c20-600a3617ef96" (UID: "18c4211b-62ec-40ee-9c20-600a3617ef96"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:10.332695 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.328052 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18c4211b-62ec-40ee-9c20-600a3617ef96-tls" pod "18c4211b-62ec-40ee-9c20-600a3617ef96" (UID: "18c4211b-62ec-40ee-9c20-600a3617ef96"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:10.339570 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.339532 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c4211b-62ec-40ee-9c20-600a3617ef96-kube-api-access-kgbg8" pod "18c4211b-62ec-40ee-9c20-600a3617ef96" (UID: "18c4211b-62ec-40ee-9c20-600a3617ef96"). InnerVolumeSpecName "kube-api-access-kgbg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:10.355233 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.355194 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" Oct 30 04:20:10.360115 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.360087 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" Oct 30 04:20:10.362615 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.362591 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:20:10.424375 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.424345 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b270c6f6-8593-4237-8877-b9f2ac197942-kube-api-access-wp897\" (UniqueName: \"kubernetes.io/projected/b270c6f6-8593-4237-8877-b9f2ac197942-kube-api-access-wp897\") pod \"b270c6f6-8593-4237-8877-b9f2ac197942\" (UID: \"b270c6f6-8593-4237-8877-b9f2ac197942\") " Oct 30 04:20:10.424549 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.424386 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf03981-68cb-48af-8f02-edb980939be8-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/2cf03981-68cb-48af-8f02-edb980939be8-prometheus-ca\") pod \"2cf03981-68cb-48af-8f02-edb980939be8\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " Oct 30 04:20:10.424549 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.424425 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-kube-api-access-zqrxg\" (UniqueName: \"kubernetes.io/projected/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-kube-api-access-zqrxg\") pod \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\" (UID: \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\") " Oct 30 04:20:10.424862 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.424598 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-tls\" (UniqueName: \"kubernetes.io/secret/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-tls\") pod \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\" (UID: \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\") " Oct 30 04:20:10.424862 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.424830 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-config\" (UniqueName: \"kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-config\") pod \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\" (UID: \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\") " Oct 30 04:20:10.424862 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.424854 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf03981-68cb-48af-8f02-edb980939be8-prometheus-ca" pod "2cf03981-68cb-48af-8f02-edb980939be8" (UID: "2cf03981-68cb-48af-8f02-edb980939be8"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:10.425155 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.425111 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-kube-rbac-proxy-config\") pod \"2cf03981-68cb-48af-8f02-edb980939be8\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " Oct 30 04:20:10.425314 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.425258 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-client-ca\") pod \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\" (UID: \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\") " Oct 30 04:20:10.425407 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.425343 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-config" pod "7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d" (UID: "7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:10.425562 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.425294 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-tls\" (UniqueName: \"kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-tls\") pod \"2cf03981-68cb-48af-8f02-edb980939be8\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " Oct 30 04:20:10.426408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.425539 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-prometheus-client-cert\") pod \"2cf03981-68cb-48af-8f02-edb980939be8\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " Oct 30 04:20:10.426408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.425715 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-prometheus-tls-ca\") pod \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\" (UID: \"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d\") " Oct 30 04:20:10.426408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.425801 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf03981-68cb-48af-8f02-edb980939be8-kube-api-access-l26q9\" (UniqueName: \"kubernetes.io/projected/2cf03981-68cb-48af-8f02-edb980939be8-kube-api-access-l26q9\") pod \"2cf03981-68cb-48af-8f02-edb980939be8\" (UID: \"2cf03981-68cb-48af-8f02-edb980939be8\") " Oct 30 04:20:10.426408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.425841 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b270c6f6-8593-4237-8877-b9f2ac197942-config\" (UniqueName: \"kubernetes.io/configmap/b270c6f6-8593-4237-8877-b9f2ac197942-config\") pod \"b270c6f6-8593-4237-8877-b9f2ac197942\" (UID: \"b270c6f6-8593-4237-8877-b9f2ac197942\") " Oct 30 04:20:10.426408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.425881 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b270c6f6-8593-4237-8877-b9f2ac197942-tls\" (UniqueName: \"kubernetes.io/secret/b270c6f6-8593-4237-8877-b9f2ac197942-tls\") pod \"b270c6f6-8593-4237-8877-b9f2ac197942\" (UID: \"b270c6f6-8593-4237-8877-b9f2ac197942\") " Oct 30 04:20:10.426408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.425584 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-client-ca" pod "7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d" (UID: "7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:10.426408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.426168 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgbg8\" (UniqueName: \"kubernetes.io/projected/18c4211b-62ec-40ee-9c20-600a3617ef96-kube-api-access-kgbg8\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.426408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.426187 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/18c4211b-62ec-40ee-9c20-600a3617ef96-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.426408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.426201 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/18c4211b-62ec-40ee-9c20-600a3617ef96-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.426408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.426213 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/2cf03981-68cb-48af-8f02-edb980939be8-prometheus-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.426408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.426242 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-prometheus-tls-ca" pod "7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d" (UID: "7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:10.426408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.426253 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.426408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.426266 4925 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-client-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.427391 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.426627 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b270c6f6-8593-4237-8877-b9f2ac197942-config" pod "b270c6f6-8593-4237-8877-b9f2ac197942" (UID: "b270c6f6-8593-4237-8877-b9f2ac197942"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:10.427891 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.427852 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b270c6f6-8593-4237-8877-b9f2ac197942-kube-api-access-wp897" pod "b270c6f6-8593-4237-8877-b9f2ac197942" (UID: "b270c6f6-8593-4237-8877-b9f2ac197942"). InnerVolumeSpecName "kube-api-access-wp897". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:10.428818 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.428780 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-tls" pod "2cf03981-68cb-48af-8f02-edb980939be8" (UID: "2cf03981-68cb-48af-8f02-edb980939be8"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:10.428920 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.428839 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-kube-api-access-zqrxg" pod "7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d" (UID: "7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d"). InnerVolumeSpecName "kube-api-access-zqrxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:10.430181 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.430156 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-tls" pod "7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d" (UID: "7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:10.431366 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.431318 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-kube-rbac-proxy-config" pod "2cf03981-68cb-48af-8f02-edb980939be8" (UID: "2cf03981-68cb-48af-8f02-edb980939be8"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:10.432294 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.432270 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b270c6f6-8593-4237-8877-b9f2ac197942-tls" pod "b270c6f6-8593-4237-8877-b9f2ac197942" (UID: "b270c6f6-8593-4237-8877-b9f2ac197942"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:10.435510 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.435488 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf03981-68cb-48af-8f02-edb980939be8-kube-api-access-l26q9" pod "2cf03981-68cb-48af-8f02-edb980939be8" (UID: "2cf03981-68cb-48af-8f02-edb980939be8"). InnerVolumeSpecName "kube-api-access-l26q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:10.436716 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.436694 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-prometheus-client-cert" pod "2cf03981-68cb-48af-8f02-edb980939be8" (UID: "2cf03981-68cb-48af-8f02-edb980939be8"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:10.489637 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.489596 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:20:10.489875 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.489852 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="06218436-4431-4f18-a5c6-9871ac950890" containerName="alertmanager" containerID="cri-o://9c0fc4a7f73ede2bf840ee5d37114159824ebe0e282ea18adefc6997d8792cee" gracePeriod=120 Oct 30 04:20:10.489960 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.489935 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="06218436-4431-4f18-a5c6-9871ac950890" containerName="config-reloader" containerID="cri-o://e673a3dc702de87a8ed417eb51443807d2b3f0f72e6bf902679b0cdcc2124e64" gracePeriod=120 Oct 30 04:20:10.528904 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.527724 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l26q9\" (UniqueName: \"kubernetes.io/projected/2cf03981-68cb-48af-8f02-edb980939be8-kube-api-access-l26q9\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.528904 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.527777 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b270c6f6-8593-4237-8877-b9f2ac197942-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.528904 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.527795 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b270c6f6-8593-4237-8877-b9f2ac197942-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.528904 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.527808 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp897\" (UniqueName: \"kubernetes.io/projected/b270c6f6-8593-4237-8877-b9f2ac197942-kube-api-access-wp897\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.528904 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.527823 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqrxg\" (UniqueName: \"kubernetes.io/projected/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-kube-api-access-zqrxg\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.528904 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.527837 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.528904 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.527850 4925 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-kube-rbac-proxy-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.528904 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.527863 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.528904 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.527876 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/2cf03981-68cb-48af-8f02-edb980939be8-prometheus-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.528904 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.527889 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d-prometheus-tls-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:10.707229 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.707196 4925 generic.go:356] "Generic (PLEG): container finished" podID="18c4211b-62ec-40ee-9c20-600a3617ef96" containerID="fbc41e6c25fcddb29c31a6d085bcd9c92b44a1be7ef36188c0abecafdbd3d0cb" exitCode=0 Oct 30 04:20:10.707229 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.707220 4925 generic.go:356] "Generic (PLEG): container finished" podID="18c4211b-62ec-40ee-9c20-600a3617ef96" containerID="7d06c5963ab4664a7bb642e3a7801be08a5ad758acfd77c892ffb7a92bed7923" exitCode=0 Oct 30 04:20:10.707486 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.707251 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" Oct 30 04:20:10.707486 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.707279 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" event={"ID":"18c4211b-62ec-40ee-9c20-600a3617ef96","Type":"ContainerDied","Data":"fbc41e6c25fcddb29c31a6d085bcd9c92b44a1be7ef36188c0abecafdbd3d0cb"} Oct 30 04:20:10.707486 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.707354 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" event={"ID":"18c4211b-62ec-40ee-9c20-600a3617ef96","Type":"ContainerDied","Data":"7d06c5963ab4664a7bb642e3a7801be08a5ad758acfd77c892ffb7a92bed7923"} Oct 30 04:20:10.707486 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.707386 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t" event={"ID":"18c4211b-62ec-40ee-9c20-600a3617ef96","Type":"ContainerDied","Data":"687cf0ddf65c5fc3b5409d1ecde9ff0187b74e8f2de0e46eb9fdc4283d6e718e"} Oct 30 04:20:10.707486 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.707419 4925 scope.go:122] "RemoveContainer" containerID="fbc41e6c25fcddb29c31a6d085bcd9c92b44a1be7ef36188c0abecafdbd3d0cb" Oct 30 04:20:10.709643 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.709622 4925 generic.go:356] "Generic (PLEG): container finished" podID="06218436-4431-4f18-a5c6-9871ac950890" containerID="e673a3dc702de87a8ed417eb51443807d2b3f0f72e6bf902679b0cdcc2124e64" exitCode=0 Oct 30 04:20:10.709777 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.709692 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"06218436-4431-4f18-a5c6-9871ac950890","Type":"ContainerDied","Data":"e673a3dc702de87a8ed417eb51443807d2b3f0f72e6bf902679b0cdcc2124e64"} Oct 30 04:20:10.711022 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.711002 4925 generic.go:356] "Generic (PLEG): container finished" podID="b270c6f6-8593-4237-8877-b9f2ac197942" containerID="20adf80be713a5a5b4b6d313773de09a3108574148e9c5b368e204bf1564565a" exitCode=0 Oct 30 04:20:10.711130 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.711053 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" Oct 30 04:20:10.711130 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.711081 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" event={"ID":"b270c6f6-8593-4237-8877-b9f2ac197942","Type":"ContainerDied","Data":"20adf80be713a5a5b4b6d313773de09a3108574148e9c5b368e204bf1564565a"} Oct 30 04:20:10.711130 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.711124 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts" event={"ID":"b270c6f6-8593-4237-8877-b9f2ac197942","Type":"ContainerDied","Data":"51ecc3d8698fb9d4e4de91cea5be39eafaf220d6470fff7f7ece8b2baf535b64"} Oct 30 04:20:10.712387 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.712367 4925 generic.go:356] "Generic (PLEG): container finished" podID="2cf03981-68cb-48af-8f02-edb980939be8" containerID="9f01a5dc208c9becb7055638b3ba6d1b4fccd7f63bf7af625494a12073bca437" exitCode=0 Oct 30 04:20:10.712519 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.712410 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" Oct 30 04:20:10.712519 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.712440 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" event={"ID":"2cf03981-68cb-48af-8f02-edb980939be8","Type":"ContainerDied","Data":"9f01a5dc208c9becb7055638b3ba6d1b4fccd7f63bf7af625494a12073bca437"} Oct 30 04:20:10.712519 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.712488 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87" event={"ID":"2cf03981-68cb-48af-8f02-edb980939be8","Type":"ContainerDied","Data":"1e8d570e02b59d9bd16d2aa60f988ca76df2447646468ae0142eb78181e6c60d"} Oct 30 04:20:10.714489 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.714465 4925 generic.go:356] "Generic (PLEG): container finished" podID="7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d" containerID="7270a63289e020b0f3e32c7a9cc183c4715f05cfda5a78b29e40a8d1b59f5ebc" exitCode=0 Oct 30 04:20:10.714489 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.714487 4925 generic.go:356] "Generic (PLEG): container finished" podID="7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d" containerID="5711fbc6867bc94782d06ec391ec52d22a6acf306dd44f35f35a535ebdbd3bd8" exitCode=0 Oct 30 04:20:10.714641 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.714514 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" Oct 30 04:20:10.714641 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.714551 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" event={"ID":"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d","Type":"ContainerDied","Data":"7270a63289e020b0f3e32c7a9cc183c4715f05cfda5a78b29e40a8d1b59f5ebc"} Oct 30 04:20:10.714641 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.714617 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" event={"ID":"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d","Type":"ContainerDied","Data":"5711fbc6867bc94782d06ec391ec52d22a6acf306dd44f35f35a535ebdbd3bd8"} Oct 30 04:20:10.714827 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.714652 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7" event={"ID":"7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d","Type":"ContainerDied","Data":"73fff250b2764c708c30313735f62316c816b3c31ad4751db3010422ef006b47"} Oct 30 04:20:10.717989 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.717545 4925 scope.go:122] "RemoveContainer" containerID="7d06c5963ab4664a7bb642e3a7801be08a5ad758acfd77c892ffb7a92bed7923" Oct 30 04:20:10.727870 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.727844 4925 scope.go:122] "RemoveContainer" containerID="fbc41e6c25fcddb29c31a6d085bcd9c92b44a1be7ef36188c0abecafdbd3d0cb" Oct 30 04:20:10.728135 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:20:10.728112 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc41e6c25fcddb29c31a6d085bcd9c92b44a1be7ef36188c0abecafdbd3d0cb\": container with ID starting with fbc41e6c25fcddb29c31a6d085bcd9c92b44a1be7ef36188c0abecafdbd3d0cb not found: ID does not exist" containerID="fbc41e6c25fcddb29c31a6d085bcd9c92b44a1be7ef36188c0abecafdbd3d0cb" Oct 30 04:20:10.728255 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.728142 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc41e6c25fcddb29c31a6d085bcd9c92b44a1be7ef36188c0abecafdbd3d0cb"} err="failed to get container status \"fbc41e6c25fcddb29c31a6d085bcd9c92b44a1be7ef36188c0abecafdbd3d0cb\": rpc error: code = NotFound desc = could not find container \"fbc41e6c25fcddb29c31a6d085bcd9c92b44a1be7ef36188c0abecafdbd3d0cb\": container with ID starting with fbc41e6c25fcddb29c31a6d085bcd9c92b44a1be7ef36188c0abecafdbd3d0cb not found: ID does not exist" Oct 30 04:20:10.728255 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.728158 4925 scope.go:122] "RemoveContainer" containerID="7d06c5963ab4664a7bb642e3a7801be08a5ad758acfd77c892ffb7a92bed7923" Oct 30 04:20:10.728419 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:20:10.728401 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d06c5963ab4664a7bb642e3a7801be08a5ad758acfd77c892ffb7a92bed7923\": container with ID starting with 7d06c5963ab4664a7bb642e3a7801be08a5ad758acfd77c892ffb7a92bed7923 not found: ID does not exist" containerID="7d06c5963ab4664a7bb642e3a7801be08a5ad758acfd77c892ffb7a92bed7923" Oct 30 04:20:10.728460 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.728427 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d06c5963ab4664a7bb642e3a7801be08a5ad758acfd77c892ffb7a92bed7923"} err="failed to get container status \"7d06c5963ab4664a7bb642e3a7801be08a5ad758acfd77c892ffb7a92bed7923\": rpc error: code = NotFound desc = could not find container \"7d06c5963ab4664a7bb642e3a7801be08a5ad758acfd77c892ffb7a92bed7923\": container with ID starting with 7d06c5963ab4664a7bb642e3a7801be08a5ad758acfd77c892ffb7a92bed7923 not found: ID does not exist" Oct 30 04:20:10.728460 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.728442 4925 scope.go:122] "RemoveContainer" containerID="fbc41e6c25fcddb29c31a6d085bcd9c92b44a1be7ef36188c0abecafdbd3d0cb" Oct 30 04:20:10.728680 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.728663 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc41e6c25fcddb29c31a6d085bcd9c92b44a1be7ef36188c0abecafdbd3d0cb"} err="failed to get container status \"fbc41e6c25fcddb29c31a6d085bcd9c92b44a1be7ef36188c0abecafdbd3d0cb\": rpc error: code = NotFound desc = could not find container \"fbc41e6c25fcddb29c31a6d085bcd9c92b44a1be7ef36188c0abecafdbd3d0cb\": container with ID starting with fbc41e6c25fcddb29c31a6d085bcd9c92b44a1be7ef36188c0abecafdbd3d0cb not found: ID does not exist" Oct 30 04:20:10.728717 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.728680 4925 scope.go:122] "RemoveContainer" containerID="7d06c5963ab4664a7bb642e3a7801be08a5ad758acfd77c892ffb7a92bed7923" Oct 30 04:20:10.729061 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.729043 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d06c5963ab4664a7bb642e3a7801be08a5ad758acfd77c892ffb7a92bed7923"} err="failed to get container status \"7d06c5963ab4664a7bb642e3a7801be08a5ad758acfd77c892ffb7a92bed7923\": rpc error: code = NotFound desc = could not find container \"7d06c5963ab4664a7bb642e3a7801be08a5ad758acfd77c892ffb7a92bed7923\": container with ID starting with 7d06c5963ab4664a7bb642e3a7801be08a5ad758acfd77c892ffb7a92bed7923 not found: ID does not exist" Oct 30 04:20:10.729061 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.729062 4925 scope.go:122] "RemoveContainer" containerID="20adf80be713a5a5b4b6d313773de09a3108574148e9c5b368e204bf1564565a" Oct 30 04:20:10.736956 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.736939 4925 scope.go:122] "RemoveContainer" containerID="20adf80be713a5a5b4b6d313773de09a3108574148e9c5b368e204bf1564565a" Oct 30 04:20:10.737236 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:20:10.737217 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20adf80be713a5a5b4b6d313773de09a3108574148e9c5b368e204bf1564565a\": container with ID starting with 20adf80be713a5a5b4b6d313773de09a3108574148e9c5b368e204bf1564565a not found: ID does not exist" containerID="20adf80be713a5a5b4b6d313773de09a3108574148e9c5b368e204bf1564565a" Oct 30 04:20:10.737285 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.737241 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20adf80be713a5a5b4b6d313773de09a3108574148e9c5b368e204bf1564565a"} err="failed to get container status \"20adf80be713a5a5b4b6d313773de09a3108574148e9c5b368e204bf1564565a\": rpc error: code = NotFound desc = could not find container \"20adf80be713a5a5b4b6d313773de09a3108574148e9c5b368e204bf1564565a\": container with ID starting with 20adf80be713a5a5b4b6d313773de09a3108574148e9c5b368e204bf1564565a not found: ID does not exist" Oct 30 04:20:10.737285 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.737255 4925 scope.go:122] "RemoveContainer" containerID="9f01a5dc208c9becb7055638b3ba6d1b4fccd7f63bf7af625494a12073bca437" Oct 30 04:20:10.745053 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.745035 4925 scope.go:122] "RemoveContainer" containerID="9f01a5dc208c9becb7055638b3ba6d1b4fccd7f63bf7af625494a12073bca437" Oct 30 04:20:10.745308 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:20:10.745289 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f01a5dc208c9becb7055638b3ba6d1b4fccd7f63bf7af625494a12073bca437\": container with ID starting with 9f01a5dc208c9becb7055638b3ba6d1b4fccd7f63bf7af625494a12073bca437 not found: ID does not exist" containerID="9f01a5dc208c9becb7055638b3ba6d1b4fccd7f63bf7af625494a12073bca437" Oct 30 04:20:10.745356 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.745314 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f01a5dc208c9becb7055638b3ba6d1b4fccd7f63bf7af625494a12073bca437"} err="failed to get container status \"9f01a5dc208c9becb7055638b3ba6d1b4fccd7f63bf7af625494a12073bca437\": rpc error: code = NotFound desc = could not find container \"9f01a5dc208c9becb7055638b3ba6d1b4fccd7f63bf7af625494a12073bca437\": container with ID starting with 9f01a5dc208c9becb7055638b3ba6d1b4fccd7f63bf7af625494a12073bca437 not found: ID does not exist" Oct 30 04:20:10.745356 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.745330 4925 scope.go:122] "RemoveContainer" containerID="7270a63289e020b0f3e32c7a9cc183c4715f05cfda5a78b29e40a8d1b59f5ebc" Oct 30 04:20:10.752944 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.752925 4925 scope.go:122] "RemoveContainer" containerID="5711fbc6867bc94782d06ec391ec52d22a6acf306dd44f35f35a535ebdbd3bd8" Oct 30 04:20:10.761374 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.761358 4925 scope.go:122] "RemoveContainer" containerID="7270a63289e020b0f3e32c7a9cc183c4715f05cfda5a78b29e40a8d1b59f5ebc" Oct 30 04:20:10.761616 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:20:10.761597 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7270a63289e020b0f3e32c7a9cc183c4715f05cfda5a78b29e40a8d1b59f5ebc\": container with ID starting with 7270a63289e020b0f3e32c7a9cc183c4715f05cfda5a78b29e40a8d1b59f5ebc not found: ID does not exist" containerID="7270a63289e020b0f3e32c7a9cc183c4715f05cfda5a78b29e40a8d1b59f5ebc" Oct 30 04:20:10.761697 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.761626 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7270a63289e020b0f3e32c7a9cc183c4715f05cfda5a78b29e40a8d1b59f5ebc"} err="failed to get container status \"7270a63289e020b0f3e32c7a9cc183c4715f05cfda5a78b29e40a8d1b59f5ebc\": rpc error: code = NotFound desc = could not find container \"7270a63289e020b0f3e32c7a9cc183c4715f05cfda5a78b29e40a8d1b59f5ebc\": container with ID starting with 7270a63289e020b0f3e32c7a9cc183c4715f05cfda5a78b29e40a8d1b59f5ebc not found: ID does not exist" Oct 30 04:20:10.761697 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.761648 4925 scope.go:122] "RemoveContainer" containerID="5711fbc6867bc94782d06ec391ec52d22a6acf306dd44f35f35a535ebdbd3bd8" Oct 30 04:20:10.762046 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:20:10.762027 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5711fbc6867bc94782d06ec391ec52d22a6acf306dd44f35f35a535ebdbd3bd8\": container with ID starting with 5711fbc6867bc94782d06ec391ec52d22a6acf306dd44f35f35a535ebdbd3bd8 not found: ID does not exist" containerID="5711fbc6867bc94782d06ec391ec52d22a6acf306dd44f35f35a535ebdbd3bd8" Oct 30 04:20:10.762122 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.762052 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5711fbc6867bc94782d06ec391ec52d22a6acf306dd44f35f35a535ebdbd3bd8"} err="failed to get container status \"5711fbc6867bc94782d06ec391ec52d22a6acf306dd44f35f35a535ebdbd3bd8\": rpc error: code = NotFound desc = could not find container \"5711fbc6867bc94782d06ec391ec52d22a6acf306dd44f35f35a535ebdbd3bd8\": container with ID starting with 5711fbc6867bc94782d06ec391ec52d22a6acf306dd44f35f35a535ebdbd3bd8 not found: ID does not exist" Oct 30 04:20:10.762122 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.762072 4925 scope.go:122] "RemoveContainer" containerID="7270a63289e020b0f3e32c7a9cc183c4715f05cfda5a78b29e40a8d1b59f5ebc" Oct 30 04:20:10.762266 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.762247 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7270a63289e020b0f3e32c7a9cc183c4715f05cfda5a78b29e40a8d1b59f5ebc"} err="failed to get container status \"7270a63289e020b0f3e32c7a9cc183c4715f05cfda5a78b29e40a8d1b59f5ebc\": rpc error: code = NotFound desc = could not find container \"7270a63289e020b0f3e32c7a9cc183c4715f05cfda5a78b29e40a8d1b59f5ebc\": container with ID starting with 7270a63289e020b0f3e32c7a9cc183c4715f05cfda5a78b29e40a8d1b59f5ebc not found: ID does not exist" Oct 30 04:20:10.762330 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.762269 4925 scope.go:122] "RemoveContainer" containerID="5711fbc6867bc94782d06ec391ec52d22a6acf306dd44f35f35a535ebdbd3bd8" Oct 30 04:20:10.762509 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.762491 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5711fbc6867bc94782d06ec391ec52d22a6acf306dd44f35f35a535ebdbd3bd8"} err="failed to get container status \"5711fbc6867bc94782d06ec391ec52d22a6acf306dd44f35f35a535ebdbd3bd8\": rpc error: code = NotFound desc = could not find container \"5711fbc6867bc94782d06ec391ec52d22a6acf306dd44f35f35a535ebdbd3bd8\": container with ID starting with 5711fbc6867bc94782d06ec391ec52d22a6acf306dd44f35f35a535ebdbd3bd8 not found: ID does not exist" Oct 30 04:20:10.789736 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.789668 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t"] Oct 30 04:20:10.795535 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.795510 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-gnc7t"] Oct 30 04:20:10.847014 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.846983 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts"] Oct 30 04:20:10.851137 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.851110 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-g56ts"] Oct 30 04:20:10.886045 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.886017 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87"] Oct 30 04:20:10.890218 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.890190 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-wcz87"] Oct 30 04:20:10.926262 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.926221 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7"] Oct 30 04:20:10.930330 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:10.930306 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-bj7h7"] Oct 30 04:20:11.122416 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.122380 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18c4211b-62ec-40ee-9c20-600a3617ef96" path="/var/lib/kubelet/pods/18c4211b-62ec-40ee-9c20-600a3617ef96/volumes" Oct 30 04:20:11.123034 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.123014 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf03981-68cb-48af-8f02-edb980939be8" path="/var/lib/kubelet/pods/2cf03981-68cb-48af-8f02-edb980939be8/volumes" Oct 30 04:20:11.123518 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.123502 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d" path="/var/lib/kubelet/pods/7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d/volumes" Oct 30 04:20:11.124508 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.124490 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b270c6f6-8593-4237-8877-b9f2ac197942" path="/var/lib/kubelet/pods/b270c6f6-8593-4237-8877-b9f2ac197942/volumes" Oct 30 04:20:11.129006 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.128981 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:20:11.129227 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.129205 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="d50428f2-cc0e-49d8-a542-31715459a591" containerName="otc-container" containerID="cri-o://ebad9d1ee78244ed7ef623c17e6190eea43dbe35ad85a3c1031a8c03c6b05653" gracePeriod=30 Oct 30 04:20:11.384472 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.384419 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:11.436660 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.434834 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d50428f2-cc0e-49d8-a542-31715459a591-kube-api-access-ktd5g\" (UniqueName: \"kubernetes.io/projected/d50428f2-cc0e-49d8-a542-31715459a591-kube-api-access-ktd5g\") pod \"d50428f2-cc0e-49d8-a542-31715459a591\" (UID: \"d50428f2-cc0e-49d8-a542-31715459a591\") " Oct 30 04:20:11.436660 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.434896 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-data-science-collector-ta-client-cert\") pod \"d50428f2-cc0e-49d8-a542-31715459a591\" (UID: \"d50428f2-cc0e-49d8-a542-31715459a591\") " Oct 30 04:20:11.436660 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.435151 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d50428f2-cc0e-49d8-a542-31715459a591-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d50428f2-cc0e-49d8-a542-31715459a591-otc-internal\") pod \"d50428f2-cc0e-49d8-a542-31715459a591\" (UID: \"d50428f2-cc0e-49d8-a542-31715459a591\") " Oct 30 04:20:11.436660 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.435193 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-monitoring-tls-certs\") pod \"d50428f2-cc0e-49d8-a542-31715459a591\" (UID: \"d50428f2-cc0e-49d8-a542-31715459a591\") " Oct 30 04:20:11.436660 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.435215 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-tls-certs\" (UniqueName: \"kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-tls-certs\") pod \"d50428f2-cc0e-49d8-a542-31715459a591\" (UID: \"d50428f2-cc0e-49d8-a542-31715459a591\") " Oct 30 04:20:11.436660 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.435547 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50428f2-cc0e-49d8-a542-31715459a591-otc-internal" pod "d50428f2-cc0e-49d8-a542-31715459a591" (UID: "d50428f2-cc0e-49d8-a542-31715459a591"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:11.437896 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.437796 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50428f2-cc0e-49d8-a542-31715459a591-kube-api-access-ktd5g" pod "d50428f2-cc0e-49d8-a542-31715459a591" (UID: "d50428f2-cc0e-49d8-a542-31715459a591"). InnerVolumeSpecName "kube-api-access-ktd5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:11.438386 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.438343 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-data-science-collector-ta-client-cert" pod "d50428f2-cc0e-49d8-a542-31715459a591" (UID: "d50428f2-cc0e-49d8-a542-31715459a591"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:11.439136 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.439109 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-tls-certs" pod "d50428f2-cc0e-49d8-a542-31715459a591" (UID: "d50428f2-cc0e-49d8-a542-31715459a591"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:11.440358 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.440322 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-monitoring-tls-certs" pod "d50428f2-cc0e-49d8-a542-31715459a591" (UID: "d50428f2-cc0e-49d8-a542-31715459a591"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:11.536878 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.536837 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktd5g\" (UniqueName: \"kubernetes.io/projected/d50428f2-cc0e-49d8-a542-31715459a591-kube-api-access-ktd5g\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:11.536878 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.536870 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:11.536878 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.536882 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d50428f2-cc0e-49d8-a542-31715459a591-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:11.537133 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.536891 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:11.537133 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.536902 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50428f2-cc0e-49d8-a542-31715459a591-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:11.721851 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.721767 4925 generic.go:356] "Generic (PLEG): container finished" podID="d50428f2-cc0e-49d8-a542-31715459a591" containerID="ebad9d1ee78244ed7ef623c17e6190eea43dbe35ad85a3c1031a8c03c6b05653" exitCode=0 Oct 30 04:20:11.721851 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.721817 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:11.722109 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.721852 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d50428f2-cc0e-49d8-a542-31715459a591","Type":"ContainerDied","Data":"ebad9d1ee78244ed7ef623c17e6190eea43dbe35ad85a3c1031a8c03c6b05653"} Oct 30 04:20:11.722109 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.721913 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d50428f2-cc0e-49d8-a542-31715459a591","Type":"ContainerDied","Data":"e1c1e312dc3e75a272cc303cac924bcc80605319c3218c13426657e10544e1be"} Oct 30 04:20:11.722109 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.721944 4925 scope.go:122] "RemoveContainer" containerID="ebad9d1ee78244ed7ef623c17e6190eea43dbe35ad85a3c1031a8c03c6b05653" Oct 30 04:20:11.731689 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.731672 4925 scope.go:122] "RemoveContainer" containerID="ebad9d1ee78244ed7ef623c17e6190eea43dbe35ad85a3c1031a8c03c6b05653" Oct 30 04:20:11.731954 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:20:11.731936 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebad9d1ee78244ed7ef623c17e6190eea43dbe35ad85a3c1031a8c03c6b05653\": container with ID starting with ebad9d1ee78244ed7ef623c17e6190eea43dbe35ad85a3c1031a8c03c6b05653 not found: ID does not exist" containerID="ebad9d1ee78244ed7ef623c17e6190eea43dbe35ad85a3c1031a8c03c6b05653" Oct 30 04:20:11.732005 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.731962 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebad9d1ee78244ed7ef623c17e6190eea43dbe35ad85a3c1031a8c03c6b05653"} err="failed to get container status \"ebad9d1ee78244ed7ef623c17e6190eea43dbe35ad85a3c1031a8c03c6b05653\": rpc error: code = NotFound desc = could not find container \"ebad9d1ee78244ed7ef623c17e6190eea43dbe35ad85a3c1031a8c03c6b05653\": container with ID starting with ebad9d1ee78244ed7ef623c17e6190eea43dbe35ad85a3c1031a8c03c6b05653 not found: ID does not exist" Oct 30 04:20:11.774419 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.774385 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:20:11.779184 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:11.779160 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:20:13.119406 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:13.119373 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d50428f2-cc0e-49d8-a542-31715459a591" path="/var/lib/kubelet/pods/d50428f2-cc0e-49d8-a542-31715459a591/volumes" Oct 30 04:20:14.542523 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:14.542479 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="06218436-4431-4f18-a5c6-9871ac950890" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.153:9093/-/ready\": dial tcp 10.132.0.153:9093: connect: connection refused" Oct 30 04:20:17.135153 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135119 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk"] Oct 30 04:20:17.135545 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135433 4925 cpu_manager.go:414] "containerMap: removing container" podUID="7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d" containerName="kube-rbac-proxy" Oct 30 04:20:17.135545 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135444 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d" containerName="kube-rbac-proxy" Oct 30 04:20:17.135545 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135461 4925 cpu_manager.go:414] "containerMap: removing container" podUID="2cf03981-68cb-48af-8f02-edb980939be8" containerName="kube-rbac-proxy" Oct 30 04:20:17.135545 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135466 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf03981-68cb-48af-8f02-edb980939be8" containerName="kube-rbac-proxy" Oct 30 04:20:17.135545 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135481 4925 cpu_manager.go:414] "containerMap: removing container" podUID="7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d" containerName="prom-label-proxy" Oct 30 04:20:17.135545 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135486 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d" containerName="prom-label-proxy" Oct 30 04:20:17.135545 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135494 4925 cpu_manager.go:414] "containerMap: removing container" podUID="18c4211b-62ec-40ee-9c20-600a3617ef96" containerName="prom-label-proxy" Oct 30 04:20:17.135545 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135499 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18c4211b-62ec-40ee-9c20-600a3617ef96" containerName="prom-label-proxy" Oct 30 04:20:17.135545 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135505 4925 cpu_manager.go:414] "containerMap: removing container" podUID="d50428f2-cc0e-49d8-a542-31715459a591" containerName="otc-container" Oct 30 04:20:17.135545 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135510 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d50428f2-cc0e-49d8-a542-31715459a591" containerName="otc-container" Oct 30 04:20:17.135545 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135520 4925 cpu_manager.go:414] "containerMap: removing container" podUID="18c4211b-62ec-40ee-9c20-600a3617ef96" containerName="kube-rbac-proxy" Oct 30 04:20:17.135545 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135525 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18c4211b-62ec-40ee-9c20-600a3617ef96" containerName="kube-rbac-proxy" Oct 30 04:20:17.135545 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135534 4925 cpu_manager.go:414] "containerMap: removing container" podUID="b270c6f6-8593-4237-8877-b9f2ac197942" containerName="korrel8r" Oct 30 04:20:17.135545 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135538 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b270c6f6-8593-4237-8877-b9f2ac197942" containerName="korrel8r" Oct 30 04:20:17.136002 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135591 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf03981-68cb-48af-8f02-edb980939be8" containerName="kube-rbac-proxy" Oct 30 04:20:17.136002 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135600 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d" containerName="prom-label-proxy" Oct 30 04:20:17.136002 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135606 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="18c4211b-62ec-40ee-9c20-600a3617ef96" containerName="prom-label-proxy" Oct 30 04:20:17.136002 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135612 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d50428f2-cc0e-49d8-a542-31715459a591" containerName="otc-container" Oct 30 04:20:17.136002 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135619 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="18c4211b-62ec-40ee-9c20-600a3617ef96" containerName="kube-rbac-proxy" Oct 30 04:20:17.136002 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135624 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b270c6f6-8593-4237-8877-b9f2ac197942" containerName="korrel8r" Oct 30 04:20:17.136002 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.135631 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dfa3bc8-85a6-4387-98fe-bf3d3fdeb91d" containerName="kube-rbac-proxy" Oct 30 04:20:17.140144 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.140122 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:17.157620 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.157597 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 04:20:17.169922 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.169888 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 04:20:17.170078 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.169946 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-jlsb6\"" Oct 30 04:20:17.170078 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.169918 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 04:20:17.170078 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.169924 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 04:20:17.171583 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.171563 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk"] Oct 30 04:20:17.185675 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.185650 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-9t6vk\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:17.185839 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.185688 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztv64\" (UniqueName: \"kubernetes.io/projected/b8f035ab-52f5-46c0-9bbe-9e94699c3416-kube-api-access-ztv64\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-9t6vk\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:17.185839 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.185718 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/b8f035ab-52f5-46c0-9bbe-9e94699c3416-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-9t6vk\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:17.185939 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.185854 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-9t6vk\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:17.185939 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.185892 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-9t6vk\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:17.186094 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.186073 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s"] Oct 30 04:20:17.189623 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.189607 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" Oct 30 04:20:17.213646 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.213614 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s"] Oct 30 04:20:17.257670 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.257633 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 04:20:17.257670 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.257637 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 04:20:17.257883 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.257701 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-2cwff\"" Oct 30 04:20:17.257883 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.257715 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 04:20:17.279719 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.279683 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g"] Oct 30 04:20:17.283045 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.283028 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" Oct 30 04:20:17.287057 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.287033 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-mtb9s\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" Oct 30 04:20:17.287168 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.287071 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-9t6vk\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:17.287168 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.287138 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztv64\" (UniqueName: \"kubernetes.io/projected/b8f035ab-52f5-46c0-9bbe-9e94699c3416-kube-api-access-ztv64\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-9t6vk\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:17.287294 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.287182 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/91407852-200e-4ce6-89a1-ef11eed5a272-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-mtb9s\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" Oct 30 04:20:17.287294 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.287214 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/b8f035ab-52f5-46c0-9bbe-9e94699c3416-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-9t6vk\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:17.287407 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.287306 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-9t6vk\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:17.287407 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.287335 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wlxc\" (UniqueName: \"kubernetes.io/projected/91407852-200e-4ce6-89a1-ef11eed5a272-kube-api-access-7wlxc\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-mtb9s\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" Oct 30 04:20:17.287407 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.287365 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-mtb9s\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" Oct 30 04:20:17.287407 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.287390 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-9t6vk\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:17.287606 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.287415 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-mtb9s\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" Oct 30 04:20:17.287606 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:20:17.287430 4925 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 04:20:17.287606 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:20:17.287481 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-prometheus-client-cert podName:b8f035ab-52f5-46c0-9bbe-9e94699c3416 nodeName:}" failed. No retries permitted until 2026-10-30 04:20:17.78746439 +0000 UTC m=+5978.800742472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" (UID: "b8f035ab-52f5-46c0-9bbe-9e94699c3416") : secret "prometheus-operated-tls" not found Oct 30 04:20:17.287906 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.287877 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/b8f035ab-52f5-46c0-9bbe-9e94699c3416-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-9t6vk\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:17.289853 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.289830 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-9t6vk\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:17.289853 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.289853 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-9t6vk\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:17.306808 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.306780 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g"] Oct 30 04:20:17.315961 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.315937 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztv64\" (UniqueName: \"kubernetes.io/projected/b8f035ab-52f5-46c0-9bbe-9e94699c3416-kube-api-access-ztv64\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-9t6vk\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:17.357938 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.357906 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-9qb9b\"" Oct 30 04:20:17.357938 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.357907 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 04:20:17.358115 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.357908 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 04:20:17.388446 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.388370 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wlxc\" (UniqueName: \"kubernetes.io/projected/91407852-200e-4ce6-89a1-ef11eed5a272-kube-api-access-7wlxc\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-mtb9s\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" Oct 30 04:20:17.388446 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.388420 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-mtb9s\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" Oct 30 04:20:17.388652 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.388511 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-mtb9s\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" Oct 30 04:20:17.388652 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.388539 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blqh7\" (UniqueName: \"kubernetes.io/projected/7d087d6e-df54-4262-90e3-86142f1a39cc-kube-api-access-blqh7\") pod \"korrel8r-5cbcc89b8f-lns9g\" (UID: \"7d087d6e-df54-4262-90e3-86142f1a39cc\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" Oct 30 04:20:17.388652 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.388567 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-mtb9s\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" Oct 30 04:20:17.388652 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.388594 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7d087d6e-df54-4262-90e3-86142f1a39cc-tls\") pod \"korrel8r-5cbcc89b8f-lns9g\" (UID: \"7d087d6e-df54-4262-90e3-86142f1a39cc\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" Oct 30 04:20:17.388900 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.388801 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/91407852-200e-4ce6-89a1-ef11eed5a272-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-mtb9s\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" Oct 30 04:20:17.388900 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.388843 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7d087d6e-df54-4262-90e3-86142f1a39cc-config\") pod \"korrel8r-5cbcc89b8f-lns9g\" (UID: \"7d087d6e-df54-4262-90e3-86142f1a39cc\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" Oct 30 04:20:17.389005 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:20:17.388942 4925 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-prometheus-namespace-proxy-tls: secret "data-science-prometheus-namespace-proxy-tls" not found Oct 30 04:20:17.389071 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:20:17.389013 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91407852-200e-4ce6-89a1-ef11eed5a272-tls podName:91407852-200e-4ce6-89a1-ef11eed5a272 nodeName:}" failed. No retries permitted until 2026-10-30 04:20:17.888993808 +0000 UTC m=+5978.902271906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/91407852-200e-4ce6-89a1-ef11eed5a272-tls") pod "data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" (UID: "91407852-200e-4ce6-89a1-ef11eed5a272") : secret "data-science-prometheus-namespace-proxy-tls" not found Oct 30 04:20:17.389231 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.389211 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-mtb9s\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" Oct 30 04:20:17.389294 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.389284 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-mtb9s\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" Oct 30 04:20:17.389384 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.389364 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-mtb9s\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" Oct 30 04:20:17.414639 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.414613 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wlxc\" (UniqueName: \"kubernetes.io/projected/91407852-200e-4ce6-89a1-ef11eed5a272-kube-api-access-7wlxc\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-mtb9s\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" Oct 30 04:20:17.490070 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.490038 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blqh7\" (UniqueName: \"kubernetes.io/projected/7d087d6e-df54-4262-90e3-86142f1a39cc-kube-api-access-blqh7\") pod \"korrel8r-5cbcc89b8f-lns9g\" (UID: \"7d087d6e-df54-4262-90e3-86142f1a39cc\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" Oct 30 04:20:17.490242 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.490095 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7d087d6e-df54-4262-90e3-86142f1a39cc-tls\") pod \"korrel8r-5cbcc89b8f-lns9g\" (UID: \"7d087d6e-df54-4262-90e3-86142f1a39cc\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" Oct 30 04:20:17.490242 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.490154 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7d087d6e-df54-4262-90e3-86142f1a39cc-config\") pod \"korrel8r-5cbcc89b8f-lns9g\" (UID: \"7d087d6e-df54-4262-90e3-86142f1a39cc\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" Oct 30 04:20:17.490985 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.490967 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7d087d6e-df54-4262-90e3-86142f1a39cc-config\") pod \"korrel8r-5cbcc89b8f-lns9g\" (UID: \"7d087d6e-df54-4262-90e3-86142f1a39cc\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" Oct 30 04:20:17.492440 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.492420 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7d087d6e-df54-4262-90e3-86142f1a39cc-tls\") pod \"korrel8r-5cbcc89b8f-lns9g\" (UID: \"7d087d6e-df54-4262-90e3-86142f1a39cc\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" Oct 30 04:20:17.513920 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.513894 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blqh7\" (UniqueName: \"kubernetes.io/projected/7d087d6e-df54-4262-90e3-86142f1a39cc-kube-api-access-blqh7\") pod \"korrel8r-5cbcc89b8f-lns9g\" (UID: \"7d087d6e-df54-4262-90e3-86142f1a39cc\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" Oct 30 04:20:17.603994 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.603958 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" Oct 30 04:20:17.793074 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.793029 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-9t6vk\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:17.795357 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.795336 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-9t6vk\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:17.894158 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.894110 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/91407852-200e-4ce6-89a1-ef11eed5a272-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-mtb9s\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" Oct 30 04:20:17.896518 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.896492 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/91407852-200e-4ce6-89a1-ef11eed5a272-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-mtb9s\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" Oct 30 04:20:17.901985 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:17.901965 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g"] Oct 30 04:20:17.905077 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:20:17.905043 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d087d6e_df54_4262_90e3_86142f1a39cc.slice/crio-333f62af95308377932494f82c63f6f23f636ada76b6e292fa280b5efdf97aff WatchSource:0}: Error finding container 333f62af95308377932494f82c63f6f23f636ada76b6e292fa280b5efdf97aff: Status 404 returned error can't find the container with id 333f62af95308377932494f82c63f6f23f636ada76b6e292fa280b5efdf97aff Oct 30 04:20:18.053214 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:18.053186 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:18.104789 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:18.104727 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" Oct 30 04:20:18.174117 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:20:18.167818 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:20:18.354029 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:18.353998 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk"] Oct 30 04:20:18.356997 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:20:18.356964 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f035ab_52f5_46c0_9bbe_9e94699c3416.slice/crio-78585e4fd7a156914009ff611ff62583e04f2067e6da9b1ae237cb974ba5d323 WatchSource:0}: Error finding container 78585e4fd7a156914009ff611ff62583e04f2067e6da9b1ae237cb974ba5d323: Status 404 returned error can't find the container with id 78585e4fd7a156914009ff611ff62583e04f2067e6da9b1ae237cb974ba5d323 Oct 30 04:20:18.414490 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:18.414432 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s"] Oct 30 04:20:18.417372 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:20:18.417346 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91407852_200e_4ce6_89a1_ef11eed5a272.slice/crio-ded7d52008abf76312694c655667047a0f6e9e5806ee928d5c5da1abc27dc380 WatchSource:0}: Error finding container ded7d52008abf76312694c655667047a0f6e9e5806ee928d5c5da1abc27dc380: Status 404 returned error can't find the container with id ded7d52008abf76312694c655667047a0f6e9e5806ee928d5c5da1abc27dc380 Oct 30 04:20:18.765085 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:18.764993 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" event={"ID":"91407852-200e-4ce6-89a1-ef11eed5a272","Type":"ContainerStarted","Data":"1233b6cb419d7a7dde39020a96cef5bd539e1f9534e76896ad064925c860c562"} Oct 30 04:20:18.765085 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:18.765062 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" event={"ID":"91407852-200e-4ce6-89a1-ef11eed5a272","Type":"ContainerStarted","Data":"d1be7bbe29b72950685ddefbfe22ceefa1a3aaa877ee81415ffab6fa1fc3ef1c"} Oct 30 04:20:18.765368 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:18.765100 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" event={"ID":"91407852-200e-4ce6-89a1-ef11eed5a272","Type":"ContainerStarted","Data":"ded7d52008abf76312694c655667047a0f6e9e5806ee928d5c5da1abc27dc380"} Oct 30 04:20:18.767166 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:18.767130 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" event={"ID":"b8f035ab-52f5-46c0-9bbe-9e94699c3416","Type":"ContainerStarted","Data":"78585e4fd7a156914009ff611ff62583e04f2067e6da9b1ae237cb974ba5d323"} Oct 30 04:20:18.768780 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:18.768733 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" event={"ID":"7d087d6e-df54-4262-90e3-86142f1a39cc","Type":"ContainerStarted","Data":"4b3883b0c719dd186a92f786766730eb2802eda4255c3aaee2d6d2c1de03fd8b"} Oct 30 04:20:18.768877 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:18.768805 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" event={"ID":"7d087d6e-df54-4262-90e3-86142f1a39cc","Type":"ContainerStarted","Data":"333f62af95308377932494f82c63f6f23f636ada76b6e292fa280b5efdf97aff"} Oct 30 04:20:18.768946 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:18.768932 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" Oct 30 04:20:18.810606 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:18.810560 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" podStartSLOduration=1.810548268 podStartE2EDuration="1.810548268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:18.809055521 +0000 UTC m=+5979.822333624" watchObservedRunningTime="2026-10-30 04:20:18.810548268 +0000 UTC m=+5979.823826371" Oct 30 04:20:18.851597 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:18.851507 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" podStartSLOduration=1.8514856389999998 podStartE2EDuration="1.851485639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:18.851175332 +0000 UTC m=+5979.864453435" watchObservedRunningTime="2026-10-30 04:20:18.851485639 +0000 UTC m=+5979.864763742" Oct 30 04:20:19.542242 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:19.542200 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="06218436-4431-4f18-a5c6-9871ac950890" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.153:9093/-/ready\": dial tcp 10.132.0.153:9093: connect: connection refused" Oct 30 04:20:19.775636 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:19.775592 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" event={"ID":"b8f035ab-52f5-46c0-9bbe-9e94699c3416","Type":"ContainerStarted","Data":"3de50932319ffa0936aa930a9619281590462d7365d42201aa42d9b4eaad919a"} Oct 30 04:20:19.776127 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:19.776088 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:19.820988 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:19.820890 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" podStartSLOduration=2.820878009 podStartE2EDuration="2.820878009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:19.8191248 +0000 UTC m=+5980.832402904" watchObservedRunningTime="2026-10-30 04:20:19.820878009 +0000 UTC m=+5980.834156106" Oct 30 04:20:20.736415 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.736388 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:20.782503 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.782424 4925 generic.go:356] "Generic (PLEG): container finished" podID="06218436-4431-4f18-a5c6-9871ac950890" containerID="9c0fc4a7f73ede2bf840ee5d37114159824ebe0e282ea18adefc6997d8792cee" exitCode=0 Oct 30 04:20:20.782503 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.782485 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:20.782680 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.782511 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"06218436-4431-4f18-a5c6-9871ac950890","Type":"ContainerDied","Data":"9c0fc4a7f73ede2bf840ee5d37114159824ebe0e282ea18adefc6997d8792cee"} Oct 30 04:20:20.782680 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.782573 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"06218436-4431-4f18-a5c6-9871ac950890","Type":"ContainerDied","Data":"43f48a37017384bd49e4b23b831c085acebc95af2e8a327db39a86889326b826"} Oct 30 04:20:20.782680 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.782602 4925 scope.go:122] "RemoveContainer" containerID="e673a3dc702de87a8ed417eb51443807d2b3f0f72e6bf902679b0cdcc2124e64" Oct 30 04:20:20.790937 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.790922 4925 scope.go:122] "RemoveContainer" containerID="9c0fc4a7f73ede2bf840ee5d37114159824ebe0e282ea18adefc6997d8792cee" Oct 30 04:20:20.799554 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.799538 4925 scope.go:122] "RemoveContainer" containerID="3ddcaec80adcdbe1ffdb1d6d9da3e14468b8e74651d6a917308ba7558b5bd5b8" Oct 30 04:20:20.807731 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.807707 4925 scope.go:122] "RemoveContainer" containerID="e673a3dc702de87a8ed417eb51443807d2b3f0f72e6bf902679b0cdcc2124e64" Oct 30 04:20:20.808048 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:20:20.808022 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e673a3dc702de87a8ed417eb51443807d2b3f0f72e6bf902679b0cdcc2124e64\": container with ID starting with e673a3dc702de87a8ed417eb51443807d2b3f0f72e6bf902679b0cdcc2124e64 not found: ID does not exist" containerID="e673a3dc702de87a8ed417eb51443807d2b3f0f72e6bf902679b0cdcc2124e64" Oct 30 04:20:20.808147 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.808056 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e673a3dc702de87a8ed417eb51443807d2b3f0f72e6bf902679b0cdcc2124e64"} err="failed to get container status \"e673a3dc702de87a8ed417eb51443807d2b3f0f72e6bf902679b0cdcc2124e64\": rpc error: code = NotFound desc = could not find container \"e673a3dc702de87a8ed417eb51443807d2b3f0f72e6bf902679b0cdcc2124e64\": container with ID starting with e673a3dc702de87a8ed417eb51443807d2b3f0f72e6bf902679b0cdcc2124e64 not found: ID does not exist" Oct 30 04:20:20.808147 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.808074 4925 scope.go:122] "RemoveContainer" containerID="9c0fc4a7f73ede2bf840ee5d37114159824ebe0e282ea18adefc6997d8792cee" Oct 30 04:20:20.808336 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:20:20.808320 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c0fc4a7f73ede2bf840ee5d37114159824ebe0e282ea18adefc6997d8792cee\": container with ID starting with 9c0fc4a7f73ede2bf840ee5d37114159824ebe0e282ea18adefc6997d8792cee not found: ID does not exist" containerID="9c0fc4a7f73ede2bf840ee5d37114159824ebe0e282ea18adefc6997d8792cee" Oct 30 04:20:20.808408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.808339 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c0fc4a7f73ede2bf840ee5d37114159824ebe0e282ea18adefc6997d8792cee"} err="failed to get container status \"9c0fc4a7f73ede2bf840ee5d37114159824ebe0e282ea18adefc6997d8792cee\": rpc error: code = NotFound desc = could not find container \"9c0fc4a7f73ede2bf840ee5d37114159824ebe0e282ea18adefc6997d8792cee\": container with ID starting with 9c0fc4a7f73ede2bf840ee5d37114159824ebe0e282ea18adefc6997d8792cee not found: ID does not exist" Oct 30 04:20:20.808408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.808351 4925 scope.go:122] "RemoveContainer" containerID="3ddcaec80adcdbe1ffdb1d6d9da3e14468b8e74651d6a917308ba7558b5bd5b8" Oct 30 04:20:20.808627 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:20:20.808605 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ddcaec80adcdbe1ffdb1d6d9da3e14468b8e74651d6a917308ba7558b5bd5b8\": container with ID starting with 3ddcaec80adcdbe1ffdb1d6d9da3e14468b8e74651d6a917308ba7558b5bd5b8 not found: ID does not exist" containerID="3ddcaec80adcdbe1ffdb1d6d9da3e14468b8e74651d6a917308ba7558b5bd5b8" Oct 30 04:20:20.808698 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.808630 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ddcaec80adcdbe1ffdb1d6d9da3e14468b8e74651d6a917308ba7558b5bd5b8"} err="failed to get container status \"3ddcaec80adcdbe1ffdb1d6d9da3e14468b8e74651d6a917308ba7558b5bd5b8\": rpc error: code = NotFound desc = could not find container \"3ddcaec80adcdbe1ffdb1d6d9da3e14468b8e74651d6a917308ba7558b5bd5b8\": container with ID starting with 3ddcaec80adcdbe1ffdb1d6d9da3e14468b8e74651d6a917308ba7558b5bd5b8 not found: ID does not exist" Oct 30 04:20:20.819828 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.819803 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06218436-4431-4f18-a5c6-9871ac950890-config-out\" (UniqueName: \"kubernetes.io/empty-dir/06218436-4431-4f18-a5c6-9871ac950890-config-out\") pod \"06218436-4431-4f18-a5c6-9871ac950890\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " Oct 30 04:20:20.819955 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.819895 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-config-volume\" (UniqueName: \"kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-config-volume\") pod \"06218436-4431-4f18-a5c6-9871ac950890\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " Oct 30 04:20:20.819955 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.819926 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06218436-4431-4f18-a5c6-9871ac950890-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/06218436-4431-4f18-a5c6-9871ac950890-alertmanager-data-science-monitoringstack-db\") pod \"06218436-4431-4f18-a5c6-9871ac950890\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " Oct 30 04:20:20.819955 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.819944 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-web-config\" (UniqueName: \"kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-web-config\") pod \"06218436-4431-4f18-a5c6-9871ac950890\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " Oct 30 04:20:20.820138 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.819975 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06218436-4431-4f18-a5c6-9871ac950890-kube-api-access-x2p4c\" (UniqueName: \"kubernetes.io/projected/06218436-4431-4f18-a5c6-9871ac950890-kube-api-access-x2p4c\") pod \"06218436-4431-4f18-a5c6-9871ac950890\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " Oct 30 04:20:20.820138 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.820109 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06218436-4431-4f18-a5c6-9871ac950890-tls-assets\" (UniqueName: \"kubernetes.io/projected/06218436-4431-4f18-a5c6-9871ac950890-tls-assets\") pod \"06218436-4431-4f18-a5c6-9871ac950890\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " Oct 30 04:20:20.820258 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.820194 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-cluster-tls-config\") pod \"06218436-4431-4f18-a5c6-9871ac950890\" (UID: \"06218436-4431-4f18-a5c6-9871ac950890\") " Oct 30 04:20:20.820314 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.820298 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06218436-4431-4f18-a5c6-9871ac950890-alertmanager-data-science-monitoringstack-db" pod "06218436-4431-4f18-a5c6-9871ac950890" (UID: "06218436-4431-4f18-a5c6-9871ac950890"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:20:20.821305 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.821278 4925 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/06218436-4431-4f18-a5c6-9871ac950890-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:20.822490 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.822462 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-config-volume" pod "06218436-4431-4f18-a5c6-9871ac950890" (UID: "06218436-4431-4f18-a5c6-9871ac950890"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:20.822585 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.822486 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06218436-4431-4f18-a5c6-9871ac950890-tls-assets" pod "06218436-4431-4f18-a5c6-9871ac950890" (UID: "06218436-4431-4f18-a5c6-9871ac950890"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:20.823074 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.823049 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06218436-4431-4f18-a5c6-9871ac950890-kube-api-access-x2p4c" pod "06218436-4431-4f18-a5c6-9871ac950890" (UID: "06218436-4431-4f18-a5c6-9871ac950890"). InnerVolumeSpecName "kube-api-access-x2p4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:20.823270 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.823253 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06218436-4431-4f18-a5c6-9871ac950890-config-out" pod "06218436-4431-4f18-a5c6-9871ac950890" (UID: "06218436-4431-4f18-a5c6-9871ac950890"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:20:20.827642 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.827622 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-cluster-tls-config" pod "06218436-4431-4f18-a5c6-9871ac950890" (UID: "06218436-4431-4f18-a5c6-9871ac950890"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:20.834239 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.834217 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-web-config" pod "06218436-4431-4f18-a5c6-9871ac950890" (UID: "06218436-4431-4f18-a5c6-9871ac950890"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:20.922559 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.922528 4925 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-config-volume\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:20.922559 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.922555 4925 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-web-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:20.922559 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.922565 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2p4c\" (UniqueName: \"kubernetes.io/projected/06218436-4431-4f18-a5c6-9871ac950890-kube-api-access-x2p4c\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:20.922805 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.922575 4925 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06218436-4431-4f18-a5c6-9871ac950890-tls-assets\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:20.922805 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.922584 4925 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06218436-4431-4f18-a5c6-9871ac950890-cluster-tls-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:20.922805 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:20.922592 4925 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06218436-4431-4f18-a5c6-9871ac950890-config-out\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:21.128242 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:21.128210 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:20:21.140586 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:21.140557 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:20:22.388185 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.388136 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:20:22.388675 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.388655 4925 cpu_manager.go:414] "containerMap: removing container" podUID="06218436-4431-4f18-a5c6-9871ac950890" containerName="alertmanager" Oct 30 04:20:22.388740 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.388676 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06218436-4431-4f18-a5c6-9871ac950890" containerName="alertmanager" Oct 30 04:20:22.388740 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.388699 4925 cpu_manager.go:414] "containerMap: removing container" podUID="06218436-4431-4f18-a5c6-9871ac950890" containerName="config-reloader" Oct 30 04:20:22.388740 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.388704 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06218436-4431-4f18-a5c6-9871ac950890" containerName="config-reloader" Oct 30 04:20:22.388740 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.388713 4925 cpu_manager.go:414] "containerMap: removing container" podUID="06218436-4431-4f18-a5c6-9871ac950890" containerName="init-config-reloader" Oct 30 04:20:22.388740 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.388719 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06218436-4431-4f18-a5c6-9871ac950890" containerName="init-config-reloader" Oct 30 04:20:22.389017 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.388801 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="06218436-4431-4f18-a5c6-9871ac950890" containerName="config-reloader" Oct 30 04:20:22.389017 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.388815 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="06218436-4431-4f18-a5c6-9871ac950890" containerName="alertmanager" Oct 30 04:20:22.393868 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.393842 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.425177 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.425148 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:20:22.434995 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.434969 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/5a655f24-a28d-45ee-ad1d-d395051e550a-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.435117 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.435009 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a655f24-a28d-45ee-ad1d-d395051e550a-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.435117 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.435032 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.435219 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.435143 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a655f24-a28d-45ee-ad1d-d395051e550a-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.435219 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.435211 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvvkz\" (UniqueName: \"kubernetes.io/projected/5a655f24-a28d-45ee-ad1d-d395051e550a-kube-api-access-pvvkz\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.435287 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.435243 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.435346 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.435330 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.476091 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.476048 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 04:20:22.476261 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.476166 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 04:20:22.476261 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.476188 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 04:20:22.476261 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.476194 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-fp654\"" Oct 30 04:20:22.476261 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.476227 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 04:20:22.535766 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.535720 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/5a655f24-a28d-45ee-ad1d-d395051e550a-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.535951 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.535787 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a655f24-a28d-45ee-ad1d-d395051e550a-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.535951 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.535810 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.535951 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.535827 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a655f24-a28d-45ee-ad1d-d395051e550a-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.535951 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.535848 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvvkz\" (UniqueName: \"kubernetes.io/projected/5a655f24-a28d-45ee-ad1d-d395051e550a-kube-api-access-pvvkz\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.535951 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.535872 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.535951 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.535909 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.536290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.536155 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/5a655f24-a28d-45ee-ad1d-d395051e550a-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.538375 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.538352 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.538517 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.538445 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.538517 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.538452 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.538517 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.538487 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a655f24-a28d-45ee-ad1d-d395051e550a-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.538641 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.538597 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a655f24-a28d-45ee-ad1d-d395051e550a-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.559449 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.559418 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvvkz\" (UniqueName: \"kubernetes.io/projected/5a655f24-a28d-45ee-ad1d-d395051e550a-kube-api-access-pvvkz\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:22.715285 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:22.715199 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:23.049369 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:23.049343 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:20:23.051809 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:20:23.051771 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a655f24_a28d_45ee_ad1d_d395051e550a.slice/crio-19a0553f07add9f88f873f6508774976e4265419a5b38e0f9ba5bbd0c4ddfe9f WatchSource:0}: Error finding container 19a0553f07add9f88f873f6508774976e4265419a5b38e0f9ba5bbd0c4ddfe9f: Status 404 returned error can't find the container with id 19a0553f07add9f88f873f6508774976e4265419a5b38e0f9ba5bbd0c4ddfe9f Oct 30 04:20:23.119455 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:23.119425 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06218436-4431-4f18-a5c6-9871ac950890" path="/var/lib/kubelet/pods/06218436-4431-4f18-a5c6-9871ac950890/volumes" Oct 30 04:20:23.807696 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:23.807607 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"5a655f24-a28d-45ee-ad1d-d395051e550a","Type":"ContainerStarted","Data":"fc662faa47a9e9a14c67a364ef539a74f05f697753b8541c4689484099fb6706"} Oct 30 04:20:23.807696 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:23.807669 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"5a655f24-a28d-45ee-ad1d-d395051e550a","Type":"ContainerStarted","Data":"19a0553f07add9f88f873f6508774976e4265419a5b38e0f9ba5bbd0c4ddfe9f"} Oct 30 04:20:26.790306 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:26.790275 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:27.831176 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:27.831143 4925 generic.go:356] "Generic (PLEG): container finished" podID="5a655f24-a28d-45ee-ad1d-d395051e550a" containerID="fc662faa47a9e9a14c67a364ef539a74f05f697753b8541c4689484099fb6706" exitCode=0 Oct 30 04:20:27.831570 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:27.831219 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"5a655f24-a28d-45ee-ad1d-d395051e550a","Type":"ContainerDied","Data":"fc662faa47a9e9a14c67a364ef539a74f05f697753b8541c4689484099fb6706"} Oct 30 04:20:28.840333 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:28.840297 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"5a655f24-a28d-45ee-ad1d-d395051e550a","Type":"ContainerStarted","Data":"e726aae75fe3d1f1c5ef73a037d017bfe2cfd1e2d38f0c47b71444e568d8d7aa"} Oct 30 04:20:28.840677 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:28.840392 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"5a655f24-a28d-45ee-ad1d-d395051e550a","Type":"ContainerStarted","Data":"74403dee8fed26680e92b48124e58174fa97757b7f69dc6ee388668af34cd8a5"} Oct 30 04:20:28.840677 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:28.840583 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:28.913816 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:28.913746 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.42855847 podStartE2EDuration="6.913731365s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.485172895s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:28.900641323 +0000 UTC m=+5989.913919417" watchObservedRunningTime="2026-10-30 04:20:28.913731365 +0000 UTC m=+5989.927009469" Oct 30 04:20:29.778350 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:29.778315 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" Oct 30 04:20:34.847055 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:34.847026 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:20:39.741347 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:39.741235 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 04:20:39.750877 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:39.750854 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 04:20:39.761198 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:39.761172 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 04:20:39.773060 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:39.773041 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b995w_1bac52de-7a27-4fce-828b-9eedcfd60de5/ovn-acl-logging/0.log" Oct 30 04:20:39.983498 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:39.983463 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h"] Oct 30 04:20:39.987100 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:39.987076 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" Oct 30 04:20:40.001995 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:40.001970 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 30 04:20:40.014426 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:40.014402 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-d5kln\"" Oct 30 04:20:40.014545 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:40.014439 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 30 04:20:40.016163 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:40.016142 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h"] Oct 30 04:20:40.084562 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:40.084533 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/d636c395-8d40-43f9-aa83-60e89193951f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6d55967598-v7c4h\" (UID: \"d636c395-8d40-43f9-aa83-60e89193951f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" Oct 30 04:20:40.084774 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:40.084569 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/d636c395-8d40-43f9-aa83-60e89193951f-ta-internal\") pod \"data-science-collector-targetallocator-6d55967598-v7c4h\" (UID: \"d636c395-8d40-43f9-aa83-60e89193951f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" Oct 30 04:20:40.084774 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:40.084718 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9ffp\" (UniqueName: \"kubernetes.io/projected/d636c395-8d40-43f9-aa83-60e89193951f-kube-api-access-p9ffp\") pod \"data-science-collector-targetallocator-6d55967598-v7c4h\" (UID: \"d636c395-8d40-43f9-aa83-60e89193951f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" Oct 30 04:20:40.185712 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:40.185670 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9ffp\" (UniqueName: \"kubernetes.io/projected/d636c395-8d40-43f9-aa83-60e89193951f-kube-api-access-p9ffp\") pod \"data-science-collector-targetallocator-6d55967598-v7c4h\" (UID: \"d636c395-8d40-43f9-aa83-60e89193951f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" Oct 30 04:20:40.185926 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:40.185738 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/d636c395-8d40-43f9-aa83-60e89193951f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6d55967598-v7c4h\" (UID: \"d636c395-8d40-43f9-aa83-60e89193951f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" Oct 30 04:20:40.185926 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:40.185792 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/d636c395-8d40-43f9-aa83-60e89193951f-ta-internal\") pod \"data-science-collector-targetallocator-6d55967598-v7c4h\" (UID: \"d636c395-8d40-43f9-aa83-60e89193951f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" Oct 30 04:20:40.186465 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:40.186439 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/d636c395-8d40-43f9-aa83-60e89193951f-ta-internal\") pod \"data-science-collector-targetallocator-6d55967598-v7c4h\" (UID: \"d636c395-8d40-43f9-aa83-60e89193951f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" Oct 30 04:20:40.188245 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:40.188220 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/d636c395-8d40-43f9-aa83-60e89193951f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6d55967598-v7c4h\" (UID: \"d636c395-8d40-43f9-aa83-60e89193951f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" Oct 30 04:20:40.208943 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:40.208915 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9ffp\" (UniqueName: \"kubernetes.io/projected/d636c395-8d40-43f9-aa83-60e89193951f-kube-api-access-p9ffp\") pod \"data-science-collector-targetallocator-6d55967598-v7c4h\" (UID: \"d636c395-8d40-43f9-aa83-60e89193951f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" Oct 30 04:20:40.298618 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:40.298536 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" Oct 30 04:20:40.604791 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:40.604744 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h"] Oct 30 04:20:40.613476 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:20:40.613442 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd636c395_8d40_43f9_aa83_60e89193951f.slice/crio-7bcd848755f349ed5cc2b46221e7a66367cae1ccabf00c58d7fa27058b3d40b1 WatchSource:0}: Error finding container 7bcd848755f349ed5cc2b46221e7a66367cae1ccabf00c58d7fa27058b3d40b1: Status 404 returned error can't find the container with id 7bcd848755f349ed5cc2b46221e7a66367cae1ccabf00c58d7fa27058b3d40b1 Oct 30 04:20:40.615940 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:40.615904 4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:20:40.898661 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:40.898583 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" event={"ID":"d636c395-8d40-43f9-aa83-60e89193951f","Type":"ContainerStarted","Data":"7bcd848755f349ed5cc2b46221e7a66367cae1ccabf00c58d7fa27058b3d40b1"} Oct 30 04:20:41.478966 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.478927 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:20:41.484635 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.484614 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:41.514395 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.514367 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:20:41.514542 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.514473 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:20:41.514776 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.514729 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:20:41.516609 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.516581 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:20:41.526989 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.526967 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:20:41.527101 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.527010 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-r7bt7\"" Oct 30 04:20:41.600049 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.600008 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:41.600227 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.600129 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:41.600227 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.600192 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:41.600367 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.600287 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:41.600367 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.600312 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcbrj\" (UniqueName: \"kubernetes.io/projected/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-kube-api-access-gcbrj\") pod \"data-science-collector-collector-1\" (UID: \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:41.701501 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.701466 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:41.701668 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.701513 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:41.701668 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.701554 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:41.701668 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.701581 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcbrj\" (UniqueName: \"kubernetes.io/projected/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-kube-api-access-gcbrj\") pod \"data-science-collector-collector-1\" (UID: \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:41.701668 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.701653 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:41.702336 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.702311 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:41.704342 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.704317 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:41.704453 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.704339 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:41.704453 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.704329 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:41.725869 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.725845 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcbrj\" (UniqueName: \"kubernetes.io/projected/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-kube-api-access-gcbrj\") pod \"data-science-collector-collector-1\" (UID: \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:41.798523 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:41.798453 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:42.106410 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:42.106374 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:20:42.110566 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:20:42.110535 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e50148_0bcb_475b_b9a3_477b06fb1bb3.slice/crio-2348abb0b80f6c24a5d19b6bc4c6853467546e409da57607ef7bad2bb30c42f8 WatchSource:0}: Error finding container 2348abb0b80f6c24a5d19b6bc4c6853467546e409da57607ef7bad2bb30c42f8: Status 404 returned error can't find the container with id 2348abb0b80f6c24a5d19b6bc4c6853467546e409da57607ef7bad2bb30c42f8 Oct 30 04:20:42.113180 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:42.113141 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:20:42.113279 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:42.113214 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:20:42.808381 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:42.808352 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:20:42.808504 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:42.808411 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:20:42.914201 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:42.914165 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" event={"ID":"d636c395-8d40-43f9-aa83-60e89193951f","Type":"ContainerStarted","Data":"40cc41d1d38864e1d79e9028a3b8d3a47062b709782cb348c6288829a19c3ecc"} Oct 30 04:20:42.914389 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:42.914222 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" Oct 30 04:20:42.915331 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:42.915303 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" podUID="d636c395-8d40-43f9-aa83-60e89193951f" containerName="ta-container" probeResult="failure" output="Get \"http://10.132.0.160:8080/readyz\": dial tcp 10.132.0.160:8080: connect: connection refused" Oct 30 04:20:42.915975 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:42.915955 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b9e50148-0bcb-475b-b9a3-477b06fb1bb3","Type":"ContainerStarted","Data":"f05ccd1c7786a13ca7d0dda7d0a812bcf0a05022391be6ffe0fea3cdad07dc30"} Oct 30 04:20:42.916044 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:42.915995 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b9e50148-0bcb-475b-b9a3-477b06fb1bb3","Type":"ContainerStarted","Data":"2348abb0b80f6c24a5d19b6bc4c6853467546e409da57607ef7bad2bb30c42f8"} Oct 30 04:20:42.955814 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:42.955764 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" podStartSLOduration=1.764628538 podStartE2EDuration="3.955738734s" totalImagesPullingTime="2.191110196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:42.954708578 +0000 UTC m=+6003.967986685" watchObservedRunningTime="2026-10-30 04:20:42.955738734 +0000 UTC m=+6003.969016837" Oct 30 04:20:42.998395 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:42.998349 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.998336924 podStartE2EDuration="1.998336924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:42.996438844 +0000 UTC m=+6004.009716948" watchObservedRunningTime="2026-10-30 04:20:42.998336924 +0000 UTC m=+6004.011615026" Oct 30 04:20:43.921871 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:43.921840 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" Oct 30 04:20:48.161705 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:20:48.161668 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:20:54.311688 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:54.311651 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:20:54.312126 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:54.311872 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="b9e50148-0bcb-475b-b9a3-477b06fb1bb3" containerName="otc-container" containerID="cri-o://f05ccd1c7786a13ca7d0dda7d0a812bcf0a05022391be6ffe0fea3cdad07dc30" gracePeriod=30 Oct 30 04:20:55.570335 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.570307 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:55.727186 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.727097 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-monitoring-tls-certs\") pod \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\" (UID: \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\") " Oct 30 04:20:55.727186 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.727175 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-tls-certs\") pod \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\" (UID: \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\") " Oct 30 04:20:55.727418 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.727210 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-data-science-collector-ta-client-cert\") pod \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\" (UID: \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\") " Oct 30 04:20:55.727418 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.727252 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-otc-internal\" (UniqueName: \"kubernetes.io/configmap/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-otc-internal\") pod \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\" (UID: \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\") " Oct 30 04:20:55.727418 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.727274 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-kube-api-access-gcbrj\" (UniqueName: \"kubernetes.io/projected/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-kube-api-access-gcbrj\") pod \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\" (UID: \"b9e50148-0bcb-475b-b9a3-477b06fb1bb3\") " Oct 30 04:20:55.727654 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.727617 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-otc-internal" pod "b9e50148-0bcb-475b-b9a3-477b06fb1bb3" (UID: "b9e50148-0bcb-475b-b9a3-477b06fb1bb3"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:55.729408 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.729379 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-tls-certs" pod "b9e50148-0bcb-475b-b9a3-477b06fb1bb3" (UID: "b9e50148-0bcb-475b-b9a3-477b06fb1bb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:55.729556 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.729489 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-kube-api-access-gcbrj" pod "b9e50148-0bcb-475b-b9a3-477b06fb1bb3" (UID: "b9e50148-0bcb-475b-b9a3-477b06fb1bb3"). InnerVolumeSpecName "kube-api-access-gcbrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:55.729811 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.729791 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-monitoring-tls-certs" pod "b9e50148-0bcb-475b-b9a3-477b06fb1bb3" (UID: "b9e50148-0bcb-475b-b9a3-477b06fb1bb3"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:55.729893 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.729820 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-data-science-collector-ta-client-cert" pod "b9e50148-0bcb-475b-b9a3-477b06fb1bb3" (UID: "b9e50148-0bcb-475b-b9a3-477b06fb1bb3"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:55.828418 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.828382 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:55.828418 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.828413 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:55.828418 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.828426 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:55.828667 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.828435 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:55.828667 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.828445 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcbrj\" (UniqueName: \"kubernetes.io/projected/b9e50148-0bcb-475b-b9a3-477b06fb1bb3-kube-api-access-gcbrj\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:55.982815 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.982711 4925 generic.go:356] "Generic (PLEG): container finished" podID="b9e50148-0bcb-475b-b9a3-477b06fb1bb3" containerID="f05ccd1c7786a13ca7d0dda7d0a812bcf0a05022391be6ffe0fea3cdad07dc30" exitCode=0 Oct 30 04:20:55.982815 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.982775 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:55.983052 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.982777 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b9e50148-0bcb-475b-b9a3-477b06fb1bb3","Type":"ContainerDied","Data":"f05ccd1c7786a13ca7d0dda7d0a812bcf0a05022391be6ffe0fea3cdad07dc30"} Oct 30 04:20:55.983052 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.982896 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b9e50148-0bcb-475b-b9a3-477b06fb1bb3","Type":"ContainerDied","Data":"2348abb0b80f6c24a5d19b6bc4c6853467546e409da57607ef7bad2bb30c42f8"} Oct 30 04:20:55.983052 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.982917 4925 scope.go:122] "RemoveContainer" containerID="f05ccd1c7786a13ca7d0dda7d0a812bcf0a05022391be6ffe0fea3cdad07dc30" Oct 30 04:20:55.992684 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.992660 4925 scope.go:122] "RemoveContainer" containerID="f05ccd1c7786a13ca7d0dda7d0a812bcf0a05022391be6ffe0fea3cdad07dc30" Oct 30 04:20:55.992936 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:20:55.992917 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f05ccd1c7786a13ca7d0dda7d0a812bcf0a05022391be6ffe0fea3cdad07dc30\": container with ID starting with f05ccd1c7786a13ca7d0dda7d0a812bcf0a05022391be6ffe0fea3cdad07dc30 not found: ID does not exist" containerID="f05ccd1c7786a13ca7d0dda7d0a812bcf0a05022391be6ffe0fea3cdad07dc30" Oct 30 04:20:55.992993 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:55.992946 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f05ccd1c7786a13ca7d0dda7d0a812bcf0a05022391be6ffe0fea3cdad07dc30"} err="failed to get container status \"f05ccd1c7786a13ca7d0dda7d0a812bcf0a05022391be6ffe0fea3cdad07dc30\": rpc error: code = NotFound desc = could not find container \"f05ccd1c7786a13ca7d0dda7d0a812bcf0a05022391be6ffe0fea3cdad07dc30\": container with ID starting with f05ccd1c7786a13ca7d0dda7d0a812bcf0a05022391be6ffe0fea3cdad07dc30 not found: ID does not exist" Oct 30 04:20:56.035152 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.035121 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:20:56.047514 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.047488 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:20:56.077760 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.077730 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:20:56.078114 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.078101 4925 cpu_manager.go:414] "containerMap: removing container" podUID="b9e50148-0bcb-475b-b9a3-477b06fb1bb3" containerName="otc-container" Oct 30 04:20:56.078161 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.078115 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9e50148-0bcb-475b-b9a3-477b06fb1bb3" containerName="otc-container" Oct 30 04:20:56.078202 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.078192 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9e50148-0bcb-475b-b9a3-477b06fb1bb3" containerName="otc-container" Oct 30 04:20:56.082635 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.082610 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:56.097884 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.097859 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\"" Oct 30 04:20:56.102992 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.102965 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:20:56.114132 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.114093 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-r7bt7\"" Oct 30 04:20:56.114263 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.114252 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 04:20:56.114329 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.114269 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 04:20:56.114386 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.114346 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 04:20:56.231887 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.231855 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e80e59c5-dfc1-4874-bd19-76a660c353c5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e80e59c5-dfc1-4874-bd19-76a660c353c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:56.231887 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.231893 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e80e59c5-dfc1-4874-bd19-76a660c353c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:56.232167 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.232088 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e80e59c5-dfc1-4874-bd19-76a660c353c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:56.232167 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.232139 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e80e59c5-dfc1-4874-bd19-76a660c353c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:56.232281 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.232236 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6rhf\" (UniqueName: \"kubernetes.io/projected/e80e59c5-dfc1-4874-bd19-76a660c353c5-kube-api-access-g6rhf\") pod \"data-science-collector-collector-1\" (UID: \"e80e59c5-dfc1-4874-bd19-76a660c353c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:56.333184 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.333151 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e80e59c5-dfc1-4874-bd19-76a660c353c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:56.333379 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.333195 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e80e59c5-dfc1-4874-bd19-76a660c353c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:56.333379 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.333228 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rhf\" (UniqueName: \"kubernetes.io/projected/e80e59c5-dfc1-4874-bd19-76a660c353c5-kube-api-access-g6rhf\") pod \"data-science-collector-collector-1\" (UID: \"e80e59c5-dfc1-4874-bd19-76a660c353c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:56.333379 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.333253 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e80e59c5-dfc1-4874-bd19-76a660c353c5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e80e59c5-dfc1-4874-bd19-76a660c353c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:56.333379 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.333274 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e80e59c5-dfc1-4874-bd19-76a660c353c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:56.334020 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.333994 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e80e59c5-dfc1-4874-bd19-76a660c353c5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e80e59c5-dfc1-4874-bd19-76a660c353c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:56.335600 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.335572 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e80e59c5-dfc1-4874-bd19-76a660c353c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:56.335712 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.335649 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e80e59c5-dfc1-4874-bd19-76a660c353c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:56.335712 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.335685 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e80e59c5-dfc1-4874-bd19-76a660c353c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:56.358399 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.358365 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6rhf\" (UniqueName: \"kubernetes.io/projected/e80e59c5-dfc1-4874-bd19-76a660c353c5-kube-api-access-g6rhf\") pod \"data-science-collector-collector-1\" (UID: \"e80e59c5-dfc1-4874-bd19-76a660c353c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:56.386028 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.385994 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h"] Oct 30 04:20:56.386238 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.386206 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" podUID="d636c395-8d40-43f9-aa83-60e89193951f" containerName="ta-container" containerID="cri-o://40cc41d1d38864e1d79e9028a3b8d3a47062b709782cb348c6288829a19c3ecc" gracePeriod=30 Oct 30 04:20:56.395181 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.395155 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:20:56.631970 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.631943 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" Oct 30 04:20:56.699163 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.699136 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:20:56.701914 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:20:56.701886 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80e59c5_dfc1_4874_bd19_76a660c353c5.slice/crio-be80acea67234f8f707117b78066fa153d88194994ff0a0086bb96a381a5751a WatchSource:0}: Error finding container be80acea67234f8f707117b78066fa153d88194994ff0a0086bb96a381a5751a: Status 404 returned error can't find the container with id be80acea67234f8f707117b78066fa153d88194994ff0a0086bb96a381a5751a Oct 30 04:20:56.704115 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.704083 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:20:56.704215 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.704139 4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 04:20:56.736619 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.736596 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d636c395-8d40-43f9-aa83-60e89193951f-kube-api-access-p9ffp\" (UniqueName: \"kubernetes.io/projected/d636c395-8d40-43f9-aa83-60e89193951f-kube-api-access-p9ffp\") pod \"d636c395-8d40-43f9-aa83-60e89193951f\" (UID: \"d636c395-8d40-43f9-aa83-60e89193951f\") " Oct 30 04:20:56.736712 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.736639 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d636c395-8d40-43f9-aa83-60e89193951f-ta-internal\" (UniqueName: \"kubernetes.io/configmap/d636c395-8d40-43f9-aa83-60e89193951f-ta-internal\") pod \"d636c395-8d40-43f9-aa83-60e89193951f\" (UID: \"d636c395-8d40-43f9-aa83-60e89193951f\") " Oct 30 04:20:56.736845 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.736714 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d636c395-8d40-43f9-aa83-60e89193951f-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/d636c395-8d40-43f9-aa83-60e89193951f-data-science-collector-ta-server-cert\") pod \"d636c395-8d40-43f9-aa83-60e89193951f\" (UID: \"d636c395-8d40-43f9-aa83-60e89193951f\") " Oct 30 04:20:56.737099 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.737075 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d636c395-8d40-43f9-aa83-60e89193951f-ta-internal" pod "d636c395-8d40-43f9-aa83-60e89193951f" (UID: "d636c395-8d40-43f9-aa83-60e89193951f"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:56.738699 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.738665 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d636c395-8d40-43f9-aa83-60e89193951f-kube-api-access-p9ffp" pod "d636c395-8d40-43f9-aa83-60e89193951f" (UID: "d636c395-8d40-43f9-aa83-60e89193951f"). InnerVolumeSpecName "kube-api-access-p9ffp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:56.738812 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.738702 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d636c395-8d40-43f9-aa83-60e89193951f-data-science-collector-ta-server-cert" pod "d636c395-8d40-43f9-aa83-60e89193951f" (UID: "d636c395-8d40-43f9-aa83-60e89193951f"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:56.838174 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.838137 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9ffp\" (UniqueName: \"kubernetes.io/projected/d636c395-8d40-43f9-aa83-60e89193951f-kube-api-access-p9ffp\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:56.838174 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.838171 4925 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/d636c395-8d40-43f9-aa83-60e89193951f-ta-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:56.838174 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.838181 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/d636c395-8d40-43f9-aa83-60e89193951f-data-science-collector-ta-server-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:56.989730 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.989633 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e80e59c5-dfc1-4874-bd19-76a660c353c5","Type":"ContainerStarted","Data":"065d794106a3035a84d9a428497a10e188aef42683c5ee490738672b4109eaf8"} Oct 30 04:20:56.989730 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.989703 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e80e59c5-dfc1-4874-bd19-76a660c353c5","Type":"ContainerStarted","Data":"be80acea67234f8f707117b78066fa153d88194994ff0a0086bb96a381a5751a"} Oct 30 04:20:56.991921 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.991896 4925 generic.go:356] "Generic (PLEG): container finished" podID="d636c395-8d40-43f9-aa83-60e89193951f" containerID="40cc41d1d38864e1d79e9028a3b8d3a47062b709782cb348c6288829a19c3ecc" exitCode=0 Oct 30 04:20:56.992039 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.991925 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" Oct 30 04:20:56.992039 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.991928 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" event={"ID":"d636c395-8d40-43f9-aa83-60e89193951f","Type":"ContainerDied","Data":"40cc41d1d38864e1d79e9028a3b8d3a47062b709782cb348c6288829a19c3ecc"} Oct 30 04:20:56.992121 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.992066 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h" event={"ID":"d636c395-8d40-43f9-aa83-60e89193951f","Type":"ContainerDied","Data":"7bcd848755f349ed5cc2b46221e7a66367cae1ccabf00c58d7fa27058b3d40b1"} Oct 30 04:20:56.992121 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:56.992091 4925 scope.go:122] "RemoveContainer" containerID="40cc41d1d38864e1d79e9028a3b8d3a47062b709782cb348c6288829a19c3ecc" Oct 30 04:20:57.000864 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:57.000843 4925 scope.go:122] "RemoveContainer" containerID="40cc41d1d38864e1d79e9028a3b8d3a47062b709782cb348c6288829a19c3ecc" Oct 30 04:20:57.001132 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:20:57.001115 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40cc41d1d38864e1d79e9028a3b8d3a47062b709782cb348c6288829a19c3ecc\": container with ID starting with 40cc41d1d38864e1d79e9028a3b8d3a47062b709782cb348c6288829a19c3ecc not found: ID does not exist" containerID="40cc41d1d38864e1d79e9028a3b8d3a47062b709782cb348c6288829a19c3ecc" Oct 30 04:20:57.001204 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:57.001138 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40cc41d1d38864e1d79e9028a3b8d3a47062b709782cb348c6288829a19c3ecc"} err="failed to get container status \"40cc41d1d38864e1d79e9028a3b8d3a47062b709782cb348c6288829a19c3ecc\": rpc error: code = NotFound desc = could not find container \"40cc41d1d38864e1d79e9028a3b8d3a47062b709782cb348c6288829a19c3ecc\": container with ID starting with 40cc41d1d38864e1d79e9028a3b8d3a47062b709782cb348c6288829a19c3ecc not found: ID does not exist" Oct 30 04:20:57.033489 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:57.033435 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.033421989 podStartE2EDuration="1.033421989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:57.03315053 +0000 UTC m=+6018.046428633" watchObservedRunningTime="2026-10-30 04:20:57.033421989 +0000 UTC m=+6018.046700092" Oct 30 04:20:57.070706 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:57.070666 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h"] Oct 30 04:20:57.076246 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:57.076219 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6d55967598-v7c4h"] Oct 30 04:20:57.123653 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:57.123616 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9e50148-0bcb-475b-b9a3-477b06fb1bb3" path="/var/lib/kubelet/pods/b9e50148-0bcb-475b-b9a3-477b06fb1bb3/volumes" Oct 30 04:20:57.124031 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:57.124017 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d636c395-8d40-43f9-aa83-60e89193951f" path="/var/lib/kubelet/pods/d636c395-8d40-43f9-aa83-60e89193951f/volumes" Oct 30 04:20:59.098806 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.098740 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk"] Oct 30 04:20:59.099223 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.098960 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" podUID="b8f035ab-52f5-46c0-9bbe-9e94699c3416" containerName="kube-rbac-proxy" containerID="cri-o://3de50932319ffa0936aa930a9619281590462d7365d42201aa42d9b4eaad919a" gracePeriod=30 Oct 30 04:20:59.112092 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.112065 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s"] Oct 30 04:20:59.112291 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.112271 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" podUID="91407852-200e-4ce6-89a1-ef11eed5a272" containerName="kube-rbac-proxy" containerID="cri-o://d1be7bbe29b72950685ddefbfe22ceefa1a3aaa877ee81415ffab6fa1fc3ef1c" gracePeriod=30 Oct 30 04:20:59.112352 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.112275 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" podUID="91407852-200e-4ce6-89a1-ef11eed5a272" containerName="prom-label-proxy" containerID="cri-o://1233b6cb419d7a7dde39020a96cef5bd539e1f9534e76896ad064925c860c562" gracePeriod=30 Oct 30 04:20:59.150409 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.150371 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g"] Oct 30 04:20:59.150612 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.150588 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" podUID="7d087d6e-df54-4262-90e3-86142f1a39cc" containerName="korrel8r" containerID="cri-o://4b3883b0c719dd186a92f786766730eb2802eda4255c3aaee2d6d2c1de03fd8b" gracePeriod=30 Oct 30 04:20:59.407093 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.407066 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:20:59.434188 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.434159 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" Oct 30 04:20:59.437699 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.437670 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" Oct 30 04:20:59.460304 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.460267 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-prometheus-tls-ca\") pod \"91407852-200e-4ce6-89a1-ef11eed5a272\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " Oct 30 04:20:59.460450 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.460408 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-config\" (UniqueName: \"kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-config\") pod \"91407852-200e-4ce6-89a1-ef11eed5a272\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " Oct 30 04:20:59.460450 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.460443 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91407852-200e-4ce6-89a1-ef11eed5a272-kube-api-access-7wlxc\" (UniqueName: \"kubernetes.io/projected/91407852-200e-4ce6-89a1-ef11eed5a272-kube-api-access-7wlxc\") pod \"91407852-200e-4ce6-89a1-ef11eed5a272\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " Oct 30 04:20:59.460556 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.460474 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8f035ab-52f5-46c0-9bbe-9e94699c3416-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/b8f035ab-52f5-46c0-9bbe-9e94699c3416-prometheus-ca\") pod \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " Oct 30 04:20:59.460556 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.460506 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d087d6e-df54-4262-90e3-86142f1a39cc-kube-api-access-blqh7\" (UniqueName: \"kubernetes.io/projected/7d087d6e-df54-4262-90e3-86142f1a39cc-kube-api-access-blqh7\") pod \"7d087d6e-df54-4262-90e3-86142f1a39cc\" (UID: \"7d087d6e-df54-4262-90e3-86142f1a39cc\") " Oct 30 04:20:59.460556 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.460544 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-kube-rbac-proxy-config\") pod \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " Oct 30 04:20:59.460722 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.460604 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8f035ab-52f5-46c0-9bbe-9e94699c3416-kube-api-access-ztv64\" (UniqueName: \"kubernetes.io/projected/b8f035ab-52f5-46c0-9bbe-9e94699c3416-kube-api-access-ztv64\") pod \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " Oct 30 04:20:59.460722 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.460632 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91407852-200e-4ce6-89a1-ef11eed5a272-tls\" (UniqueName: \"kubernetes.io/secret/91407852-200e-4ce6-89a1-ef11eed5a272-tls\") pod \"91407852-200e-4ce6-89a1-ef11eed5a272\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " Oct 30 04:20:59.460722 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.460678 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d087d6e-df54-4262-90e3-86142f1a39cc-config\" (UniqueName: \"kubernetes.io/configmap/7d087d6e-df54-4262-90e3-86142f1a39cc-config\") pod \"7d087d6e-df54-4262-90e3-86142f1a39cc\" (UID: \"7d087d6e-df54-4262-90e3-86142f1a39cc\") " Oct 30 04:20:59.460722 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.460711 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-client-ca\" (UniqueName: \"kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-client-ca\") pod \"91407852-200e-4ce6-89a1-ef11eed5a272\" (UID: \"91407852-200e-4ce6-89a1-ef11eed5a272\") " Oct 30 04:20:59.460956 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.460743 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d087d6e-df54-4262-90e3-86142f1a39cc-tls\" (UniqueName: \"kubernetes.io/secret/7d087d6e-df54-4262-90e3-86142f1a39cc-tls\") pod \"7d087d6e-df54-4262-90e3-86142f1a39cc\" (UID: \"7d087d6e-df54-4262-90e3-86142f1a39cc\") " Oct 30 04:20:59.460956 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.460791 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-config" pod "91407852-200e-4ce6-89a1-ef11eed5a272" (UID: "91407852-200e-4ce6-89a1-ef11eed5a272"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:59.460956 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.460823 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-tls\" (UniqueName: \"kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-tls\") pod \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " Oct 30 04:20:59.460956 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.460863 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-prometheus-client-cert\") pod \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\" (UID: \"b8f035ab-52f5-46c0-9bbe-9e94699c3416\") " Oct 30 04:20:59.461195 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.461119 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:59.461250 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.461232 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d087d6e-df54-4262-90e3-86142f1a39cc-config" pod "7d087d6e-df54-4262-90e3-86142f1a39cc" (UID: "7d087d6e-df54-4262-90e3-86142f1a39cc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:59.461353 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.461327 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8f035ab-52f5-46c0-9bbe-9e94699c3416-prometheus-ca" pod "b8f035ab-52f5-46c0-9bbe-9e94699c3416" (UID: "b8f035ab-52f5-46c0-9bbe-9e94699c3416"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:59.461631 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.461607 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-prometheus-tls-ca" pod "91407852-200e-4ce6-89a1-ef11eed5a272" (UID: "91407852-200e-4ce6-89a1-ef11eed5a272"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:59.462740 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.462701 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-client-ca" pod "91407852-200e-4ce6-89a1-ef11eed5a272" (UID: "91407852-200e-4ce6-89a1-ef11eed5a272"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:59.464614 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.464584 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-prometheus-client-cert" pod "b8f035ab-52f5-46c0-9bbe-9e94699c3416" (UID: "b8f035ab-52f5-46c0-9bbe-9e94699c3416"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:59.466080 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.465736 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d087d6e-df54-4262-90e3-86142f1a39cc-tls" pod "7d087d6e-df54-4262-90e3-86142f1a39cc" (UID: "7d087d6e-df54-4262-90e3-86142f1a39cc"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:59.466080 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.466063 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91407852-200e-4ce6-89a1-ef11eed5a272-tls" pod "91407852-200e-4ce6-89a1-ef11eed5a272" (UID: "91407852-200e-4ce6-89a1-ef11eed5a272"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:59.468214 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.468148 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-kube-rbac-proxy-config" pod "b8f035ab-52f5-46c0-9bbe-9e94699c3416" (UID: "b8f035ab-52f5-46c0-9bbe-9e94699c3416"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:59.468380 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.468276 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d087d6e-df54-4262-90e3-86142f1a39cc-kube-api-access-blqh7" pod "7d087d6e-df54-4262-90e3-86142f1a39cc" (UID: "7d087d6e-df54-4262-90e3-86142f1a39cc"). InnerVolumeSpecName "kube-api-access-blqh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:59.468630 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.468610 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91407852-200e-4ce6-89a1-ef11eed5a272-kube-api-access-7wlxc" pod "91407852-200e-4ce6-89a1-ef11eed5a272" (UID: "91407852-200e-4ce6-89a1-ef11eed5a272"). InnerVolumeSpecName "kube-api-access-7wlxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:59.469032 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.468997 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f035ab-52f5-46c0-9bbe-9e94699c3416-kube-api-access-ztv64" pod "b8f035ab-52f5-46c0-9bbe-9e94699c3416" (UID: "b8f035ab-52f5-46c0-9bbe-9e94699c3416"). InnerVolumeSpecName "kube-api-access-ztv64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:59.469128 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.469080 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-tls" pod "b8f035ab-52f5-46c0-9bbe-9e94699c3416" (UID: "b8f035ab-52f5-46c0-9bbe-9e94699c3416"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:59.561829 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.561792 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-prometheus-tls-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:59.561829 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.561829 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wlxc\" (UniqueName: \"kubernetes.io/projected/91407852-200e-4ce6-89a1-ef11eed5a272-kube-api-access-7wlxc\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:59.562040 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.561846 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/b8f035ab-52f5-46c0-9bbe-9e94699c3416-prometheus-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:59.562040 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.561861 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blqh7\" (UniqueName: \"kubernetes.io/projected/7d087d6e-df54-4262-90e3-86142f1a39cc-kube-api-access-blqh7\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:59.562040 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.561876 4925 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-kube-rbac-proxy-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:59.562040 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.561891 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztv64\" (UniqueName: \"kubernetes.io/projected/b8f035ab-52f5-46c0-9bbe-9e94699c3416-kube-api-access-ztv64\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:59.562040 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.561905 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/91407852-200e-4ce6-89a1-ef11eed5a272-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:59.562040 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.561918 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7d087d6e-df54-4262-90e3-86142f1a39cc-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:59.562040 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.561930 4925 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/91407852-200e-4ce6-89a1-ef11eed5a272-client-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:59.562040 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.561940 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7d087d6e-df54-4262-90e3-86142f1a39cc-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:59.562040 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.561948 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:59.562040 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.561956 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/b8f035ab-52f5-46c0-9bbe-9e94699c3416-prometheus-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:20:59.670784 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.670682 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:20:59.670971 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.670945 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="5a655f24-a28d-45ee-ad1d-d395051e550a" containerName="alertmanager" containerID="cri-o://74403dee8fed26680e92b48124e58174fa97757b7f69dc6ee388668af34cd8a5" gracePeriod=120 Oct 30 04:20:59.671072 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.670956 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="5a655f24-a28d-45ee-ad1d-d395051e550a" containerName="config-reloader" containerID="cri-o://e726aae75fe3d1f1c5ef73a037d017bfe2cfd1e2d38f0c47b71444e568d8d7aa" gracePeriod=120 Oct 30 04:20:59.846008 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:20:59.845976 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="5a655f24-a28d-45ee-ad1d-d395051e550a" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.159:9093/-/ready\": dial tcp 10.132.0.159:9093: connect: connection refused" Oct 30 04:21:00.008811 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.008708 4925 generic.go:356] "Generic (PLEG): container finished" podID="91407852-200e-4ce6-89a1-ef11eed5a272" containerID="1233b6cb419d7a7dde39020a96cef5bd539e1f9534e76896ad064925c860c562" exitCode=0 Oct 30 04:21:00.008811 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.008737 4925 generic.go:356] "Generic (PLEG): container finished" podID="91407852-200e-4ce6-89a1-ef11eed5a272" containerID="d1be7bbe29b72950685ddefbfe22ceefa1a3aaa877ee81415ffab6fa1fc3ef1c" exitCode=0 Oct 30 04:21:00.008811 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.008779 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" Oct 30 04:21:00.008811 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.008793 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" event={"ID":"91407852-200e-4ce6-89a1-ef11eed5a272","Type":"ContainerDied","Data":"1233b6cb419d7a7dde39020a96cef5bd539e1f9534e76896ad064925c860c562"} Oct 30 04:21:00.009120 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.008849 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" event={"ID":"91407852-200e-4ce6-89a1-ef11eed5a272","Type":"ContainerDied","Data":"d1be7bbe29b72950685ddefbfe22ceefa1a3aaa877ee81415ffab6fa1fc3ef1c"} Oct 30 04:21:00.009120 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.008872 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s" event={"ID":"91407852-200e-4ce6-89a1-ef11eed5a272","Type":"ContainerDied","Data":"ded7d52008abf76312694c655667047a0f6e9e5806ee928d5c5da1abc27dc380"} Oct 30 04:21:00.009120 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.008907 4925 scope.go:122] "RemoveContainer" containerID="1233b6cb419d7a7dde39020a96cef5bd539e1f9534e76896ad064925c860c562" Oct 30 04:21:00.011714 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.011689 4925 generic.go:356] "Generic (PLEG): container finished" podID="5a655f24-a28d-45ee-ad1d-d395051e550a" containerID="e726aae75fe3d1f1c5ef73a037d017bfe2cfd1e2d38f0c47b71444e568d8d7aa" exitCode=0 Oct 30 04:21:00.011852 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.011724 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"5a655f24-a28d-45ee-ad1d-d395051e550a","Type":"ContainerDied","Data":"e726aae75fe3d1f1c5ef73a037d017bfe2cfd1e2d38f0c47b71444e568d8d7aa"} Oct 30 04:21:00.013047 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.013028 4925 generic.go:356] "Generic (PLEG): container finished" podID="b8f035ab-52f5-46c0-9bbe-9e94699c3416" containerID="3de50932319ffa0936aa930a9619281590462d7365d42201aa42d9b4eaad919a" exitCode=0 Oct 30 04:21:00.013176 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.013073 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" Oct 30 04:21:00.013176 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.013117 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" event={"ID":"b8f035ab-52f5-46c0-9bbe-9e94699c3416","Type":"ContainerDied","Data":"3de50932319ffa0936aa930a9619281590462d7365d42201aa42d9b4eaad919a"} Oct 30 04:21:00.013176 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.013169 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk" event={"ID":"b8f035ab-52f5-46c0-9bbe-9e94699c3416","Type":"ContainerDied","Data":"78585e4fd7a156914009ff611ff62583e04f2067e6da9b1ae237cb974ba5d323"} Oct 30 04:21:00.014360 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.014301 4925 generic.go:356] "Generic (PLEG): container finished" podID="7d087d6e-df54-4262-90e3-86142f1a39cc" containerID="4b3883b0c719dd186a92f786766730eb2802eda4255c3aaee2d6d2c1de03fd8b" exitCode=0 Oct 30 04:21:00.014454 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.014360 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" event={"ID":"7d087d6e-df54-4262-90e3-86142f1a39cc","Type":"ContainerDied","Data":"4b3883b0c719dd186a92f786766730eb2802eda4255c3aaee2d6d2c1de03fd8b"} Oct 30 04:21:00.014454 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.014395 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" event={"ID":"7d087d6e-df54-4262-90e3-86142f1a39cc","Type":"ContainerDied","Data":"333f62af95308377932494f82c63f6f23f636ada76b6e292fa280b5efdf97aff"} Oct 30 04:21:00.014454 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.014362 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g" Oct 30 04:21:00.020274 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.020258 4925 scope.go:122] "RemoveContainer" containerID="d1be7bbe29b72950685ddefbfe22ceefa1a3aaa877ee81415ffab6fa1fc3ef1c" Oct 30 04:21:00.030594 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.030566 4925 scope.go:122] "RemoveContainer" containerID="1233b6cb419d7a7dde39020a96cef5bd539e1f9534e76896ad064925c860c562" Oct 30 04:21:00.030898 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:21:00.030880 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1233b6cb419d7a7dde39020a96cef5bd539e1f9534e76896ad064925c860c562\": container with ID starting with 1233b6cb419d7a7dde39020a96cef5bd539e1f9534e76896ad064925c860c562 not found: ID does not exist" containerID="1233b6cb419d7a7dde39020a96cef5bd539e1f9534e76896ad064925c860c562" Oct 30 04:21:00.030962 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.030906 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1233b6cb419d7a7dde39020a96cef5bd539e1f9534e76896ad064925c860c562"} err="failed to get container status \"1233b6cb419d7a7dde39020a96cef5bd539e1f9534e76896ad064925c860c562\": rpc error: code = NotFound desc = could not find container \"1233b6cb419d7a7dde39020a96cef5bd539e1f9534e76896ad064925c860c562\": container with ID starting with 1233b6cb419d7a7dde39020a96cef5bd539e1f9534e76896ad064925c860c562 not found: ID does not exist" Oct 30 04:21:00.030962 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.030923 4925 scope.go:122] "RemoveContainer" containerID="d1be7bbe29b72950685ddefbfe22ceefa1a3aaa877ee81415ffab6fa1fc3ef1c" Oct 30 04:21:00.031200 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:21:00.031179 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1be7bbe29b72950685ddefbfe22ceefa1a3aaa877ee81415ffab6fa1fc3ef1c\": container with ID starting with d1be7bbe29b72950685ddefbfe22ceefa1a3aaa877ee81415ffab6fa1fc3ef1c not found: ID does not exist" containerID="d1be7bbe29b72950685ddefbfe22ceefa1a3aaa877ee81415ffab6fa1fc3ef1c" Oct 30 04:21:00.031250 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.031208 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1be7bbe29b72950685ddefbfe22ceefa1a3aaa877ee81415ffab6fa1fc3ef1c"} err="failed to get container status \"d1be7bbe29b72950685ddefbfe22ceefa1a3aaa877ee81415ffab6fa1fc3ef1c\": rpc error: code = NotFound desc = could not find container \"d1be7bbe29b72950685ddefbfe22ceefa1a3aaa877ee81415ffab6fa1fc3ef1c\": container with ID starting with d1be7bbe29b72950685ddefbfe22ceefa1a3aaa877ee81415ffab6fa1fc3ef1c not found: ID does not exist" Oct 30 04:21:00.031250 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.031223 4925 scope.go:122] "RemoveContainer" containerID="1233b6cb419d7a7dde39020a96cef5bd539e1f9534e76896ad064925c860c562" Oct 30 04:21:00.031432 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.031414 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1233b6cb419d7a7dde39020a96cef5bd539e1f9534e76896ad064925c860c562"} err="failed to get container status \"1233b6cb419d7a7dde39020a96cef5bd539e1f9534e76896ad064925c860c562\": rpc error: code = NotFound desc = could not find container \"1233b6cb419d7a7dde39020a96cef5bd539e1f9534e76896ad064925c860c562\": container with ID starting with 1233b6cb419d7a7dde39020a96cef5bd539e1f9534e76896ad064925c860c562 not found: ID does not exist" Oct 30 04:21:00.031477 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.031433 4925 scope.go:122] "RemoveContainer" containerID="d1be7bbe29b72950685ddefbfe22ceefa1a3aaa877ee81415ffab6fa1fc3ef1c" Oct 30 04:21:00.031645 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.031625 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1be7bbe29b72950685ddefbfe22ceefa1a3aaa877ee81415ffab6fa1fc3ef1c"} err="failed to get container status \"d1be7bbe29b72950685ddefbfe22ceefa1a3aaa877ee81415ffab6fa1fc3ef1c\": rpc error: code = NotFound desc = could not find container \"d1be7bbe29b72950685ddefbfe22ceefa1a3aaa877ee81415ffab6fa1fc3ef1c\": container with ID starting with d1be7bbe29b72950685ddefbfe22ceefa1a3aaa877ee81415ffab6fa1fc3ef1c not found: ID does not exist" Oct 30 04:21:00.031645 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.031645 4925 scope.go:122] "RemoveContainer" containerID="3de50932319ffa0936aa930a9619281590462d7365d42201aa42d9b4eaad919a" Oct 30 04:21:00.040068 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.040050 4925 scope.go:122] "RemoveContainer" containerID="3de50932319ffa0936aa930a9619281590462d7365d42201aa42d9b4eaad919a" Oct 30 04:21:00.040381 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:21:00.040354 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de50932319ffa0936aa930a9619281590462d7365d42201aa42d9b4eaad919a\": container with ID starting with 3de50932319ffa0936aa930a9619281590462d7365d42201aa42d9b4eaad919a not found: ID does not exist" containerID="3de50932319ffa0936aa930a9619281590462d7365d42201aa42d9b4eaad919a" Oct 30 04:21:00.040473 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.040388 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de50932319ffa0936aa930a9619281590462d7365d42201aa42d9b4eaad919a"} err="failed to get container status \"3de50932319ffa0936aa930a9619281590462d7365d42201aa42d9b4eaad919a\": rpc error: code = NotFound desc = could not find container \"3de50932319ffa0936aa930a9619281590462d7365d42201aa42d9b4eaad919a\": container with ID starting with 3de50932319ffa0936aa930a9619281590462d7365d42201aa42d9b4eaad919a not found: ID does not exist" Oct 30 04:21:00.040473 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.040408 4925 scope.go:122] "RemoveContainer" containerID="4b3883b0c719dd186a92f786766730eb2802eda4255c3aaee2d6d2c1de03fd8b" Oct 30 04:21:00.048216 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.048199 4925 scope.go:122] "RemoveContainer" containerID="4b3883b0c719dd186a92f786766730eb2802eda4255c3aaee2d6d2c1de03fd8b" Oct 30 04:21:00.048509 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:21:00.048490 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b3883b0c719dd186a92f786766730eb2802eda4255c3aaee2d6d2c1de03fd8b\": container with ID starting with 4b3883b0c719dd186a92f786766730eb2802eda4255c3aaee2d6d2c1de03fd8b not found: ID does not exist" containerID="4b3883b0c719dd186a92f786766730eb2802eda4255c3aaee2d6d2c1de03fd8b" Oct 30 04:21:00.048579 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.048521 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b3883b0c719dd186a92f786766730eb2802eda4255c3aaee2d6d2c1de03fd8b"} err="failed to get container status \"4b3883b0c719dd186a92f786766730eb2802eda4255c3aaee2d6d2c1de03fd8b\": rpc error: code = NotFound desc = could not find container \"4b3883b0c719dd186a92f786766730eb2802eda4255c3aaee2d6d2c1de03fd8b\": container with ID starting with 4b3883b0c719dd186a92f786766730eb2802eda4255c3aaee2d6d2c1de03fd8b not found: ID does not exist" Oct 30 04:21:00.077223 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.077183 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s"] Oct 30 04:21:00.083147 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.083112 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-mtb9s"] Oct 30 04:21:00.134982 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.134945 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk"] Oct 30 04:21:00.138805 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.138779 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-9t6vk"] Oct 30 04:21:00.175015 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.174986 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g"] Oct 30 04:21:00.179626 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.179592 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lns9g"] Oct 30 04:21:00.264810 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.264717 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:21:00.264959 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.264928 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="e80e59c5-dfc1-4874-bd19-76a660c353c5" containerName="otc-container" containerID="cri-o://065d794106a3035a84d9a428497a10e188aef42683c5ee490738672b4109eaf8" gracePeriod=30 Oct 30 04:21:00.526385 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.526328 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:00.570313 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.570277 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e80e59c5-dfc1-4874-bd19-76a660c353c5-kube-api-access-g6rhf\" (UniqueName: \"kubernetes.io/projected/e80e59c5-dfc1-4874-bd19-76a660c353c5-kube-api-access-g6rhf\") pod \"e80e59c5-dfc1-4874-bd19-76a660c353c5\" (UID: \"e80e59c5-dfc1-4874-bd19-76a660c353c5\") " Oct 30 04:21:00.570494 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.570329 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-tls-certs\") pod \"e80e59c5-dfc1-4874-bd19-76a660c353c5\" (UID: \"e80e59c5-dfc1-4874-bd19-76a660c353c5\") " Oct 30 04:21:00.570494 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.570363 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-data-science-collector-ta-client-cert\") pod \"e80e59c5-dfc1-4874-bd19-76a660c353c5\" (UID: \"e80e59c5-dfc1-4874-bd19-76a660c353c5\") " Oct 30 04:21:00.570494 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.570403 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-monitoring-tls-certs\") pod \"e80e59c5-dfc1-4874-bd19-76a660c353c5\" (UID: \"e80e59c5-dfc1-4874-bd19-76a660c353c5\") " Oct 30 04:21:00.570494 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.570451 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e80e59c5-dfc1-4874-bd19-76a660c353c5-otc-internal\" (UniqueName: \"kubernetes.io/configmap/e80e59c5-dfc1-4874-bd19-76a660c353c5-otc-internal\") pod \"e80e59c5-dfc1-4874-bd19-76a660c353c5\" (UID: \"e80e59c5-dfc1-4874-bd19-76a660c353c5\") " Oct 30 04:21:00.572392 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.571068 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80e59c5-dfc1-4874-bd19-76a660c353c5-otc-internal" pod "e80e59c5-dfc1-4874-bd19-76a660c353c5" (UID: "e80e59c5-dfc1-4874-bd19-76a660c353c5"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:21:00.573861 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.573821 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-tls-certs" pod "e80e59c5-dfc1-4874-bd19-76a660c353c5" (UID: "e80e59c5-dfc1-4874-bd19-76a660c353c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:00.573988 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.573866 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80e59c5-dfc1-4874-bd19-76a660c353c5-kube-api-access-g6rhf" pod "e80e59c5-dfc1-4874-bd19-76a660c353c5" (UID: "e80e59c5-dfc1-4874-bd19-76a660c353c5"). InnerVolumeSpecName "kube-api-access-g6rhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:00.574854 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.574826 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-monitoring-tls-certs" pod "e80e59c5-dfc1-4874-bd19-76a660c353c5" (UID: "e80e59c5-dfc1-4874-bd19-76a660c353c5"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:00.575036 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.575017 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-data-science-collector-ta-client-cert" pod "e80e59c5-dfc1-4874-bd19-76a660c353c5" (UID: "e80e59c5-dfc1-4874-bd19-76a660c353c5"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:00.671630 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.671597 4925 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e80e59c5-dfc1-4874-bd19-76a660c353c5-otc-internal\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:00.671630 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.671625 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6rhf\" (UniqueName: \"kubernetes.io/projected/e80e59c5-dfc1-4874-bd19-76a660c353c5-kube-api-access-g6rhf\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:00.671843 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.671640 4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:00.671843 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.671649 4925 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-data-science-collector-ta-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:00.671843 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:00.671657 4925 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e80e59c5-dfc1-4874-bd19-76a660c353c5-monitoring-tls-certs\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:01.023580 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:01.023547 4925 generic.go:356] "Generic (PLEG): container finished" podID="e80e59c5-dfc1-4874-bd19-76a660c353c5" containerID="065d794106a3035a84d9a428497a10e188aef42683c5ee490738672b4109eaf8" exitCode=0 Oct 30 04:21:01.023824 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:01.023591 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e80e59c5-dfc1-4874-bd19-76a660c353c5","Type":"ContainerDied","Data":"065d794106a3035a84d9a428497a10e188aef42683c5ee490738672b4109eaf8"} Oct 30 04:21:01.023824 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:01.023608 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 04:21:01.023824 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:01.023657 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e80e59c5-dfc1-4874-bd19-76a660c353c5","Type":"ContainerDied","Data":"be80acea67234f8f707117b78066fa153d88194994ff0a0086bb96a381a5751a"} Oct 30 04:21:01.023824 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:01.023679 4925 scope.go:122] "RemoveContainer" containerID="065d794106a3035a84d9a428497a10e188aef42683c5ee490738672b4109eaf8" Oct 30 04:21:01.033254 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:01.033227 4925 scope.go:122] "RemoveContainer" containerID="065d794106a3035a84d9a428497a10e188aef42683c5ee490738672b4109eaf8" Oct 30 04:21:01.033540 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:21:01.033519 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"065d794106a3035a84d9a428497a10e188aef42683c5ee490738672b4109eaf8\": container with ID starting with 065d794106a3035a84d9a428497a10e188aef42683c5ee490738672b4109eaf8 not found: ID does not exist" containerID="065d794106a3035a84d9a428497a10e188aef42683c5ee490738672b4109eaf8" Oct 30 04:21:01.033621 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:01.033548 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"065d794106a3035a84d9a428497a10e188aef42683c5ee490738672b4109eaf8"} err="failed to get container status \"065d794106a3035a84d9a428497a10e188aef42683c5ee490738672b4109eaf8\": rpc error: code = NotFound desc = could not find container \"065d794106a3035a84d9a428497a10e188aef42683c5ee490738672b4109eaf8\": container with ID starting with 065d794106a3035a84d9a428497a10e188aef42683c5ee490738672b4109eaf8 not found: ID does not exist" Oct 30 04:21:01.075663 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:01.075634 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:21:01.079831 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:01.079808 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 04:21:01.119694 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:01.119662 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d087d6e-df54-4262-90e3-86142f1a39cc" path="/var/lib/kubelet/pods/7d087d6e-df54-4262-90e3-86142f1a39cc/volumes" Oct 30 04:21:01.120098 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:01.120083 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91407852-200e-4ce6-89a1-ef11eed5a272" path="/var/lib/kubelet/pods/91407852-200e-4ce6-89a1-ef11eed5a272/volumes" Oct 30 04:21:01.120476 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:01.120465 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8f035ab-52f5-46c0-9bbe-9e94699c3416" path="/var/lib/kubelet/pods/b8f035ab-52f5-46c0-9bbe-9e94699c3416/volumes" Oct 30 04:21:01.121105 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:01.121088 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e80e59c5-dfc1-4874-bd19-76a660c353c5" path="/var/lib/kubelet/pods/e80e59c5-dfc1-4874-bd19-76a660c353c5/volumes" Oct 30 04:21:04.846945 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:04.846905 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="5a655f24-a28d-45ee-ad1d-d395051e550a" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.159:9093/-/ready\": dial tcp 10.132.0.159:9093: connect: connection refused" Oct 30 04:21:09.920916 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:09.920893 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:21:09.951950 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:09.951888 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a655f24-a28d-45ee-ad1d-d395051e550a-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/5a655f24-a28d-45ee-ad1d-d395051e550a-alertmanager-data-science-monitoringstack-db\") pod \"5a655f24-a28d-45ee-ad1d-d395051e550a\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " Oct 30 04:21:09.951950 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:09.951925 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a655f24-a28d-45ee-ad1d-d395051e550a-kube-api-access-pvvkz\" (UniqueName: \"kubernetes.io/projected/5a655f24-a28d-45ee-ad1d-d395051e550a-kube-api-access-pvvkz\") pod \"5a655f24-a28d-45ee-ad1d-d395051e550a\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " Oct 30 04:21:09.951950 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:09.951951 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-config-volume\" (UniqueName: \"kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-config-volume\") pod \"5a655f24-a28d-45ee-ad1d-d395051e550a\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " Oct 30 04:21:09.952199 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:09.951976 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a655f24-a28d-45ee-ad1d-d395051e550a-config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a655f24-a28d-45ee-ad1d-d395051e550a-config-out\") pod \"5a655f24-a28d-45ee-ad1d-d395051e550a\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " Oct 30 04:21:09.952199 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:09.951997 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-web-config\" (UniqueName: \"kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-web-config\") pod \"5a655f24-a28d-45ee-ad1d-d395051e550a\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " Oct 30 04:21:09.952199 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:09.952110 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-cluster-tls-config\") pod \"5a655f24-a28d-45ee-ad1d-d395051e550a\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " Oct 30 04:21:09.952199 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:09.952170 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a655f24-a28d-45ee-ad1d-d395051e550a-tls-assets\" (UniqueName: \"kubernetes.io/projected/5a655f24-a28d-45ee-ad1d-d395051e550a-tls-assets\") pod \"5a655f24-a28d-45ee-ad1d-d395051e550a\" (UID: \"5a655f24-a28d-45ee-ad1d-d395051e550a\") " Oct 30 04:21:09.952417 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:09.952301 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a655f24-a28d-45ee-ad1d-d395051e550a-alertmanager-data-science-monitoringstack-db" pod "5a655f24-a28d-45ee-ad1d-d395051e550a" (UID: "5a655f24-a28d-45ee-ad1d-d395051e550a"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:21:09.952514 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:09.952494 4925 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/5a655f24-a28d-45ee-ad1d-d395051e550a-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:09.954566 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:09.954541 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a655f24-a28d-45ee-ad1d-d395051e550a-config-out" pod "5a655f24-a28d-45ee-ad1d-d395051e550a" (UID: "5a655f24-a28d-45ee-ad1d-d395051e550a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:21:09.954654 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:09.954572 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-config-volume" pod "5a655f24-a28d-45ee-ad1d-d395051e550a" (UID: "5a655f24-a28d-45ee-ad1d-d395051e550a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:09.954654 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:09.954572 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a655f24-a28d-45ee-ad1d-d395051e550a-kube-api-access-pvvkz" pod "5a655f24-a28d-45ee-ad1d-d395051e550a" (UID: "5a655f24-a28d-45ee-ad1d-d395051e550a"). InnerVolumeSpecName "kube-api-access-pvvkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:09.954738 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:09.954667 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a655f24-a28d-45ee-ad1d-d395051e550a-tls-assets" pod "5a655f24-a28d-45ee-ad1d-d395051e550a" (UID: "5a655f24-a28d-45ee-ad1d-d395051e550a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:09.958339 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:09.958318 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-cluster-tls-config" pod "5a655f24-a28d-45ee-ad1d-d395051e550a" (UID: "5a655f24-a28d-45ee-ad1d-d395051e550a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:09.966414 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:09.966390 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-web-config" pod "5a655f24-a28d-45ee-ad1d-d395051e550a" (UID: "5a655f24-a28d-45ee-ad1d-d395051e550a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:10.053709 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.053675 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvvkz\" (UniqueName: \"kubernetes.io/projected/5a655f24-a28d-45ee-ad1d-d395051e550a-kube-api-access-pvvkz\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:10.053709 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.053707 4925 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-config-volume\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:10.053931 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.053720 4925 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a655f24-a28d-45ee-ad1d-d395051e550a-config-out\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:10.053931 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.053732 4925 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-web-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:10.053931 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.053742 4925 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a655f24-a28d-45ee-ad1d-d395051e550a-cluster-tls-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:10.053931 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.053778 4925 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a655f24-a28d-45ee-ad1d-d395051e550a-tls-assets\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:10.071173 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.071144 4925 generic.go:356] "Generic (PLEG): container finished" podID="5a655f24-a28d-45ee-ad1d-d395051e550a" containerID="74403dee8fed26680e92b48124e58174fa97757b7f69dc6ee388668af34cd8a5" exitCode=0 Oct 30 04:21:10.071290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.071205 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 04:21:10.071290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.071223 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"5a655f24-a28d-45ee-ad1d-d395051e550a","Type":"ContainerDied","Data":"74403dee8fed26680e92b48124e58174fa97757b7f69dc6ee388668af34cd8a5"} Oct 30 04:21:10.071290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.071283 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"5a655f24-a28d-45ee-ad1d-d395051e550a","Type":"ContainerDied","Data":"19a0553f07add9f88f873f6508774976e4265419a5b38e0f9ba5bbd0c4ddfe9f"} Oct 30 04:21:10.071407 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.071313 4925 scope.go:122] "RemoveContainer" containerID="e726aae75fe3d1f1c5ef73a037d017bfe2cfd1e2d38f0c47b71444e568d8d7aa" Oct 30 04:21:10.079956 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.079939 4925 scope.go:122] "RemoveContainer" containerID="74403dee8fed26680e92b48124e58174fa97757b7f69dc6ee388668af34cd8a5" Oct 30 04:21:10.087890 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.087873 4925 scope.go:122] "RemoveContainer" containerID="fc662faa47a9e9a14c67a364ef539a74f05f697753b8541c4689484099fb6706" Oct 30 04:21:10.095346 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.095319 4925 scope.go:122] "RemoveContainer" containerID="e726aae75fe3d1f1c5ef73a037d017bfe2cfd1e2d38f0c47b71444e568d8d7aa" Oct 30 04:21:10.095602 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:21:10.095582 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e726aae75fe3d1f1c5ef73a037d017bfe2cfd1e2d38f0c47b71444e568d8d7aa\": container with ID starting with e726aae75fe3d1f1c5ef73a037d017bfe2cfd1e2d38f0c47b71444e568d8d7aa not found: ID does not exist" containerID="e726aae75fe3d1f1c5ef73a037d017bfe2cfd1e2d38f0c47b71444e568d8d7aa" Oct 30 04:21:10.095669 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.095613 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e726aae75fe3d1f1c5ef73a037d017bfe2cfd1e2d38f0c47b71444e568d8d7aa"} err="failed to get container status \"e726aae75fe3d1f1c5ef73a037d017bfe2cfd1e2d38f0c47b71444e568d8d7aa\": rpc error: code = NotFound desc = could not find container \"e726aae75fe3d1f1c5ef73a037d017bfe2cfd1e2d38f0c47b71444e568d8d7aa\": container with ID starting with e726aae75fe3d1f1c5ef73a037d017bfe2cfd1e2d38f0c47b71444e568d8d7aa not found: ID does not exist" Oct 30 04:21:10.095669 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.095638 4925 scope.go:122] "RemoveContainer" containerID="74403dee8fed26680e92b48124e58174fa97757b7f69dc6ee388668af34cd8a5" Oct 30 04:21:10.095924 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:21:10.095906 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74403dee8fed26680e92b48124e58174fa97757b7f69dc6ee388668af34cd8a5\": container with ID starting with 74403dee8fed26680e92b48124e58174fa97757b7f69dc6ee388668af34cd8a5 not found: ID does not exist" containerID="74403dee8fed26680e92b48124e58174fa97757b7f69dc6ee388668af34cd8a5" Oct 30 04:21:10.095988 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.095933 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74403dee8fed26680e92b48124e58174fa97757b7f69dc6ee388668af34cd8a5"} err="failed to get container status \"74403dee8fed26680e92b48124e58174fa97757b7f69dc6ee388668af34cd8a5\": rpc error: code = NotFound desc = could not find container \"74403dee8fed26680e92b48124e58174fa97757b7f69dc6ee388668af34cd8a5\": container with ID starting with 74403dee8fed26680e92b48124e58174fa97757b7f69dc6ee388668af34cd8a5 not found: ID does not exist" Oct 30 04:21:10.095988 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.095953 4925 scope.go:122] "RemoveContainer" containerID="fc662faa47a9e9a14c67a364ef539a74f05f697753b8541c4689484099fb6706" Oct 30 04:21:10.096202 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:21:10.096181 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc662faa47a9e9a14c67a364ef539a74f05f697753b8541c4689484099fb6706\": container with ID starting with fc662faa47a9e9a14c67a364ef539a74f05f697753b8541c4689484099fb6706 not found: ID does not exist" containerID="fc662faa47a9e9a14c67a364ef539a74f05f697753b8541c4689484099fb6706" Oct 30 04:21:10.096251 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.096208 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc662faa47a9e9a14c67a364ef539a74f05f697753b8541c4689484099fb6706"} err="failed to get container status \"fc662faa47a9e9a14c67a364ef539a74f05f697753b8541c4689484099fb6706\": rpc error: code = NotFound desc = could not find container \"fc662faa47a9e9a14c67a364ef539a74f05f697753b8541c4689484099fb6706\": container with ID starting with fc662faa47a9e9a14c67a364ef539a74f05f697753b8541c4689484099fb6706 not found: ID does not exist" Oct 30 04:21:10.124398 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.124373 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:21:10.128167 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:10.128146 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 04:21:11.119355 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.119313 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a655f24-a28d-45ee-ad1d-d395051e550a" path="/var/lib/kubelet/pods/5a655f24-a28d-45ee-ad1d-d395051e550a/volumes" Oct 30 04:21:11.969929 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.969896 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg"] Oct 30 04:21:11.970219 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970206 4925 cpu_manager.go:414] "containerMap: removing container" podUID="b8f035ab-52f5-46c0-9bbe-9e94699c3416" containerName="kube-rbac-proxy" Oct 30 04:21:11.970272 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970219 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8f035ab-52f5-46c0-9bbe-9e94699c3416" containerName="kube-rbac-proxy" Oct 30 04:21:11.970272 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970234 4925 cpu_manager.go:414] "containerMap: removing container" podUID="d636c395-8d40-43f9-aa83-60e89193951f" containerName="ta-container" Oct 30 04:21:11.970272 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970239 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d636c395-8d40-43f9-aa83-60e89193951f" containerName="ta-container" Oct 30 04:21:11.970272 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970248 4925 cpu_manager.go:414] "containerMap: removing container" podUID="5a655f24-a28d-45ee-ad1d-d395051e550a" containerName="config-reloader" Oct 30 04:21:11.970272 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970253 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a655f24-a28d-45ee-ad1d-d395051e550a" containerName="config-reloader" Oct 30 04:21:11.970272 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970262 4925 cpu_manager.go:414] "containerMap: removing container" podUID="5a655f24-a28d-45ee-ad1d-d395051e550a" containerName="init-config-reloader" Oct 30 04:21:11.970272 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970266 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a655f24-a28d-45ee-ad1d-d395051e550a" containerName="init-config-reloader" Oct 30 04:21:11.970272 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970273 4925 cpu_manager.go:414] "containerMap: removing container" podUID="7d087d6e-df54-4262-90e3-86142f1a39cc" containerName="korrel8r" Oct 30 04:21:11.970615 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970277 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d087d6e-df54-4262-90e3-86142f1a39cc" containerName="korrel8r" Oct 30 04:21:11.970615 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970287 4925 cpu_manager.go:414] "containerMap: removing container" podUID="e80e59c5-dfc1-4874-bd19-76a660c353c5" containerName="otc-container" Oct 30 04:21:11.970615 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970293 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e80e59c5-dfc1-4874-bd19-76a660c353c5" containerName="otc-container" Oct 30 04:21:11.970615 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970309 4925 cpu_manager.go:414] "containerMap: removing container" podUID="91407852-200e-4ce6-89a1-ef11eed5a272" containerName="prom-label-proxy" Oct 30 04:21:11.970615 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970313 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91407852-200e-4ce6-89a1-ef11eed5a272" containerName="prom-label-proxy" Oct 30 04:21:11.970615 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970319 4925 cpu_manager.go:414] "containerMap: removing container" podUID="5a655f24-a28d-45ee-ad1d-d395051e550a" containerName="alertmanager" Oct 30 04:21:11.970615 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970323 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a655f24-a28d-45ee-ad1d-d395051e550a" containerName="alertmanager" Oct 30 04:21:11.970615 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970331 4925 cpu_manager.go:414] "containerMap: removing container" podUID="91407852-200e-4ce6-89a1-ef11eed5a272" containerName="kube-rbac-proxy" Oct 30 04:21:11.970615 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970335 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91407852-200e-4ce6-89a1-ef11eed5a272" containerName="kube-rbac-proxy" Oct 30 04:21:11.970615 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970380 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d636c395-8d40-43f9-aa83-60e89193951f" containerName="ta-container" Oct 30 04:21:11.970615 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970388 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a655f24-a28d-45ee-ad1d-d395051e550a" containerName="config-reloader" Oct 30 04:21:11.970615 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970395 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d087d6e-df54-4262-90e3-86142f1a39cc" containerName="korrel8r" Oct 30 04:21:11.970615 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970432 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e80e59c5-dfc1-4874-bd19-76a660c353c5" containerName="otc-container" Oct 30 04:21:11.970615 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970440 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="91407852-200e-4ce6-89a1-ef11eed5a272" containerName="prom-label-proxy" Oct 30 04:21:11.970615 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970445 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a655f24-a28d-45ee-ad1d-d395051e550a" containerName="alertmanager" Oct 30 04:21:11.970615 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970461 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="91407852-200e-4ce6-89a1-ef11eed5a272" containerName="kube-rbac-proxy" Oct 30 04:21:11.970615 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.970471 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8f035ab-52f5-46c0-9bbe-9e94699c3416" containerName="kube-rbac-proxy" Oct 30 04:21:11.975289 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:11.975271 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:12.005670 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.005640 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 04:21:12.005670 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.005663 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 30 04:21:12.005917 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.005688 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-f8xnq\"" Oct 30 04:21:12.005917 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.005734 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 30 04:21:12.005917 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.005765 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 04:21:12.005917 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.005796 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 04:21:12.006129 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.006108 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 04:21:12.008191 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.008173 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg"] Oct 30 04:21:12.022649 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.022623 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w"] Oct 30 04:21:12.026120 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.026105 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" Oct 30 04:21:12.040352 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.040330 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w"] Oct 30 04:21:12.062133 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.062110 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv"] Oct 30 04:21:12.065537 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.065520 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" Oct 30 04:21:12.068293 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.068270 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/cc1117b6-7a42-4305-b53a-cabde6bbec35-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-dprzg\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:12.068363 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.068324 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8gq2w\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" Oct 30 04:21:12.068481 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.068458 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-dprzg\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:12.068521 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.068507 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-dprzg\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:12.068579 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.068564 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-dprzg\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:12.068650 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.068637 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8gq2w\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" Oct 30 04:21:12.068691 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.068662 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgxqz\" (UniqueName: \"kubernetes.io/projected/97fb0902-2312-4d02-9e97-7ed1af146ebd-kube-api-access-zgxqz\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8gq2w\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" Oct 30 04:21:12.068691 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.068685 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/97fb0902-2312-4d02-9e97-7ed1af146ebd-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8gq2w\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" Oct 30 04:21:12.068828 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.068807 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9drhp\" (UniqueName: \"kubernetes.io/projected/cc1117b6-7a42-4305-b53a-cabde6bbec35-kube-api-access-9drhp\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-dprzg\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:12.068881 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.068842 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8gq2w\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" Oct 30 04:21:12.087896 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.087874 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv"] Oct 30 04:21:12.093945 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.093923 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 04:21:12.093945 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.093943 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 04:21:12.094131 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.093953 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-ntr8b\"" Oct 30 04:21:12.094131 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.093999 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 04:21:12.094131 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.094008 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 04:21:12.094131 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.094058 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 04:21:12.094131 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.094125 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-gvw47\"" Oct 30 04:21:12.109946 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.109925 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc"] Oct 30 04:21:12.113304 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.113280 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" Oct 30 04:21:12.133313 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.133291 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc"] Oct 30 04:21:12.169206 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.169178 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8gq2w\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" Oct 30 04:21:12.169322 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.169209 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-tls\") pod \"korrel8r-5cbcc89b8f-p22gc\" (UID: \"2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" Oct 30 04:21:12.169322 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.169226 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/fd77fca5-d5b0-4778-9df6-253c4a6a4800-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-r22lv\" (UID: \"fd77fca5-d5b0-4778-9df6-253c4a6a4800\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" Oct 30 04:21:12.169419 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.169333 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgxqz\" (UniqueName: \"kubernetes.io/projected/97fb0902-2312-4d02-9e97-7ed1af146ebd-kube-api-access-zgxqz\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8gq2w\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" Oct 30 04:21:12.169419 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.169396 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd77fca5-d5b0-4778-9df6-253c4a6a4800-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-r22lv\" (UID: \"fd77fca5-d5b0-4778-9df6-253c4a6a4800\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" Oct 30 04:21:12.169532 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.169432 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/97fb0902-2312-4d02-9e97-7ed1af146ebd-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8gq2w\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" Oct 30 04:21:12.169532 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.169460 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9drhp\" (UniqueName: \"kubernetes.io/projected/cc1117b6-7a42-4305-b53a-cabde6bbec35-kube-api-access-9drhp\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-dprzg\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:12.169532 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.169480 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8gq2w\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" Oct 30 04:21:12.169532 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.169506 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-config\") pod \"korrel8r-5cbcc89b8f-p22gc\" (UID: \"2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" Oct 30 04:21:12.169866 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.169547 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/cc1117b6-7a42-4305-b53a-cabde6bbec35-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-dprzg\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:12.169866 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:21:12.169563 4925 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-prometheus-namespace-proxy-tls: secret "data-science-prometheus-namespace-proxy-tls" not found Oct 30 04:21:12.169866 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.169593 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8gq2w\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" Oct 30 04:21:12.169866 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:21:12.169619 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97fb0902-2312-4d02-9e97-7ed1af146ebd-tls podName:97fb0902-2312-4d02-9e97-7ed1af146ebd nodeName:}" failed. No retries permitted until 2026-10-30 04:21:12.669599718 +0000 UTC m=+6033.682877802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/97fb0902-2312-4d02-9e97-7ed1af146ebd-tls") pod "data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" (UID: "97fb0902-2312-4d02-9e97-7ed1af146ebd") : secret "data-science-prometheus-namespace-proxy-tls" not found Oct 30 04:21:12.169866 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.169649 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b8tb\" (UniqueName: \"kubernetes.io/projected/fd77fca5-d5b0-4778-9df6-253c4a6a4800-kube-api-access-2b8tb\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-r22lv\" (UID: \"fd77fca5-d5b0-4778-9df6-253c4a6a4800\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" Oct 30 04:21:12.169866 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.169691 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mddv2\" (UniqueName: \"kubernetes.io/projected/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-kube-api-access-mddv2\") pod \"korrel8r-5cbcc89b8f-p22gc\" (UID: \"2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" Oct 30 04:21:12.169866 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.169849 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-dprzg\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:12.170188 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.169891 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-dprzg\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:12.170188 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.169920 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-dprzg\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:12.170447 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.170262 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8gq2w\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" Oct 30 04:21:12.170512 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.170426 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8gq2w\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" Oct 30 04:21:12.170610 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:21:12.170582 4925 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 30 04:21:12.170884 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.170858 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/cc1117b6-7a42-4305-b53a-cabde6bbec35-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-dprzg\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:12.170954 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:21:12.170922 4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-prometheus-client-cert podName:cc1117b6-7a42-4305-b53a-cabde6bbec35 nodeName:}" failed. No retries permitted until 2026-10-30 04:21:12.670894408 +0000 UTC m=+6033.684172506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-dprzg" (UID: "cc1117b6-7a42-4305-b53a-cabde6bbec35") : secret "prometheus-operated-tls" not found Oct 30 04:21:12.171518 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.171487 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8gq2w\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" Oct 30 04:21:12.179167 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.179146 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-dprzg\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:12.180069 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.180050 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-dprzg\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:12.193325 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.193302 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9drhp\" (UniqueName: \"kubernetes.io/projected/cc1117b6-7a42-4305-b53a-cabde6bbec35-kube-api-access-9drhp\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-dprzg\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:12.193414 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.193302 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgxqz\" (UniqueName: \"kubernetes.io/projected/97fb0902-2312-4d02-9e97-7ed1af146ebd-kube-api-access-zgxqz\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8gq2w\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" Oct 30 04:21:12.194012 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.193988 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 04:21:12.194012 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.194005 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 04:21:12.194137 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.193989 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-zqw6f\"" Oct 30 04:21:12.271398 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.271331 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-tls\") pod \"korrel8r-5cbcc89b8f-p22gc\" (UID: \"2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" Oct 30 04:21:12.271398 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.271360 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/fd77fca5-d5b0-4778-9df6-253c4a6a4800-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-r22lv\" (UID: \"fd77fca5-d5b0-4778-9df6-253c4a6a4800\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" Oct 30 04:21:12.271398 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.271391 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd77fca5-d5b0-4778-9df6-253c4a6a4800-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-r22lv\" (UID: \"fd77fca5-d5b0-4778-9df6-253c4a6a4800\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" Oct 30 04:21:12.271641 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.271430 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-config\") pod \"korrel8r-5cbcc89b8f-p22gc\" (UID: \"2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" Oct 30 04:21:12.271641 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.271479 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b8tb\" (UniqueName: \"kubernetes.io/projected/fd77fca5-d5b0-4778-9df6-253c4a6a4800-kube-api-access-2b8tb\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-r22lv\" (UID: \"fd77fca5-d5b0-4778-9df6-253c4a6a4800\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" Oct 30 04:21:12.271837 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.271816 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mddv2\" (UniqueName: \"kubernetes.io/projected/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-kube-api-access-mddv2\") pod \"korrel8r-5cbcc89b8f-p22gc\" (UID: \"2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" Oct 30 04:21:12.272240 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.272204 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd77fca5-d5b0-4778-9df6-253c4a6a4800-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-r22lv\" (UID: \"fd77fca5-d5b0-4778-9df6-253c4a6a4800\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" Oct 30 04:21:12.272353 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.272337 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-config\") pod \"korrel8r-5cbcc89b8f-p22gc\" (UID: \"2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" Oct 30 04:21:12.273483 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.273462 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-tls\") pod \"korrel8r-5cbcc89b8f-p22gc\" (UID: \"2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" Oct 30 04:21:12.273789 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.273770 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/fd77fca5-d5b0-4778-9df6-253c4a6a4800-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-r22lv\" (UID: \"fd77fca5-d5b0-4778-9df6-253c4a6a4800\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" Oct 30 04:21:12.295000 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.294974 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mddv2\" (UniqueName: \"kubernetes.io/projected/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-kube-api-access-mddv2\") pod \"korrel8r-5cbcc89b8f-p22gc\" (UID: \"2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" Oct 30 04:21:12.295090 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.295070 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b8tb\" (UniqueName: \"kubernetes.io/projected/fd77fca5-d5b0-4778-9df6-253c4a6a4800-kube-api-access-2b8tb\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-r22lv\" (UID: \"fd77fca5-d5b0-4778-9df6-253c4a6a4800\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" Oct 30 04:21:12.380674 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.380647 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" Oct 30 04:21:12.426614 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.426559 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" Oct 30 04:21:12.675293 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.675255 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-dprzg\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:12.675483 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.675324 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/97fb0902-2312-4d02-9e97-7ed1af146ebd-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8gq2w\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" Oct 30 04:21:12.677042 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.676986 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv"] Oct 30 04:21:12.677945 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.677923 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/97fb0902-2312-4d02-9e97-7ed1af146ebd-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8gq2w\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" Oct 30 04:21:12.678458 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.678409 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-dprzg\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:12.679950 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:21:12.679929 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd77fca5_d5b0_4778_9df6_253c4a6a4800.slice/crio-81ed4498cf3a59bfa051583989b1572c15d662e0b519ec2ae3354eb4acd93a30 WatchSource:0}: Error finding container 81ed4498cf3a59bfa051583989b1572c15d662e0b519ec2ae3354eb4acd93a30: Status 404 returned error can't find the container with id 81ed4498cf3a59bfa051583989b1572c15d662e0b519ec2ae3354eb4acd93a30 Oct 30 04:21:12.737502 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.737412 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc"] Oct 30 04:21:12.742003 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:21:12.741968 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9ed35a_bf89_4e24_a47a_0d0cb0ed3f7f.slice/crio-abdc1c3eec19274488a6669d30c3e7dc8a7b3730c9a567ede1c1ff85894beae1 WatchSource:0}: Error finding container abdc1c3eec19274488a6669d30c3e7dc8a7b3730c9a567ede1c1ff85894beae1: Status 404 returned error can't find the container with id abdc1c3eec19274488a6669d30c3e7dc8a7b3730c9a567ede1c1ff85894beae1 Oct 30 04:21:12.846568 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.846535 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="5a655f24-a28d-45ee-ad1d-d395051e550a" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.159:9093/-/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:21:12.888201 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.888179 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:12.942198 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:12.942166 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" Oct 30 04:21:13.090625 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:13.090584 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" event={"ID":"fd77fca5-d5b0-4778-9df6-253c4a6a4800","Type":"ContainerStarted","Data":"535398829a4bf0f070f8b93fe32fd98f5308ef2902710de95ffe4e4dc63c081b"} Oct 30 04:21:13.090836 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:13.090658 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" event={"ID":"fd77fca5-d5b0-4778-9df6-253c4a6a4800","Type":"ContainerStarted","Data":"b3832e017abbbb44d463945b8bbccc200532bb2c53f459efe7a411347560d70c"} Oct 30 04:21:13.090836 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:13.090692 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" event={"ID":"fd77fca5-d5b0-4778-9df6-253c4a6a4800","Type":"ContainerStarted","Data":"81ed4498cf3a59bfa051583989b1572c15d662e0b519ec2ae3354eb4acd93a30"} Oct 30 04:21:13.092217 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:13.092190 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" event={"ID":"2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f","Type":"ContainerStarted","Data":"27fc6a6a2fb1a777e6836998eedf0ed8c1c8dc3043944ccbc358c8ff2677e6db"} Oct 30 04:21:13.092313 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:13.092238 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" Oct 30 04:21:13.092313 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:13.092255 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" event={"ID":"2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f","Type":"ContainerStarted","Data":"abdc1c3eec19274488a6669d30c3e7dc8a7b3730c9a567ede1c1ff85894beae1"} Oct 30 04:21:13.133136 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:13.133090 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" podStartSLOduration=1.133077356 podStartE2EDuration="1.133077356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:13.131654583 +0000 UTC m=+6034.144932688" watchObservedRunningTime="2026-10-30 04:21:13.133077356 +0000 UTC m=+6034.146355449" Oct 30 04:21:13.173517 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:13.173466 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" podStartSLOduration=1.173450433 podStartE2EDuration="1.173450433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:13.171745187 +0000 UTC m=+6034.185023289" watchObservedRunningTime="2026-10-30 04:21:13.173450433 +0000 UTC m=+6034.186728548" Oct 30 04:21:13.186923 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:13.186865 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg"] Oct 30 04:21:13.189282 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:21:13.189245 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1117b6_7a42_4305_b53a_cabde6bbec35.slice/crio-0fc68b34c3e226884a5441f26daf27812139bcc35200bb022f5241e6595435ea WatchSource:0}: Error finding container 0fc68b34c3e226884a5441f26daf27812139bcc35200bb022f5241e6595435ea: Status 404 returned error can't find the container with id 0fc68b34c3e226884a5441f26daf27812139bcc35200bb022f5241e6595435ea Oct 30 04:21:13.248153 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:13.248132 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w"] Oct 30 04:21:13.330133 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:21:13.330101 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97fb0902_2312_4d02_9e97_7ed1af146ebd.slice/crio-c9d31a67a04564c2be026d51957936a1af184c7c37c115a55024d414de4aad33 WatchSource:0}: Error finding container c9d31a67a04564c2be026d51957936a1af184c7c37c115a55024d414de4aad33: Status 404 returned error can't find the container with id c9d31a67a04564c2be026d51957936a1af184c7c37c115a55024d414de4aad33 Oct 30 04:21:14.098824 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:14.098786 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" event={"ID":"97fb0902-2312-4d02-9e97-7ed1af146ebd","Type":"ContainerStarted","Data":"16d35faa59b3aea5f6bea969c2ef221c72c14f44642a7fa72f057bfb07e68ec3"} Oct 30 04:21:14.099022 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:14.098854 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" event={"ID":"97fb0902-2312-4d02-9e97-7ed1af146ebd","Type":"ContainerStarted","Data":"dca0c3509195f1ffe6584f94b5b36a92bef90c04af3af963291184e2a21c810d"} Oct 30 04:21:14.099022 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:14.098893 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" event={"ID":"97fb0902-2312-4d02-9e97-7ed1af146ebd","Type":"ContainerStarted","Data":"c9d31a67a04564c2be026d51957936a1af184c7c37c115a55024d414de4aad33"} Oct 30 04:21:14.100279 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:14.100255 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" event={"ID":"cc1117b6-7a42-4305-b53a-cabde6bbec35","Type":"ContainerStarted","Data":"011e8c4ca28afa76ae8b5287f704c52857bef53852734348813d9a46cd5e7dee"} Oct 30 04:21:14.100339 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:14.100298 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" event={"ID":"cc1117b6-7a42-4305-b53a-cabde6bbec35","Type":"ContainerStarted","Data":"0fc68b34c3e226884a5441f26daf27812139bcc35200bb022f5241e6595435ea"} Oct 30 04:21:14.100562 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:14.100547 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:14.179251 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:14.179197 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" podStartSLOduration=3.179184229 podStartE2EDuration="3.179184229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:14.178477618 +0000 UTC m=+6035.191755725" watchObservedRunningTime="2026-10-30 04:21:14.179184229 +0000 UTC m=+6035.192462396" Oct 30 04:21:14.223068 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:14.223010 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" podStartSLOduration=3.22299761 podStartE2EDuration="3.22299761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:14.221431657 +0000 UTC m=+6035.234709760" watchObservedRunningTime="2026-10-30 04:21:14.22299761 +0000 UTC m=+6035.236275714" Oct 30 04:21:18.161637 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:21:18.161601 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:21:20.107982 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:20.107949 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:24.102946 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:24.102911 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" Oct 30 04:21:40.065110 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.065067 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg"] Oct 30 04:21:40.065524 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.065255 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" podUID="cc1117b6-7a42-4305-b53a-cabde6bbec35" containerName="kube-rbac-proxy" containerID="cri-o://011e8c4ca28afa76ae8b5287f704c52857bef53852734348813d9a46cd5e7dee" gracePeriod=30 Oct 30 04:21:40.087642 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.087608 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w"] Oct 30 04:21:40.087928 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.087901 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" podUID="97fb0902-2312-4d02-9e97-7ed1af146ebd" containerName="kube-rbac-proxy" containerID="cri-o://dca0c3509195f1ffe6584f94b5b36a92bef90c04af3af963291184e2a21c810d" gracePeriod=30 Oct 30 04:21:40.088008 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.087901 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" podUID="97fb0902-2312-4d02-9e97-7ed1af146ebd" containerName="prom-label-proxy" containerID="cri-o://16d35faa59b3aea5f6bea969c2ef221c72c14f44642a7fa72f057bfb07e68ec3" gracePeriod=30 Oct 30 04:21:40.105665 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.105636 4925 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" podUID="cc1117b6-7a42-4305-b53a-cabde6bbec35" containerName="kube-rbac-proxy" probeResult="failure" output="dial tcp 10.132.0.163:8443: connect: connection refused" Oct 30 04:21:40.107168 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.107138 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv"] Oct 30 04:21:40.107464 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.107419 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" podUID="fd77fca5-d5b0-4778-9df6-253c4a6a4800" containerName="kube-rbac-proxy" containerID="cri-o://b3832e017abbbb44d463945b8bbccc200532bb2c53f459efe7a411347560d70c" gracePeriod=30 Oct 30 04:21:40.107584 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.107544 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" podUID="fd77fca5-d5b0-4778-9df6-253c4a6a4800" containerName="prom-label-proxy" containerID="cri-o://535398829a4bf0f070f8b93fe32fd98f5308ef2902710de95ffe4e4dc63c081b" gracePeriod=30 Oct 30 04:21:40.125572 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.125547 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc"] Oct 30 04:21:40.125734 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.125714 4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" podUID="2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f" containerName="korrel8r" containerID="cri-o://27fc6a6a2fb1a777e6836998eedf0ed8c1c8dc3043944ccbc358c8ff2677e6db" gracePeriod=30 Oct 30 04:21:40.229059 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.229023 4925 generic.go:356] "Generic (PLEG): container finished" podID="97fb0902-2312-4d02-9e97-7ed1af146ebd" containerID="16d35faa59b3aea5f6bea969c2ef221c72c14f44642a7fa72f057bfb07e68ec3" exitCode=0 Oct 30 04:21:40.229059 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.229045 4925 generic.go:356] "Generic (PLEG): container finished" podID="97fb0902-2312-4d02-9e97-7ed1af146ebd" containerID="dca0c3509195f1ffe6584f94b5b36a92bef90c04af3af963291184e2a21c810d" exitCode=0 Oct 30 04:21:40.229290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.229112 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" event={"ID":"97fb0902-2312-4d02-9e97-7ed1af146ebd","Type":"ContainerDied","Data":"16d35faa59b3aea5f6bea969c2ef221c72c14f44642a7fa72f057bfb07e68ec3"} Oct 30 04:21:40.229290 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.229182 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" event={"ID":"97fb0902-2312-4d02-9e97-7ed1af146ebd","Type":"ContainerDied","Data":"dca0c3509195f1ffe6584f94b5b36a92bef90c04af3af963291184e2a21c810d"} Oct 30 04:21:40.231182 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.231163 4925 generic.go:356] "Generic (PLEG): container finished" podID="fd77fca5-d5b0-4778-9df6-253c4a6a4800" containerID="535398829a4bf0f070f8b93fe32fd98f5308ef2902710de95ffe4e4dc63c081b" exitCode=0 Oct 30 04:21:40.231182 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.231180 4925 generic.go:356] "Generic (PLEG): container finished" podID="fd77fca5-d5b0-4778-9df6-253c4a6a4800" containerID="b3832e017abbbb44d463945b8bbccc200532bb2c53f459efe7a411347560d70c" exitCode=0 Oct 30 04:21:40.231337 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.231221 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" event={"ID":"fd77fca5-d5b0-4778-9df6-253c4a6a4800","Type":"ContainerDied","Data":"535398829a4bf0f070f8b93fe32fd98f5308ef2902710de95ffe4e4dc63c081b"} Oct 30 04:21:40.231440 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.231260 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" event={"ID":"fd77fca5-d5b0-4778-9df6-253c4a6a4800","Type":"ContainerDied","Data":"b3832e017abbbb44d463945b8bbccc200532bb2c53f459efe7a411347560d70c"} Oct 30 04:21:40.390736 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.390703 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" Oct 30 04:21:40.434774 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.434558 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" Oct 30 04:21:40.483209 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.483176 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:40.485394 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.485368 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" Oct 30 04:21:40.521461 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.521425 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-prometheus-tls-ca\") pod \"97fb0902-2312-4d02-9e97-7ed1af146ebd\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " Oct 30 04:21:40.521652 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.521482 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97fb0902-2312-4d02-9e97-7ed1af146ebd-kube-api-access-zgxqz\" (UniqueName: \"kubernetes.io/projected/97fb0902-2312-4d02-9e97-7ed1af146ebd-kube-api-access-zgxqz\") pod \"97fb0902-2312-4d02-9e97-7ed1af146ebd\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " Oct 30 04:21:40.521652 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.521514 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd77fca5-d5b0-4778-9df6-253c4a6a4800-kube-api-access-2b8tb\" (UniqueName: \"kubernetes.io/projected/fd77fca5-d5b0-4778-9df6-253c4a6a4800-kube-api-access-2b8tb\") pod \"fd77fca5-d5b0-4778-9df6-253c4a6a4800\" (UID: \"fd77fca5-d5b0-4778-9df6-253c4a6a4800\") " Oct 30 04:21:40.521652 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.521542 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd77fca5-d5b0-4778-9df6-253c4a6a4800-config\" (UniqueName: \"kubernetes.io/configmap/fd77fca5-d5b0-4778-9df6-253c4a6a4800-config\") pod \"fd77fca5-d5b0-4778-9df6-253c4a6a4800\" (UID: \"fd77fca5-d5b0-4778-9df6-253c4a6a4800\") " Oct 30 04:21:40.521891 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.521793 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-prometheus-tls-ca" pod "97fb0902-2312-4d02-9e97-7ed1af146ebd" (UID: "97fb0902-2312-4d02-9e97-7ed1af146ebd"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:21:40.522009 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.521920 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd77fca5-d5b0-4778-9df6-253c4a6a4800-config" pod "fd77fca5-d5b0-4778-9df6-253c4a6a4800" (UID: "fd77fca5-d5b0-4778-9df6-253c4a6a4800"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:21:40.522068 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.522030 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-client-ca" pod "97fb0902-2312-4d02-9e97-7ed1af146ebd" (UID: "97fb0902-2312-4d02-9e97-7ed1af146ebd"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:21:40.522068 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.521745 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-client-ca\" (UniqueName: \"kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-client-ca\") pod \"97fb0902-2312-4d02-9e97-7ed1af146ebd\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " Oct 30 04:21:40.522204 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.522080 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-config\" (UniqueName: \"kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-config\") pod \"97fb0902-2312-4d02-9e97-7ed1af146ebd\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " Oct 30 04:21:40.522204 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.522108 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97fb0902-2312-4d02-9e97-7ed1af146ebd-tls\" (UniqueName: \"kubernetes.io/secret/97fb0902-2312-4d02-9e97-7ed1af146ebd-tls\") pod \"97fb0902-2312-4d02-9e97-7ed1af146ebd\" (UID: \"97fb0902-2312-4d02-9e97-7ed1af146ebd\") " Oct 30 04:21:40.522204 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.522162 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd77fca5-d5b0-4778-9df6-253c4a6a4800-tls\" (UniqueName: \"kubernetes.io/secret/fd77fca5-d5b0-4778-9df6-253c4a6a4800-tls\") pod \"fd77fca5-d5b0-4778-9df6-253c4a6a4800\" (UID: \"fd77fca5-d5b0-4778-9df6-253c4a6a4800\") " Oct 30 04:21:40.522487 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.522463 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-config" pod "97fb0902-2312-4d02-9e97-7ed1af146ebd" (UID: "97fb0902-2312-4d02-9e97-7ed1af146ebd"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:21:40.522556 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.522468 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-prometheus-tls-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:40.522556 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.522519 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd77fca5-d5b0-4778-9df6-253c4a6a4800-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:40.522556 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.522537 4925 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-client-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:40.523965 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.523898 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd77fca5-d5b0-4778-9df6-253c4a6a4800-kube-api-access-2b8tb" pod "fd77fca5-d5b0-4778-9df6-253c4a6a4800" (UID: "fd77fca5-d5b0-4778-9df6-253c4a6a4800"). InnerVolumeSpecName "kube-api-access-2b8tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:40.524241 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.524216 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97fb0902-2312-4d02-9e97-7ed1af146ebd-tls" pod "97fb0902-2312-4d02-9e97-7ed1af146ebd" (UID: "97fb0902-2312-4d02-9e97-7ed1af146ebd"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:40.524870 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.524620 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97fb0902-2312-4d02-9e97-7ed1af146ebd-kube-api-access-zgxqz" pod "97fb0902-2312-4d02-9e97-7ed1af146ebd" (UID: "97fb0902-2312-4d02-9e97-7ed1af146ebd"). InnerVolumeSpecName "kube-api-access-zgxqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:40.525092 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.525049 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd77fca5-d5b0-4778-9df6-253c4a6a4800-tls" pod "fd77fca5-d5b0-4778-9df6-253c4a6a4800" (UID: "fd77fca5-d5b0-4778-9df6-253c4a6a4800"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:40.623030 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.622954 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-prometheus-client-cert\") pod \"cc1117b6-7a42-4305-b53a-cabde6bbec35\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " Oct 30 04:21:40.623030 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.622995 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-tls\" (UniqueName: \"kubernetes.io/secret/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-tls\") pod \"2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f\" (UID: \"2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f\") " Oct 30 04:21:40.623030 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.623025 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-config\" (UniqueName: \"kubernetes.io/configmap/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-config\") pod \"2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f\" (UID: \"2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f\") " Oct 30 04:21:40.623293 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.623060 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1117b6-7a42-4305-b53a-cabde6bbec35-kube-api-access-9drhp\" (UniqueName: \"kubernetes.io/projected/cc1117b6-7a42-4305-b53a-cabde6bbec35-kube-api-access-9drhp\") pod \"cc1117b6-7a42-4305-b53a-cabde6bbec35\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " Oct 30 04:21:40.623293 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.623095 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-tls\" (UniqueName: \"kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-tls\") pod \"cc1117b6-7a42-4305-b53a-cabde6bbec35\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " Oct 30 04:21:40.623293 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.623131 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-kube-rbac-proxy-config\") pod \"cc1117b6-7a42-4305-b53a-cabde6bbec35\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " Oct 30 04:21:40.623293 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.623162 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1117b6-7a42-4305-b53a-cabde6bbec35-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/cc1117b6-7a42-4305-b53a-cabde6bbec35-prometheus-ca\") pod \"cc1117b6-7a42-4305-b53a-cabde6bbec35\" (UID: \"cc1117b6-7a42-4305-b53a-cabde6bbec35\") " Oct 30 04:21:40.623293 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.623186 4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-kube-api-access-mddv2\" (UniqueName: \"kubernetes.io/projected/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-kube-api-access-mddv2\") pod \"2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f\" (UID: \"2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f\") " Oct 30 04:21:40.623560 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.623433 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/97fb0902-2312-4d02-9e97-7ed1af146ebd-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:40.623560 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.623429 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-config" pod "2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f" (UID: "2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:21:40.623560 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.623454 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/97fb0902-2312-4d02-9e97-7ed1af146ebd-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:40.623560 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.623490 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/fd77fca5-d5b0-4778-9df6-253c4a6a4800-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:40.623560 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.623506 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgxqz\" (UniqueName: \"kubernetes.io/projected/97fb0902-2312-4d02-9e97-7ed1af146ebd-kube-api-access-zgxqz\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:40.623560 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.623520 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b8tb\" (UniqueName: \"kubernetes.io/projected/fd77fca5-d5b0-4778-9df6-253c4a6a4800-kube-api-access-2b8tb\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:40.624198 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.624124 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1117b6-7a42-4305-b53a-cabde6bbec35-prometheus-ca" pod "cc1117b6-7a42-4305-b53a-cabde6bbec35" (UID: "cc1117b6-7a42-4305-b53a-cabde6bbec35"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:21:40.625705 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.625675 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1117b6-7a42-4305-b53a-cabde6bbec35-kube-api-access-9drhp" pod "cc1117b6-7a42-4305-b53a-cabde6bbec35" (UID: "cc1117b6-7a42-4305-b53a-cabde6bbec35"). InnerVolumeSpecName "kube-api-access-9drhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:40.625857 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.625719 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-kube-api-access-mddv2" pod "2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f" (UID: "2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f"). InnerVolumeSpecName "kube-api-access-mddv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:40.625857 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.625786 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-tls" pod "cc1117b6-7a42-4305-b53a-cabde6bbec35" (UID: "cc1117b6-7a42-4305-b53a-cabde6bbec35"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:40.625985 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.625890 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-prometheus-client-cert" pod "cc1117b6-7a42-4305-b53a-cabde6bbec35" (UID: "cc1117b6-7a42-4305-b53a-cabde6bbec35"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:40.626066 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.626048 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-tls" pod "2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f" (UID: "2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:40.626138 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.626123 4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-kube-rbac-proxy-config" pod "cc1117b6-7a42-4305-b53a-cabde6bbec35" (UID: "cc1117b6-7a42-4305-b53a-cabde6bbec35"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:21:40.724641 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.724604 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9drhp\" (UniqueName: \"kubernetes.io/projected/cc1117b6-7a42-4305-b53a-cabde6bbec35-kube-api-access-9drhp\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:40.724641 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.724633 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:40.724641 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.724642 4925 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-kube-rbac-proxy-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:40.724641 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.724651 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/cc1117b6-7a42-4305-b53a-cabde6bbec35-prometheus-ca\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:40.724948 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.724659 4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mddv2\" (UniqueName: \"kubernetes.io/projected/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-kube-api-access-mddv2\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:40.724948 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.724668 4925 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/cc1117b6-7a42-4305-b53a-cabde6bbec35-prometheus-client-cert\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:40.724948 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.724676 4925 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-tls\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:40.724948 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:40.724684 4925 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f-config\") on node \"ip-10-0-139-19.ec2.internal\" DevicePath \"\"" Oct 30 04:21:41.240937 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.240267 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" Oct 30 04:21:41.240937 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.240314 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w" event={"ID":"97fb0902-2312-4d02-9e97-7ed1af146ebd","Type":"ContainerDied","Data":"c9d31a67a04564c2be026d51957936a1af184c7c37c115a55024d414de4aad33"} Oct 30 04:21:41.240937 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.240397 4925 scope.go:122] "RemoveContainer" containerID="16d35faa59b3aea5f6bea969c2ef221c72c14f44642a7fa72f057bfb07e68ec3" Oct 30 04:21:41.245398 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.245368 4925 generic.go:356] "Generic (PLEG): container finished" podID="cc1117b6-7a42-4305-b53a-cabde6bbec35" containerID="011e8c4ca28afa76ae8b5287f704c52857bef53852734348813d9a46cd5e7dee" exitCode=0 Oct 30 04:21:41.245537 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.245423 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" Oct 30 04:21:41.245537 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.245487 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" event={"ID":"cc1117b6-7a42-4305-b53a-cabde6bbec35","Type":"ContainerDied","Data":"011e8c4ca28afa76ae8b5287f704c52857bef53852734348813d9a46cd5e7dee"} Oct 30 04:21:41.245537 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.245526 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg" event={"ID":"cc1117b6-7a42-4305-b53a-cabde6bbec35","Type":"ContainerDied","Data":"0fc68b34c3e226884a5441f26daf27812139bcc35200bb022f5241e6595435ea"} Oct 30 04:21:41.248325 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.248299 4925 generic.go:356] "Generic (PLEG): container finished" podID="2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f" containerID="27fc6a6a2fb1a777e6836998eedf0ed8c1c8dc3043944ccbc358c8ff2677e6db" exitCode=0 Oct 30 04:21:41.248478 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.248460 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" event={"ID":"2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f","Type":"ContainerDied","Data":"27fc6a6a2fb1a777e6836998eedf0ed8c1c8dc3043944ccbc358c8ff2677e6db"} Oct 30 04:21:41.248557 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.248505 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" event={"ID":"2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f","Type":"ContainerDied","Data":"abdc1c3eec19274488a6669d30c3e7dc8a7b3730c9a567ede1c1ff85894beae1"} Oct 30 04:21:41.248683 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.248578 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc" Oct 30 04:21:41.251250 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.251092 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" event={"ID":"fd77fca5-d5b0-4778-9df6-253c4a6a4800","Type":"ContainerDied","Data":"81ed4498cf3a59bfa051583989b1572c15d662e0b519ec2ae3354eb4acd93a30"} Oct 30 04:21:41.251250 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.251198 4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv" Oct 30 04:21:41.256139 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.256115 4925 scope.go:122] "RemoveContainer" containerID="dca0c3509195f1ffe6584f94b5b36a92bef90c04af3af963291184e2a21c810d" Oct 30 04:21:41.265243 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.265224 4925 scope.go:122] "RemoveContainer" containerID="011e8c4ca28afa76ae8b5287f704c52857bef53852734348813d9a46cd5e7dee" Oct 30 04:21:41.278876 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.278855 4925 scope.go:122] "RemoveContainer" containerID="011e8c4ca28afa76ae8b5287f704c52857bef53852734348813d9a46cd5e7dee" Oct 30 04:21:41.279154 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:21:41.279135 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"011e8c4ca28afa76ae8b5287f704c52857bef53852734348813d9a46cd5e7dee\": container with ID starting with 011e8c4ca28afa76ae8b5287f704c52857bef53852734348813d9a46cd5e7dee not found: ID does not exist" containerID="011e8c4ca28afa76ae8b5287f704c52857bef53852734348813d9a46cd5e7dee" Oct 30 04:21:41.279261 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.279165 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011e8c4ca28afa76ae8b5287f704c52857bef53852734348813d9a46cd5e7dee"} err="failed to get container status \"011e8c4ca28afa76ae8b5287f704c52857bef53852734348813d9a46cd5e7dee\": rpc error: code = NotFound desc = could not find container \"011e8c4ca28afa76ae8b5287f704c52857bef53852734348813d9a46cd5e7dee\": container with ID starting with 011e8c4ca28afa76ae8b5287f704c52857bef53852734348813d9a46cd5e7dee not found: ID does not exist" Oct 30 04:21:41.279261 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.279189 4925 scope.go:122] "RemoveContainer" containerID="27fc6a6a2fb1a777e6836998eedf0ed8c1c8dc3043944ccbc358c8ff2677e6db" Oct 30 04:21:41.287460 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.287374 4925 scope.go:122] "RemoveContainer" containerID="27fc6a6a2fb1a777e6836998eedf0ed8c1c8dc3043944ccbc358c8ff2677e6db" Oct 30 04:21:41.287778 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:21:41.287732 4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27fc6a6a2fb1a777e6836998eedf0ed8c1c8dc3043944ccbc358c8ff2677e6db\": container with ID starting with 27fc6a6a2fb1a777e6836998eedf0ed8c1c8dc3043944ccbc358c8ff2677e6db not found: ID does not exist" containerID="27fc6a6a2fb1a777e6836998eedf0ed8c1c8dc3043944ccbc358c8ff2677e6db" Oct 30 04:21:41.287861 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.287775 4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27fc6a6a2fb1a777e6836998eedf0ed8c1c8dc3043944ccbc358c8ff2677e6db"} err="failed to get container status \"27fc6a6a2fb1a777e6836998eedf0ed8c1c8dc3043944ccbc358c8ff2677e6db\": rpc error: code = NotFound desc = could not find container \"27fc6a6a2fb1a777e6836998eedf0ed8c1c8dc3043944ccbc358c8ff2677e6db\": container with ID starting with 27fc6a6a2fb1a777e6836998eedf0ed8c1c8dc3043944ccbc358c8ff2677e6db not found: ID does not exist" Oct 30 04:21:41.287861 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.287795 4925 scope.go:122] "RemoveContainer" containerID="535398829a4bf0f070f8b93fe32fd98f5308ef2902710de95ffe4e4dc63c081b" Oct 30 04:21:41.294338 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.294317 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg"] Oct 30 04:21:41.296592 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.296570 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-dprzg"] Oct 30 04:21:41.297189 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.297172 4925 scope.go:122] "RemoveContainer" containerID="b3832e017abbbb44d463945b8bbccc200532bb2c53f459efe7a411347560d70c" Oct 30 04:21:41.330679 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.330649 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w"] Oct 30 04:21:41.336040 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.336018 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8gq2w"] Oct 30 04:21:41.371965 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.371878 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc"] Oct 30 04:21:41.376734 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.376709 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-p22gc"] Oct 30 04:21:41.413365 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.413335 4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv"] Oct 30 04:21:41.415072 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:41.415051 4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-r22lv"] Oct 30 04:21:43.119305 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:43.119270 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f" path="/var/lib/kubelet/pods/2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f/volumes" Oct 30 04:21:43.119962 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:43.119862 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97fb0902-2312-4d02-9e97-7ed1af146ebd" path="/var/lib/kubelet/pods/97fb0902-2312-4d02-9e97-7ed1af146ebd/volumes" Oct 30 04:21:43.120338 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:43.120317 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc1117b6-7a42-4305-b53a-cabde6bbec35" path="/var/lib/kubelet/pods/cc1117b6-7a42-4305-b53a-cabde6bbec35/volumes" Oct 30 04:21:43.121017 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:43.120994 4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd77fca5-d5b0-4778-9df6-253c4a6a4800" path="/var/lib/kubelet/pods/fd77fca5-d5b0-4778-9df6-253c4a6a4800/volumes" Oct 30 04:21:48.161174 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:21:48.161137 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:21:54.420993 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:21:54.420961 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-observability-system_odh-observability-b9cbbfc76-6z5ws_dc3ab9cb-d80d-4ec7-9208-5d6510623867/manager/0.log" Oct 30 04:22:00.407451 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:00.407415 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m9zrt_ef5e843c-9ceb-44f3-bfc9-c56be367804e/global-pull-secret-syncer/0.log" Oct 30 04:22:00.583717 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:00.583688 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-69v7b_8ab12b0c-6c4b-4915-9920-9eec9c279f3b/connection-checker/0.log" Oct 30 04:22:00.707929 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:00.707852 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6tnpg_da5b2698-93b5-4b4a-8404-f22ec7840967/connection-checker/0.log" Oct 30 04:22:00.832565 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:00.832539 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-9s4nm_114fc00f-05b8-4a65-89e1-5997231c05e7/connection-checker/0.log" Oct 30 04:22:01.118263 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:01.118221 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n96rg_8f17a0e8-ce89-4a88-98f2-c7782260534d/konnectivity-agent/0.log" Oct 30 04:22:01.544414 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:01.544333 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-19.ec2.internal_36dcf6a72ae2b4d471dead03fce936be/haproxy/0.log" Oct 30 04:22:07.089686 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:07.089654 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-bshlg_bc230021-ead4-497f-9933-45fbdb9efc50/cluster-monitoring-operator/0.log" Oct 30 04:22:08.593691 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:08.593660 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cc8q2_6743f565-3d91-49c1-9c4a-9bc6ab5b434b/node-exporter/0.log" Oct 30 04:22:08.717938 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:08.717908 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cc8q2_6743f565-3d91-49c1-9c4a-9bc6ab5b434b/kube-rbac-proxy/0.log" Oct 30 04:22:08.838118 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:08.838091 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cc8q2_6743f565-3d91-49c1-9c4a-9bc6ab5b434b/init-textfile/0.log" Oct 30 04:22:09.490219 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.490185 4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d"] Oct 30 04:22:09.490583 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.490562 4925 cpu_manager.go:414] "containerMap: removing container" podUID="fd77fca5-d5b0-4778-9df6-253c4a6a4800" containerName="prom-label-proxy" Oct 30 04:22:09.490583 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.490585 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd77fca5-d5b0-4778-9df6-253c4a6a4800" containerName="prom-label-proxy" Oct 30 04:22:09.490815 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.490611 4925 cpu_manager.go:414] "containerMap: removing container" podUID="2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f" containerName="korrel8r" Oct 30 04:22:09.490815 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.490618 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f" containerName="korrel8r" Oct 30 04:22:09.490815 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.490776 4925 cpu_manager.go:414] "containerMap: removing container" podUID="97fb0902-2312-4d02-9e97-7ed1af146ebd" containerName="kube-rbac-proxy" Oct 30 04:22:09.490815 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.490794 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97fb0902-2312-4d02-9e97-7ed1af146ebd" containerName="kube-rbac-proxy" Oct 30 04:22:09.490815 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.490814 4925 cpu_manager.go:414] "containerMap: removing container" podUID="cc1117b6-7a42-4305-b53a-cabde6bbec35" containerName="kube-rbac-proxy" Oct 30 04:22:09.491010 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.490821 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1117b6-7a42-4305-b53a-cabde6bbec35" containerName="kube-rbac-proxy" Oct 30 04:22:09.491010 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.490853 4925 cpu_manager.go:414] "containerMap: removing container" podUID="fd77fca5-d5b0-4778-9df6-253c4a6a4800" containerName="kube-rbac-proxy" Oct 30 04:22:09.491010 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.490861 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd77fca5-d5b0-4778-9df6-253c4a6a4800" containerName="kube-rbac-proxy" Oct 30 04:22:09.491010 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.490884 4925 cpu_manager.go:414] "containerMap: removing container" podUID="97fb0902-2312-4d02-9e97-7ed1af146ebd" containerName="prom-label-proxy" Oct 30 04:22:09.491010 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.490890 4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97fb0902-2312-4d02-9e97-7ed1af146ebd" containerName="prom-label-proxy" Oct 30 04:22:09.491010 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.490981 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="97fb0902-2312-4d02-9e97-7ed1af146ebd" containerName="prom-label-proxy" Oct 30 04:22:09.491010 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.490991 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd77fca5-d5b0-4778-9df6-253c4a6a4800" containerName="prom-label-proxy" Oct 30 04:22:09.491010 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.490999 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9ed35a-bf89-4e24-a47a-0d0cb0ed3f7f" containerName="korrel8r" Oct 30 04:22:09.491010 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.491003 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="97fb0902-2312-4d02-9e97-7ed1af146ebd" containerName="kube-rbac-proxy" Oct 30 04:22:09.491010 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.491010 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1117b6-7a42-4305-b53a-cabde6bbec35" containerName="kube-rbac-proxy" Oct 30 04:22:09.491300 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.491017 4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd77fca5-d5b0-4778-9df6-253c4a6a4800" containerName="kube-rbac-proxy" Oct 30 04:22:09.494167 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.494149 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" Oct 30 04:22:09.514544 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.514513 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d"] Oct 30 04:22:09.514869 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.514849 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zrscj\"/\"kube-root-ca.crt\"" Oct 30 04:22:09.525300 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.525279 4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zrscj\"/\"openshift-service-ca.crt\"" Oct 30 04:22:09.549708 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.549683 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a7569ae9-40f3-431a-bb84-2dc92c577144-podres\") pod \"perf-node-gather-daemonset-vvd8d\" (UID: \"a7569ae9-40f3-431a-bb84-2dc92c577144\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" Oct 30 04:22:09.549858 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.549733 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkmfm\" (UniqueName: \"kubernetes.io/projected/a7569ae9-40f3-431a-bb84-2dc92c577144-kube-api-access-wkmfm\") pod \"perf-node-gather-daemonset-vvd8d\" (UID: \"a7569ae9-40f3-431a-bb84-2dc92c577144\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" Oct 30 04:22:09.549858 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.549842 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a7569ae9-40f3-431a-bb84-2dc92c577144-proc\") pod \"perf-node-gather-daemonset-vvd8d\" (UID: \"a7569ae9-40f3-431a-bb84-2dc92c577144\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" Oct 30 04:22:09.549984 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.549965 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7569ae9-40f3-431a-bb84-2dc92c577144-sys\") pod \"perf-node-gather-daemonset-vvd8d\" (UID: \"a7569ae9-40f3-431a-bb84-2dc92c577144\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" Oct 30 04:22:09.550024 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.549995 4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7569ae9-40f3-431a-bb84-2dc92c577144-lib-modules\") pod \"perf-node-gather-daemonset-vvd8d\" (UID: \"a7569ae9-40f3-431a-bb84-2dc92c577144\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" Oct 30 04:22:09.610030 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.610005 4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zrscj\"/\"default-dockercfg-c8v22\"" Oct 30 04:22:09.650523 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.650498 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a7569ae9-40f3-431a-bb84-2dc92c577144-proc\") pod \"perf-node-gather-daemonset-vvd8d\" (UID: \"a7569ae9-40f3-431a-bb84-2dc92c577144\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" Oct 30 04:22:09.650654 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.650550 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7569ae9-40f3-431a-bb84-2dc92c577144-sys\") pod \"perf-node-gather-daemonset-vvd8d\" (UID: \"a7569ae9-40f3-431a-bb84-2dc92c577144\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" Oct 30 04:22:09.650654 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.650636 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a7569ae9-40f3-431a-bb84-2dc92c577144-proc\") pod \"perf-node-gather-daemonset-vvd8d\" (UID: \"a7569ae9-40f3-431a-bb84-2dc92c577144\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" Oct 30 04:22:09.650802 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.650646 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7569ae9-40f3-431a-bb84-2dc92c577144-lib-modules\") pod \"perf-node-gather-daemonset-vvd8d\" (UID: \"a7569ae9-40f3-431a-bb84-2dc92c577144\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" Oct 30 04:22:09.650802 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.650700 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7569ae9-40f3-431a-bb84-2dc92c577144-sys\") pod \"perf-node-gather-daemonset-vvd8d\" (UID: \"a7569ae9-40f3-431a-bb84-2dc92c577144\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" Oct 30 04:22:09.650802 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.650710 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a7569ae9-40f3-431a-bb84-2dc92c577144-podres\") pod \"perf-node-gather-daemonset-vvd8d\" (UID: \"a7569ae9-40f3-431a-bb84-2dc92c577144\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" Oct 30 04:22:09.650802 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.650706 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7569ae9-40f3-431a-bb84-2dc92c577144-lib-modules\") pod \"perf-node-gather-daemonset-vvd8d\" (UID: \"a7569ae9-40f3-431a-bb84-2dc92c577144\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" Oct 30 04:22:09.650802 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.650789 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a7569ae9-40f3-431a-bb84-2dc92c577144-podres\") pod \"perf-node-gather-daemonset-vvd8d\" (UID: \"a7569ae9-40f3-431a-bb84-2dc92c577144\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" Oct 30 04:22:09.650802 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.650770 4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkmfm\" (UniqueName: \"kubernetes.io/projected/a7569ae9-40f3-431a-bb84-2dc92c577144-kube-api-access-wkmfm\") pod \"perf-node-gather-daemonset-vvd8d\" (UID: \"a7569ae9-40f3-431a-bb84-2dc92c577144\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" Oct 30 04:22:09.674306 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.674275 4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkmfm\" (UniqueName: \"kubernetes.io/projected/a7569ae9-40f3-431a-bb84-2dc92c577144-kube-api-access-wkmfm\") pod \"perf-node-gather-daemonset-vvd8d\" (UID: \"a7569ae9-40f3-431a-bb84-2dc92c577144\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" Oct 30 04:22:09.805799 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:09.805685 4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" Oct 30 04:22:10.106497 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:10.106292 4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d"] Oct 30 04:22:10.108989 ip-10-0-139-19 kubenswrapper[4925]: W1030 04:22:10.108963 4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda7569ae9_40f3_431a_bb84_2dc92c577144.slice/crio-09206eca2dc429a848daa834866e400de87bf5da3266afe2fe71d6a37350bf00 WatchSource:0}: Error finding container 09206eca2dc429a848daa834866e400de87bf5da3266afe2fe71d6a37350bf00: Status 404 returned error can't find the container with id 09206eca2dc429a848daa834866e400de87bf5da3266afe2fe71d6a37350bf00 Oct 30 04:22:10.393987 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:10.393902 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" event={"ID":"a7569ae9-40f3-431a-bb84-2dc92c577144","Type":"ContainerStarted","Data":"85334cb697d32e38e8b34b7a91fe20075f95e184b13585457e3b96ad264ad1de"} Oct 30 04:22:10.393987 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:10.393964 4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" event={"ID":"a7569ae9-40f3-431a-bb84-2dc92c577144","Type":"ContainerStarted","Data":"09206eca2dc429a848daa834866e400de87bf5da3266afe2fe71d6a37350bf00"} Oct 30 04:22:10.394179 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:10.394029 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" Oct 30 04:22:10.438931 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:10.438880 4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" podStartSLOduration=1.4388670129999999 podStartE2EDuration="1.438867013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:22:10.437699231 +0000 UTC m=+6091.450977335" watchObservedRunningTime="2026-10-30 04:22:10.438867013 +0000 UTC m=+6091.452145133" Oct 30 04:22:12.804018 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:12.803962 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-lttrq_3fb3b957-3183-4c9e-a45f-c6930341197c/networking-console-plugin/0.log" Oct 30 04:22:13.743074 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:13.743048 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/2.log" Oct 30 04:22:13.747316 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:13.747299 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vb5mb_21ee3aae-ce3d-4fc0-9419-60fb4c762ad3/console-operator/3.log" Oct 30 04:22:15.701996 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:15.701968 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-jsswm_4295fa46-8c63-4cde-8183-1895149bc2ce/volume-data-source-validator/0.log" Oct 30 04:22:16.408528 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:16.408499 4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vvd8d" Oct 30 04:22:18.101883 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:18.101854 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sx7m8_ba886f04-386a-4cb7-aa00-b85f3921e6cc/dns/0.log" Oct 30 04:22:18.161062 ip-10-0-139-19 kubenswrapper[4925]: E1030 04:22:18.161027 4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2rxp" containerName="extractor" Oct 30 04:22:18.224193 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:18.224164 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sx7m8_ba886f04-386a-4cb7-aa00-b85f3921e6cc/kube-rbac-proxy/0.log" Oct 30 04:22:18.630868 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:18.630840 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hbr7c_5e9def4d-df87-404d-90eb-1b21b023fbd8/dns-node-resolver/0.log" Oct 30 04:22:20.071516 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:20.071478 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zg4qh_a3c34ce6-91f3-42e4-ba49-a905f477571e/node-ca/0.log" Oct 30 04:22:21.726196 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:21.726164 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6bffcf78f7-rplsj_548e9394-45be-4ffa-937a-89fea1da3093/router/0.log" Oct 30 04:22:22.809096 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:22.809061 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r779s_02aed17c-f8e1-4fd0-bb90-4e972bd1a827/serve-healthcheck-canary/0.log" Oct 30 04:22:23.648061 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:23.648029 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-zc4bx_b0851f53-fa88-40f9-bfdb-beb956a5fb28/insights-operator/0.log" Oct 30 04:22:23.649337 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:23.649318 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-zc4bx_b0851f53-fa88-40f9-bfdb-beb956a5fb28/insights-operator/1.log" Oct 30 04:22:24.122427 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:24.122402 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j2rxp_c24d2a3f-bbb2-491e-9f25-608364102b2a/kube-rbac-proxy/0.log" Oct 30 04:22:24.243615 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:24.243589 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j2rxp_c24d2a3f-bbb2-491e-9f25-608364102b2a/exporter/0.log" Oct 30 04:22:24.363943 ip-10-0-139-19 kubenswrapper[4925]: I1030 04:22:24.363914 4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j2rxp_c24d2a3f-bbb2-491e-9f25-608364102b2a/extractor/0.log"